I don't agree with the article: odd business premise since we use S3 because it is replicated across availability zones, is very robust, and probably safe enough to base a business on. Why "de-duplicate" when redundant storage is what makes S3 secure. Also, what is the risk of trusting your business to a smaller company?
back
2 comments
The idea is to deduplicate / eliminate redundancy one layer up. So if you are storing an image in S3 let's say it's replicated 10 times to ensure data safety. If I store the same image it doesn't need to be stored another 10 times.
Getting customers to trust a smaller company's software is definitely a top risk. I wouldn't start this company now, but it might be worth doing in 2015. Watch this space!
My biggest concern about the article is that it doesn't prove its premise. In fact, the S3 service is only arbitrageable if all the other things you need to do to build your proposed service are free, or less expensive (including opportunity cost) than the profits you generate by doing them. This is far from certain, because even with the infrastructure all built it's not clear to me that you could turn a profit when you factor in your running costs like CPU and RAM used to actually make the system function, much less the support staff.
De-duplication and replication are complements. The more you can de-duplicate, the better and cheaper you can make your replication. If you know your data is highly redundant (for example, you are backing up all the files on thousands of desktops created with a standard disk image), de-duping will allow you to dramatically improve the replication (perhaps replicated 4x between two data centers nearly in real time, instead of copied to a single RAID-10 array in one data center on a nightly basis).
If you don't de-dupe your data before you replicate it, then you end up storing redundant data in a way where the redundancy is totally unhelpful (in fact, hurtful) to you. So, you can look at de-duping as the process of converting hurtful redundancy into helpful redundancy.