The evolution of cloud storage as we know it today is a fascinating journey filled with projects that built on one another to bring us to where we are today. Interestingly enough, most of the technology that is utilized to build a decentralized cloud storage network today has been available for decades. The fact that decentralized cloud storage is viable today is mostly because of the growth of storage capacity available at the edge, and the incredible increases we’ve made across the globe in bandwidth. Here are four key advancements throughout the years that have paved the way for decentralized cloud storage.
Advancement #1: Network Bandwidth Increased
There is a great paper by Charles Blake and Rodrigo Rodrigues entitled High Availability, Scalable Storage, Dynamic Peer Networks: Pick Two. It was written in 2003, when bandwidth was the most restrictive component in achieving any two, let alone all three, of these. The truth is, bandwidth is still the most limiting factor — but today high speed bandwidth is widely available.
Decentralized cloud storage relies on storage node operators across a global network. People contribute extra hard drive space, so the availability of high quality bandwidth is a key factor in allowing this model to function. It has created an environment where there is a lot of inexpensive, latent capacity available, where data can be stored and accessed incredibly fast.
We now are in a position where we almost have data center level bandwidth globally. Judiciously using that bandwidth by using a Reed-Solomon erasure coding, instead of replication, allows us to build a system that is highly available AND highly scalable AND highly dynamic.
Advancement #2: Peer-to-Peer Technologies
In 2001, Napster was released. Napster was short-lived, but it was a pioneer in peer-to-peer file sharing, allowing users to share music files. Soon after, the Freenet and Gnutella peer-to-peer networks were launched. In 2004, the BitTorrent peer-to-peer protocol was developed and is still highly used today. And more recently in 2015, the Interplanetary File System (IPFS) launched. IPFS is a protocol and peer-to-peer network for storing and sharing data in a distributed file system. It is similar to BitTorrent in that it allows users to receive and host content, but it is unique in that it is a decentralized system of user-operators. [For a complete list of peer-to-peer advancements see the full timeline here.]
All of these advancements built off one another to find ways to solve the challenges of a highly distributed environment. Our team at Storj did this when we built our decentralized cloud storage network — we utilized peer-to-peer concepts to allow for storage in a highly distributed network to a massive scale. Keeping track of the data that's broken up in lots of pieces, stored in lots of places and constantly being created, is now a reality thanks to the work done by others in this space. By employing peer-to-peer technology and having multiple copies of the data stored on different nodes across various geographies, it leads to faster download speeds at the edge, where people are trying to access the data.
Advancement #3: Industry Experience with Distributed Systems
Back in the 60s, there were only a few research universities and government entities working on developing computing networks. Fast forward to today and just about every university — and many corporations — are involved in advancing our networks and finding better ways to handle distributed systems. Because of this combined experience in the industry, there is now language and research that didn’t exist even 10 years ago. Things like how to talk about tail latency and variability across large fleets of servers.
Some of our favorite decentralized projects that helped pave the way for decentralized cloud storage are early systems, like Freenet, Gnutella and Mojo Nation — of which alums went on to build BitTorrent, Tahoe LAFS, and ZCash — and of course IPFS. We’ve been able to take the building blocks from these systems and arrange them in a different way to provide better results.
Thanks to the research and the experience in the industry, we now have a number of well applied techniques for dealing with things like a highly variable tail latency. Of course, if you used any of these products over the past 20 years, you know that one of the major challenges with using decentralized storage was having your file always be there when you need it, no matter how popular it is. That is where available bandwidth comes into play and why this is still our number one advancement for enabling decentralized storage that is fast and scalable.
Advancement #4: Blockchains and Cryptocurrency
Cryptocurrency and blockchain technology is very exciting in terms of enabling people to transfer value between themselves in a democratized way, without a central authority and without double-spend. This technology has given us the ability to create payment and settlement mechanisms that provide integrity and are also decentralized. Blockchain and cryptocurrency technology has made it possible for us to mediate transfer of value for micropayments, with storage node operators in over 90 countries, and has enabled our network to achieve global scale.
What blockchain technology doesn’t help us with is building a decentralized network. Blockchains are traditionally definitionally slow. So-called “Proof-of-Work” Blockchains are designed with a defined amount of throughput, allowing consensus to be achieved on the state of transactions within the network. While this model is effective for reliable, immutable records that are highly resistant to tampering, blockchains also have high contention and latency. As our team built Storj DCS, we designed it to minimize contention and latency by only using blockchain as a settlement layer in the background.
Recognizing the Innovation Yesterday that Led New Technologies Today
Decentralized cloud storage is seeing adoption today because of the incredible work done by the pioneers of these technologies. We give homage to the technology innovation that came before us in our whitepaper that details the timeline of advancements in the cloud storage evolution.
This article originally published at The New Stack.
