In the Object Storage Landscape, Q3 2023 report, a distributed cloud storage provider was included. This is a milestone change in the cloud storage market as it signifies the start of the disruption of a very big industry that indeed needs some disruption.
Brent Ellis, Senior Analyst with Forrester, was recently a guest speaker in a webinar panel discussion where he shared his predictions about the future of cloud, cloud storage and how distributed (or decentralized) cloud storage is disrupting it. Brent was joined by two early movers in this disruption: Mario Blandini, VP of Marketing for iXsystems, and Jordan Maltby, CEO of Shadow Magic Studios. You can watch the full webinar here.
In this article I’ll do my best to recap the highlights of this discussion on future predictions for cloud storage as well as three big problems with today’s cloud storage that distributed cloud storage solves.
Why does object storage need a disruption?
Brent wrote in the Q3, 2023 Forrester Object Storage Landscape report that, “Decentralization of data storage will disrupt centralized object storage providers.” In order for this to be true, there needs to be clear problems with today’s cloud object storage (centralized) that distributed cloud object storage (decentralized) can solve. Brent, Mario, and Jordan discussed the key problems that are becoming unsustainable with today’s cloud storage.
Cloud storage issue #1: Too much complexity to get performance
Jordan Maltby, CEO of Shadow Magic Studios, painted the picture of the unwieldy complexity in today’s cloud storage. “You spin up a bucket in the US West, and then you start uploading it to US East, and you suddenly realize you're not getting the performance that you want. You're not getting the latency you want. So you solve that problem by spinning up another bucket in the East or even adding a CDN.
And then you have production stuff going in the East, then production stuff going in the West. And then you have archival needs. Now you need to do a cloud to cloud transfer. There's a lot of management that you may need to abstract by then having management software to keep track of multiple locations.”
How distributed cloud storage solves this: Distributed cloud storage encrypts, splits and distributes the segments across tens of thousands of nodes globally. With this unique architecture, your data is available globally by default. Multi-region storage is no longer needed—which reduces complexity and cost. And the performance in this model is fantastic even without a CDN.
Jordan, who uses Storj for distributed cloud storage for his business, commented on the performance he sees. “Storj lives up to its sales pitch. When we're uploading in New York, we get full line speed bi-directionally. When we're uploading and downloading in Los Angeles, we get full line speed bi-directionally. When we're in Europe, we get full line speed bi-directionally. When we're on our cloud and we're uploading and downloading on a 10 gig line, it maximizes that as well. And the idea that you can be uploading something in California and be immediately downloading it at full line speed on a one gig fiber in Paris or London without any intermediary acceleration application is, I think it's extremely disruptive. It's very empowering in its simplicity.”
Mario chimed in with a similar experience using Storj for TrueNAS. “We tested Storj to see if our users could experience better performance versus using a CDN. The results were much faster downloads globally. And if you don’t have to pay any extra money for a CDN and you get better performance. It’s a win-win.”
Cloud storage issue #2: Risk of service loss too high
We hear it in the news daily. Whether a malicious attack, a power outage from a natural disaster, or a man-made disruption such as a war, legacy cloud object storage is still very vulnerable. It is difficult to reduce this risk with traditional, centralized cloud storage. During the panel discussion, Brent explains how much of a concern this is for enterprises. “I talk to a lot of enterprise clients that have to manage things like cloud-based outages. And they have to start thinking about replicating buckets between different regions in order to provide resilience between cloud regions.” This adds cost and complexity.
How distributed cloud storage solves this: The Storj model of global distributed storage splits data into 64Mb segments, then splits each segment into 80 pieces which are each stored on a different storage node around the world. Only 29 pieces are needed to reconstitute the data.. This builds in replication that significantly lowers the risk from any type of disaster. Plus, the system intelligently retrieves those 29 pieces from the fastest nodes so performance is high no matter where you are in the world.
This theory was tested when Russia first invaded Ukraine. Storj was asked by customers, ‘How does this affect the resiliency of the Storj network?’ The response was very simple. Storj did some testing and doubled the number of replicas. So if Storj had to cut out the entire Russian region, we wouldn't have any resilience problems.
With Storj, a global resilience issue is just a simple change in the number of replicas. And going beyond that, Storj gives the end user some ability to mitigate against single points of failure within the actual provider. Brent added, “Distributed cloud storage provides this environment where the technology itself, the architecture of the network, actively resists single points of failure and gives you this opportunity to have worldwide storage resilience without having to do duplication of your data in multiple environments.”
Cloud storage issue #3: Too difficult to address compliance
Managing your cloud storage to fulfill the data sovereignty rules for protection and privacy across geographies can be challenging for enterprises to manage. Particularly proving that you are compliant. Brent explains on the panel how it is done with centralized cloud storage. “Right now you have to kind of jump through hoops to prove that essentially you're encrypting over HTTPS or VPN to get to [public hyperscaler object storage] right now, but having it part of the technology itself and not having another layer to slow things down is an important aspect of that. As we get to a world where you need to be able to show that your underlying storage is safe and protected from start to finish, that you're compliant with local regulations and laws, building on top of a technology that's just made to do that is an important aspect for data sovereignty.”
How distributed cloud storage solves this: Distributed cloud storage has been built with zero trust and zero knowledge, making full end-to-end encryption the default. It also provides the flexibility to select storage node groups. For example, a group of storage nodes that are all in a particular region or nodes that are all in certified data centers to address compliance and other requirements.
During the panel discussion, Brent details the benefits of distributed cloud storage for data sovereignty compliance. “It’s a natural fit for dealing with localized data privacy and sovereignty rules. Object storage already has versioning and immutability aspects that help. You can go back in time and look at a previous version of your bucket and access that information.”
“But as you increase the network speeds and you reduce the network latency, that starts to become primary storage. It becomes the first storage. And even in high performance environments like video production, where you might have a RAID drive that's on set and you're immediately dumping film to that RAID drive, I could see a future where that is a highly optimized object storage array. And at the same time that you're dumping data to that local object storage array, it's replicating it to the cloud.
“So you don't even have to take it back to your facility and initiate a replication. There's just a bunch of capabilities that when you add S3 APIs on top of object storage, it eliminates a lot of the things that we have to do on a day to day basis to keep our storage safe and secure and resilient.”
What does the future of cloud storage look like?
Brent shared that he believes cloud storage is about to change. “I see a big disruption in the future, just with this idea of decentralized object storage. The major cloud providers would like to call their object storage decentralized, but ultimately it's in a region or it's in a data center or it's in an availability zone. There are very filtered ways of getting information in and out of those from an archaic topology point of view.”
In centralized cloud storage, you’re sending or retrieving a serial stream of data into a cloud provider and your upload or download can only be as fast as that serial stream of data can go. That data also isn’t split and encrypted until after it’s received by the cloud provider.
In contrast, using distributed cloud storage you can optimize the performance and security of this process. If you are writing into Storj, you can optimize that by actually pushing to many different nodes at once. And you can have the storage client on the host side, actually chunk that and encrypt that data before sending it across the wire. Brent commented, “I think that's very interesting because if you're really looking to do end-to-end encryption of data, you want to make sure that it leaves your environment encrypted. Additionally, it leaves your environment already duplicated and resilient because it's being put to multiple different hosts at once.
“You get your resilience benefit earlier in the transaction cycle. You get performance benefits because you get to write to multiple locations at once. And you get better security because you're encrypting on the host side rather than waiting to encrypt on the provider side.”
To watch the full webinar replay, click here.
