We just published a blog post outlining our official stance on helping our employees, Node Operators, and community members located in Ukraine. We share the hopes of the world that the situation is swiftly and peacefully resolved, and that the rights to peace, safety, and self-determination are preserved for the Ukrainian people.
This blog post deals with another aspect of this situation: the possibility that Russia may soon be disconnected from the internet, impacting Russian Storage Node Operators, customers, and the Storj Network as a whole. In addition, there is a possibility that international sanctions could be imposed, making it illegal for us to pay Russian Node Operators.
- On the one hand, we want to treat our Russian Node Operators and users with fairness and respect. We recognize that individual Node Operators and users are not responsible for the decisions of their government, which have received nearly universal, international condemnation.
- On the other hand, we want to make sure that we treat all of our users and customers well, and protect them from even the slightest possibility of data loss.
As you may know, we are quite proud of the fact that Storj has not lost a single file since before going into beta with our V3 network three years ago. The mathematics that make this possible are simple. Each segment is both encrypted and erasure-coded, using a 29/80 erasure code schema. This means that, initially, 80 pieces of any segment are created, of which any 29 are sufficient to put it back together. Those initial 80 pieces are broadly distributed across our network. We constantly keep track of how many pieces of any segment are around, and if any segment drops below some crucial threshold, we repair the segment by collecting the necessary number of pieces and recreating all 80 erasure shares. [Initially, this limit was set at 52, meaning that we had a buffer of 23 pieces.]
This system has worked incredibly well to date. Because the initial 80 pieces are distributed to storage nodes operated by different people, on different equipment, in different geographies, on different networks, on different power supplies, the system is highly resilient against things like drive failures, power outages, equipment wide bugs, and viruses, fires, floods, earthquakes, data center failures, etc. Power outages or storms that have impacted broad swaths of Europe or North America, for example, haven’t impacted our durability.
However, we are now facing the possibility of suddenly and permanently losing all data on all nodes operated in the most populous country in Europe and the ninth most populous in the world, a country that spans one-eighth of Earth's inhabitable landmass and that stretches across eleven time zones.
We’ve modeled out the impact of suddenly losing all Russian nodes simultaneously. The good news is that the chances that this impacts any individual customer or file are minuscule. However, given the large number of files on our system, the chances that some file somewhere would be impacted is greater than we would like.
So, we are taking some common-sense steps:
- For Satellites we operate, we are raising our repair threshold across the board. Starting four weeks ago, we raised that limit to 56 (i.e., if any segment dropped below 56 pieces, it was added to the repair queue). We have now further raised that limit to 60, meaning that the least healthy segment in our network will have more than double the number of pieces it needs to recover.
- We have identified the segments that are at most risk from a sudden loss of all nodes in or near Russia, and are prioritizing their repair. These are the segments that are closest to the repair threshold and have a statistically larger than normal concentration of pieces on Russian nodes.
- For customer-facing Satellites we operate, we are temporarily halting putting new customer data on Russian nodes. However, as soon as we have finished repairing all segments with 60 or fewer pieces, we will resume storing new data on Russian nodes.
But, in order to be as fair as possible to the Russian Node Operators and customers
- In the event that Russia is disconnected from the Internet, we will preserve the data stored by Russian users and customers, even if they are no longer able to pay. All Russian customers of our service have data distributed widely across both Russian and non-Russian nodes. So, their data should be preserved, even if disconnection from the Internet might make it more difficult for them to access their data.
- For as long as it is legal to do so, we are continuing to pay Russian Node Operators for the data that they do hold.
- We will not prematurely delete data or disqualify Russian Node Operators simply because of the possibility of Russia disconnecting–we will simply create more erasure shares to be placed throughout the network to protect our users in case the Russian nodes suddenly and simultaneously go offline.
- We will take all necessary steps to ensure that Russian Node Operators are not punished economically (provided that sanctions are not imposed).
- In the event that Russian nodes go offline temporarily due to government action, we are investigating mechanisms to allow them to gracefully rejoin Satellites we operate without being unduly penalized for circumstances beyond their control.
- In the event that sanctions are imposed, we have built systems that will allow us to track and hold payments in escrow so that we can make Russian Node Operators economically whole as soon as it is legal for us to do so.
Like all people, we regret that this situation has arisen and hope that peace can return to the region. We firmly support our team members in Ukraine and everyone fighting for Ukraine, including those Russian and Belarusian people who are doing their best to advocate ending this unjustified war against Ukraine.
If anything, this situation has helped us further battle test the decentralized internet, making it even more resilient against black swan events. We do believe that freedom and decentralization of information is one of the best defenses the world has for the preservation of democratic values.
