We’ve published several blog posts about our security model and access control layer and about how those tools can be used to strengthen privacy and security by pushing access management to the edge. At a high level, the value proposition is clear. Still, as with any new and disruptive technology, the key to maximizing the potential value of that technology is understanding how that technology can be practically applied to solve real-world problems.
Once we cover the basics, we’ll cover specific use case examples in subsequent posts.
But First, a Word on Information Architecture...
Before diving into the example, it’s important to understand the constructs of the Storj DCS platform so that an application storing data is optimized depending on the requirements for privacy, access control, sharing, etc. The main constructs to understand are shown in the figure below:
Satellite
The Satellite is a set of hosted services responsible for a range of functions on the network, including the Node discovery system, Node address information caching, per-object metadata storage, Storage Node reputation management, billing data aggregation, storage node payment, data audit, and repair, as well as user account and authorization management.
You'll create an account on a Satellite. We have them all over the world. You choose a Satellite-based on where your data will be most frequently accessed, as Satellites are where metadata is stored and node selection takes place.
Developer Account
When you create an account on a Satellite, you add some basic contact information, including a payment method if you want to use the Paid Tier Service. You can create Projects and Access Grants/Gateway Credentials, view invoices, and track usage.
You can invite other developers with accounts on your Satellite to join one or more of your Projects, or be added to other Developers' Projects.
Project
A project is the basic unit for aggregating usage, calculating billing, invoicing fees, and collecting payment. Users can create multiple projects, and projects are invoiced separately. Within a project, usage is tracked at the bucket level and aggregated for invoicing to the project. Project names are not client-side encrypted so that they may be rendered in the Satellite user interface. There are two main drivers for creating multiple projects: access management and billing.
Access Management
For access management, API keys are instantiated at the project level. A primary API key created in the Satellite admin console can perform any action on any bucket in a project, but API Keys do not work across projects. If you are a managed service provider or you have multiple applications where there must be no commonality between applications for access management (no single API key can be used to manage data across applications), then you should create a separate project per application or customer.
The key distinction is that you can create granular Access Grants within a Project with restricted access to only a single object or path; it’s also possible to create an Access Grant with all access to all buckets, paths, and objects within a Project. It is impossible to create an Access Grant with access to buckets, paths, and objects within more than one Project.
Billing
From a billing perspective, if you only have one application, or you’re an individual using an app like FileZilla or Duplicati, you probably only need one project. If you are a managed service provider or you have multiple applications, and you want separate invoices for each of the applications, you’ll want multiple projects. Usage is itemized within projects at the bucket level, but projects have separate invoices.
Projects are also useful for managing phases of software development across environments. You may want to use a separate project for development, staging, and production environments.
Bucket
A bucket is an unbounded but named collection of objects identified by paths. Every object has a unique path within a bucket. Bucket names are not client-side encrypted, so that they may be rendered in the Satellite user interface.
Usage is tracked and itemized on invoices at the Bucket level. One practical consideration when choosing how to map constructs in your application to constructs on Storj DCS is that large numbers of buckets will increase the size and complexity of your invoices.
From an access management perspective, there’s really no difference between using a bucket and using a top-level path within a bucket. The only differences are that bucket names are unencrypted and appear on invoices while top-level path names are encrypted, and billing is aggregated at the bucket level.
If you are an individual user with a single application, you may only need one Bucket. If you are building a multi-tenant or multi-user application, the best practice is to use a single bucket for the application, then create a separate top-level path to store object data associated with each tenant or user within the application. Each top-level path can be secured with a separate Restricted Access Grant.
With this structure, your application can manage the data for all of the tenants or users of your application, but 100% of the objects and object metadata will be encrypted, and the tenants or users of your application will not have any access to the data of their peers unless they specifically authorize that access.
Object Key or Path Prefix (encrypted metadata)
An object key (or path) is a unique identifier for an object within a bucket. An object key is an arbitrary string of bytes. Object keys resemble file system paths by containing forward slashes at access control boundaries. Forward slashes (referred to as the path separator) separate path components. An example path might be "videos/carlsagan/gloriousdawn.mp4", where the path components are videos, "carlsagan", and "gloriousdawn.mp4". It is possible to share just the encryption keys for objects with a common object key path component prefix.
Object (encrypted metadata)
An object is the main data type in our system. An object is referred to by an object key, contains an arbitrary amount of bytes, and has no minimum or maximum size. An object is represented by an ordered collection of one or more segments. Segments have a fixed maximum size. An object also supports a limited amount of key/value user-defined fields called user-defined metadata.
Access Grant
Storj DCS uses hierarchically deterministic Access Grants as an access management layer for objects. An Access Grant is a security envelope that contains a Satellite address, a restricted API Key, and a restricted path component prefix-based encryption key—everything an application needs to locate an object on the network, access that object, and decrypt it. The key benefit of this approach is that these Access Grants and any associated restrictions can be entirely managed client-side, without a central Access Control List or other server-side mechanism involved in the access management process. We call this delegated authorization.
The most important thing to understand about Access Grants is, even though an Access Grant contains a serialized encryption key encapsulated in the Access Grant, that encryption key is NEVER passed to a Satellite. Storj DCS Uplink clients separate the API Key from the Access Grant and only pass the API Key to Satellites. That way, a Satellite can receive an access request, evaluate the validity of the API Key associated with that request, and respond with the metadata needed to retrieve the pieces of the object associated with that request without having any ability to decrypt the underlying data or metadata or have any information about the context of the user or application making the request.
The result is that the Storj DCS service allows you to create more private and secure applications. Below is a brief description of the three components within an Access Grant:
Satellite Address
The Satellite Address is contained within the Access Grant so that an Uplink client knows which Satellite to contact to retrieve the metadata associated with an object to be retrieved.
API Key
An Access Grant contains an API Key based on a type of bearer token called a Macaroon. API Keys are both hierarchically derived from a parent and may also contain programmatic restrictions based on:
- Specific operations: Caveats can restrict whether an Access Grant can permit any of the following operations: Read, Write, Delete, List.
- Bucket: Caveats can restrict whether an Access Grant can permit operations on one or more Buckets.
- Object key and path prefix: Caveats can restrict whether an Access Grant can permit operations on Objects that share a common path component prefix or even just one specific object.
- Time window: Caveats can restrict when an Access Grant can permit operations on objects stored on the platform.
Restrictions applied to an API Key within an Access Grant are hierarchically derived from the access restrictions contained in the API Key within the Parent Access Grant from which the Restricted Access Grant was created. That means a Restricted Access Grant can have the same level of access or less access than its Parent Access Grant, but never more access.
Encryption Key
All data stored on Storj DCS is encrypted. By using hierarchically derived encryption keys, it becomes easy to share the ability to decrypt a single object or set of objects without sharing the private encryption passphrase or having to re-encrypt objects. When you create a Primary Access Grant, you provide an encryption passphrase. All Restricted Access Grants derived from that Primary Access Grant contain a path-based hierarchically derived serialized encryption key (or set of encryption keys, based on what has been shared).
If you're interested in learning more details, please read more about Encryption at your leisure.
A unique encryption key can be derived client-side for each object. That unique key is generated automatically when sharing objects, allowing users to share single objects or object key prefixes, with the ability to decrypt just the objects that are shared, without having to worry about separately managing encryption access to objects that aren’t being shared.
But, your keys are your data. If you lose the encryption passphrase, you effectively lose the ability to recover your data. Satellites never have access to your encryption keys. Satellites can’t access your data, so your privacy is ensured, but if you lose the key, that means we also can’t help you recover it. The point: don’t lose your encryption key.
Primary Access Grants
Primary Access Grants are generated via the Satellite Admin Console. Note that only the API Key and Satellite address are generated by the Satellite. The actual Access Grant is created in the browser with the Encryption Passphrase provided by you. The Satellite does not store or retain the encryption passphrase or serialized encryption key encapsulated in the Access Grant. That is entirely handled in client-side Javascript in the browser. A Primary Access Grant can be created with no restrictions or created with one or more caveats.
A Primary Access Grant can then be imported into an Uplink Client for use. The Uplink Client can then be used to create Restricted Access Grants derived from the Primary Access Key.
Restricted Access Grants
The Storj DCS service generates primary Access Grants, and restricted Access Grants are derived from a primary Access Grant or any other Access Grant via the Uplink Client. Parent-to-child access may be further restricted, but not expanded. A restricted Access Grant can never have more access than its parent.
A primary Access Grant is created in the admin console of the Satellite. A primary Access Grant has all permissions to all buckets within a project and can create a child Access Grant.
A restricted Access Grant may be created using the Satellite console or Uplink client. Restricted Access Grants are created in the context of creating an access. Essentially, you don’t explicitly create a Restricted Access Grant. The Uplink client creates one when you generate an access, which handles both access management and encryption, both restricted to the scope of access being shared.
The Storj DCS service also supports the revocation of Access Grants. Note that revoking an Access Grant adds that Access Grant to a revocation list and invalidates the Access Grant and any child Access Grant derived from the Access Grant that has been revoked. Revoking a primary access grant can be done in the UI, but currently, revoking a restricted access grant can only happen via the Uplink CLI.
Sharing access with Restricted Access Grants
Sharing access to objects stored on Storj DCS requires sending encryption and authorization information about that object from one client to another. The information is sent in a construct called a Restricted Access Grant. As noted above, an Access Grant is a security envelope that contains a satellite address, a restricted API Key, and a restricted path-based encryption key—everything an application needs to locate an object on the network, access that object, and decrypt it.
An Access contains a bearer token that is generated client-side and transmitted client-to-client. When the Uplink client generates or uses an Access, only the API Key (the bearer token) from the Access is sent to a Satellite to retrieve an object. The encryption key is retained by the Client and used to decrypt objects, metadata, and paths client-side.
To implement these constructs as easy as possible for developers, the Storj DCS developer tools abstract the complexity of encoding objects for access management and encryption/decryption. A simple share command encapsulates the Satellite address for an object’s metadata, an encryption key, and an API Key into an Access in the format of an encoded string that can be easily imported into an Uplink client. Imported Accesses are managed client-side and may be leveraged in applications via the Uplink client library.
Applying the Storj DCS information architecture
Once you understand the basic building blocks of the platform, it’s pretty easy to build some fairly sophisticated privacy and security controls into your application. In the next blog in this series, we’ll cover how one of our partners has used these tools to enable secure and private file sharing via a native integration with their file transfer service.
