Learn how to use Orchesto as a multi-point object storage gateway compatible with the Amazon S3 API.
Table of contents
Orchesto is an object storage server that can act as a forward proxy against one or more cloud storage providers. In particular, you can combine any number of buckets from multiple providers in a single namespace.
To deploy storage resources in Orchesto, you begin by making a set of storage regions available through one or more storage backends. This will typically require you to provide Orchesto with security credentials when adding the storage backend.
Once those regions are available to Orchesto, they will be also be indirectly available to AWS S3 compatible clients that connect to Orchesto. Specifically, for each underlying storage region, or backing region, we say that Orchesto provides a corresponding virtual region to clients.
You can then start to work with the available storage resouces by adding buckets, and upload or download objects to them.
Note that virtual regions do not automatically grant access to buckets in the underlying storage region. When adding a bucket to Orchesto, you have the choice to either create a new bucket, or re-use a existing bucket, in the backing region.
We refer to a bucket available in a virtual region, and its corresponding backing region, as a virtual bucket and backing bucket respectively.
All data uploaded to Orchesto is stored in the native format of the storage provider, ensuring the data lifecycle is not obstructed across storage environments. This allows direct access to the data outside Orchesto, and access to existing data through Orchesto, at the same time.
To enable storage services, you must first start Orchesto. This can be done by executing the application in a terminal, whereby you will see the gateway endpoints and administrative security credential to use when connecting to it. For example:
$ orchesto Orchesto running and accessible at: http://10.150.255.247:9090 http://127.0.0.1:9090 Access Key: SZL1M22877ZQTPB1ZDLE Secret Key: 9cvPYwaD3J6srQb7Dbbfeg5KTBVf9g9OdwCtcff3 -------------------------------------------------------- We see that you're not using SSL. For more information, see how to setup SSL for Orchesto: https://docs.orchesto.io
To use Orchesto the first time, you need to explicitly accept the End-User License Agreement (EULA) in one of two ways:
- set the
ACCEPT_EULAenvironment variable to
- use the
--accept-eulacommand line option
This option is not required if you have already accepted the EULA.
When started, there are two ways you can connect to Orchesto:
- Storage Service - Amazon S3 API compatible object storage
- Management Console - browser-based user interface to manage Orchesto
The Management Console can be accessed using a web browser to configure and monitor the Orchesto application. To use the Storage Service, you will need a AWS S3 compatible client instead. Either service can be accessed via any of the available endpoints.
A security credential is provided by way of access keys, consisting of an access key ID and a secret access key. The administrative security credential is unique and can be used to manage Orchesto with admin privileges, that is, without restriction.
With admin privileges, you can create new security credentials with user privileges, also known as user security credentials, to secure your storage environment. The difference across privilege levels is outlined below:
|Manage security credentials||✓||✗|
|Manage storage backends||✓||✗|
|Manage virtual regions||✓||✗|
|Use the S3 API||✓||✓|
With user security credentials, you can limit and revoke access to Orchesto as needed. As a security best practice, do not use your administrative security credential to access the S3 API.
To enable a storage gateway against any combination of storage providers, Orchesto features namespace federation with both virtual regions and buckets.
Virtual Region Namespace
With admin privileges, you have the option to choose virtual region names before, or after, a new storage backend is added to Orchesto. This can be useful to address deployment requirements of your organization.
By default, Orchesto will choose the same name as the backing region, unless this conflicts with the existing virtual region namespace.
Virtual Bucket Namespace
When using the S3 API to create a bucket, Orchesto will always choose the same for the corresponding backing bucket. This will result in a BucketAlreadyExists error if a backing bucket with the same name already exists.
With admin privileges, you have the option of adding a virtual bucket against an existing backing bucket and, moreover, choose a different virtual bucket name. This may help address, for example, a conflict with the existing virtual (or backing) bucket namespace.
Given a virtual bucket, you can choose to overlay its backing bucket with another backing bucket. This allows objects from separate backing buckets, in separate storage backends, to be transparently overlaid, and thereby form a single coherent object namespace.
With bucket overlaying, the new backing bucket gets priority over the overlayed backing bucket. We refer to these as the primary backing bucket and secondary backing bucket respectively.
The secondary backing bucket is automatically read-delete-only, and all create-object operations are directed to the primary backing bucket. If two objects share the same key across the primary and secondary backing buckets, the object in the primary backing bucket is prioritized.
Orchesto provides support to three different types of storage providers:
- Preferred Cloud Storage Providers
- User-Defined Amazon S3 API Compatible Solution
This enables a number of deployment configurations in order to accommodate different types of use cases.
Special support is given to select cloud storage providers to ensure a consistent unified object model for storage clients. These preferred cloud storage providers include:
- Amazon Web Services (Amazon S3)
- Exoscale (Object Storage)
- Digital Ocean (Spaces)
- DreamHost (DreamObjects)
- Alibaba Cloud (OSS: Object Storage Service)
- Google Cloud (GCS: Google Cloud Storage)
- Microsoft Azure (Blob Storage)
Other solutions compatible with industry standard Amazon S3 API can be connected as well. For these, you will need to provide additional information to describe the object storage environment.
To connect the filesystem to Orchesto, you simply select a path and define a new region name for it. Directories in the target path become backing buckets, with their corresponding files treated as objects.
Providing limited access to the host filesystem is a security-sensitive operation. To reduce the attack surface in your storage environment, lock down access to new filesystem backends via the management console, or using the
--fs-lockdown option when starting Orchesto:
$ orchesto --fs-lockdown
To disable the lock down on new filesystem backends, use the inverse