Working with CLI
The following section showcases some basic ways you can use Orchesto via the Command Line Interface (CLI) to manage data. In the examples, we will use Wasabi and Digital Ocean as our storage providers.
Launch Orchesto as described in Chapter 1 Getting Started or the Installation Guide. When launching Orchesto from a terminal, you will see the endpoints and, provided that it is the first time launch, the administrative security credentials to use when connecting to it. For example:
Use one of the gateway endpoints in your web browser to access the management console, and log in with the account credentials. We will in the following simple demonstration use both the web management console and the CLI to perform actions.
In order to drive Orchesto via the CLI, you will need to have access to the AWS Command Line Interface. Please visit https://aws.amazon.com/cli/ for installation and configure as shown below, making sure to replace the keys with your own Orchesto administrative security credentials.
$ aws configure --profile orchesto AWS Access Key ID [None]: AIKAA9GBRIXP4P7UGJ1H AWS Secret Access Key [None]: hbeRB/4@7p2ps7ixDtl0PTcy17p@Ug73TpK0Duib Default region name [None]: Default output format [None]:
When using the AWS CLI, we need to include the
--endpoint-url option to indicate the gateway endpoint and the
--profile option to indicate your security credential.
alias cloud='aws s3 --endpoint-url http://127.0.0.1:9090 --profile orchesto'
At this point, Orchesto is online and available to the AWS CLI client, but you need to define data sources in order to work with your data. We define data sources by adding storage providers to Orchesto.
Adding Storage Providers
Add storage providers to Orchesto, starting with Wasabi. Click on the Backends menu in the sidebar (found on the left-hand side of the management console). Following this, click on the Add Backend button to configure a new backend in Orchesto. In the Provider field, select Wasabi and use your Amazon S3 keys in the Access Key and Secret Key fields. Click the Add button to add the new backend.
When adding the first storage provider, Orchesto will automatically choose a default region. To see the list of regions available via Orchesto, visit the Dashboard view by clicking Dashboard in the sidebar and following this click on the current default region. A modal will now open with viewing and editing capabilities. The default region will be the target for creation of new buckets via CLI, unless an explicit reference to another region is being made.
Now let's add Digital Ocean as a storage provider. Again, click on the Add Backend button, select Digital Ocean in the Provider field, and use your Digital Ocean keys in the credential fields as above.
With these storage providers added to Orchesto, you have a set of data sources available at your disposal, via both Wasabi and Digital Ocean regions. You can now create new buckets in order to work with your data.
Working with Multi-Cloud Storage
Create buckets using the AWS CLI via the
cloud alias we defined earlier (which will ensure the necessary options are used). Here, we ask Orchesto to create the bucket
demo-falidae for us:
$ cloud mb s3://demo-felidae --region nyc3
Behind the scenes, Orchesto will ask the storage provider to create the corresponding bucket on behalf of the user, and then register this bucket as an available resource in its own configuration. We use the term virtual bucket to refer to this configuration.
In this case we made an explicit reference to a specific region - nyc3 - and our bucket was therefore created in the nyc3 region, provided by Digital Ocean. You can use this bucket just like any other S3 bucket, for example:
$ cloud cp alfons.jpg s3://demo-felidae $ cloud cp sandy.jpg s3://demo-felidae $ cloud cp kublai13.jpg s3://demo-felidae
Orchesto will ensure all uploaded files are stored in the backend on behalf of the client. Listing the available objects via Orchesto, the management console, or exploring the actual bucket in the backend will yield the same result:
$ cloud ls demo-felidae 2019-11-16 16:08:43 36010 alfons.jpg 2019-11-16 16:09:28 53658 kublai13.jpg 2019-11-16 16:08:56 296841 sandy.jpg
To create a bucket in a different region, and this time also against a different backend, use the AWS CLI with a virtual region mapping to Wasabi. For example, we create the bucket
demo-other-images in the Wasabi region us-east-1 data center by:
$ cloud mb s3://demo-other-images --region orchesto-us-east-1
Bucket names are globally unique in Amazon S3, such that, you may not be able to use a name if it is already taken by another Amazon S3 user. The same criteria also holds for other storage providers. In our example above, Wasabi's region us-east-1 was mapped against Orchesto's virtual region orchestra-us-east-1. The region modal with mapping information can always be reached in the Dashboard view.
You can see the list of all buckets available via Orchesto by clicking on the Buckets menu in the sidebar.
The corresponding list (without the virtual bucket metadata) can be obtained via the Amazon S3 interface available in Orchesto. For example, using the AWS CLI:
$ cloud ls 2019-11-16 16:06:10 demo-felidae 2019-11-16 16:20:22 demo-other-images 2019-11-13 15:18:55 hkwasabi 2019-11-06 14:51:09 my-sthlm-bucket1 2019-11-05 17:03:28 test123 2019-11-07 16:16:43 test20191107
By managing the multi-cloud setup within Orchesto, the inherit complexity of supporting multiple cloud storage systems is hidden from the client.
Working with On-Premise Filesystems
Using on-premise filesystems can be an easy way to bring in data sources local to Orchesto. In this example, we will build on our multi-cloud setup by adding a filesystem as an additional storage provider.
Filesystem as a Storage Provider
Access the Backends view and click on the Add Backend button to configure a new backend. In the Provider field, select File System, input the location of your volume in the Path field, and assign it a region name in the Region field. Click the Add button to add the new backend. For this exercise, we will use the path "/Volumes/datapool" and the region name "office-stockholm-1".
This new backend allows both existing files to be available as objects, and new objects to be stored as files conversely. Combined with the previously configured storage providers, we now have a setup that enable access to data in the cloud and on-premise via the same Amazon S3 interface provided by Orchesto.
Make sure to lock down access to new filesystem backends after you have finished this setup to reduce the attack surface in your environment. In the System view in the sidebar, toggle the switch File System Lockdown to on.
Alternatively, use the
--fs-lockdown option when starting Orchesto.
Storing Data Locally
To have data stored in the filesystem backend, start by creating a bucket in the corresponding region:
$ cloud mb s3://demo-hominidae --region office-stockholm-1
You can then use this bucket using the standard Amazon S3 interface accordingly. For example, we upload the file
steam.dmg, found in the current directory, to the
demo-hominidae bucket, like so:
$ cloud cp steam.dmg s3://demo-hominidae/
The newly created object will also be available as a regular file in the target volume:
$ find /Volumes/datapool -name ".*" -prune -o -print /Volumes/datapool /Volumes/datapool/demo-hominidae-9cfc801e9928a92f /Volumes/datapool/demo-hominidae-9cfc801e9928a92f/steam.dmg
Orchesto creates a unique name for buckets, and folders in case of filesystem backend, by suffixing a unique identifier to the bucket name. This is required in order to ensure that no naming conflicts arise.