This guide describes how you can use the Management Console to configure and monitor the Orchesto Data Management Solution.
The Management Console is a web-based management tool for the Orchesto server. To access it, browse to the Orchesto endpoint and log in with a valid security credential.
To manage Orchesto itself, make sure to use the administrative security credential. This will grant you admin privileges, providing access to all features in the management console.
On the left-hand side of the management console is the sidebar, which provides access to all principle management functions. At the top of the console is the topbar, where you will find buttons to toggle the sidebar, access documentation, notifications and user menu.
The dashboard, seen in the middle section of the screen shot below, provides an overview of all your storage regions and some key performance indicators at a glance.
If your organisation is making use of several Orchesto deployments, a software module called The Central can be used to get an overview of the deployments. Some key features provided by The Central are consolidated metrics, license information, governance policy definition and an Orchesto news channel. For more information, see the documentation for The Central.
To rename virtual regions and change default region, click on the current default region in the dashboard and Orchesto will open a modal to make the desired adjustments. A default region is automatically set by Orchesto when the first storage backend is added (process described below). The default region will be the target for creation of new buckets via CLI, unless an explicit region reference is being made.
Region configuration is disabled for non-admin users.
The Backends section in the sidebar is where you list and manage all available storage backends.
Here you can add and remove backends as well as get a read on some high level metrics for each backend. E.g.:
- When the backend was added
- Number of currently connected buckets to the backend
- Data transferred in (to the backend from Orchesto) and out (from the backend to Orchesto)
- Number of errors reported during command communication with the backend
To add a new storage storage backend, simply click the Add Backend button. Orchesto currently features 15 pre-integrated public cloud service providers and more are continuously added. Orchesto can also connect to custom private clouds that are based on the S3 or Swift protocol. In addition to object storage, there is also an opportunity to connect to filesystems. There are some slight deviations between the backends regarding what is needed to set up a connection. E.g., for AWS, an access key and a secret key is sufficient to give Orchesto access to all regions. For Azure however, an account name and an access key only give access to one region, which is determined when the connection is configured. When accessing the add backend modal, besides credentials, a backend ID will also be requested. This will be the Orchesto internal name for the backend. For instance, this is the name that will be displayed when adding a virtual bucket to a backend. Finally, when creating a new backend, Orchesto also provides the option to Cache added buckets by default. With this functionality activated, Orchesto will automatically place objects in the read cache provided that the cache configuration has been set up.
The Object Browser, accessed via Buckets in the sidebar, is where Orchesto provides capabilities to manage buckets and objects across all backing storage regions. The overview screen will hold information regarding the buckets, such as: name, region, creation date and information regarding whether or not bucket-level encryption, cache and versioning are enabled.
You can add new buckets to Orchesto, also known as virtual buckets, by clicking on the Add Virtual Bucket button. In order to add a new virtual bucket, the following needs to be defined:
- Backend or ZIDA: Select backend if you wish to map the virtual bucket to a dedicated storage provider in a specific region. Select ZIDA if you wish to map the virtual bucket to a composite ZIDA region. (In its definition, a ZIDA region will hold a cluster of other regions that will be leveraged for redundancy purposes.)
- Provider: Select backend provider for the virtual bucket. Note that if ZIDA has been selected, the Provider field is not applicable since this mapping already has been made during the definition of the ZIDA region.
- Backing Bucket: Create New is selected by default in the Orchesto 2.0 release. (Work is currently underway to support virtual buckets mapping to existing buckets, whereby the existing content automatically will be imported to Orchesto to allow for indexing and Orchesto managed functionality.)
- Name: Orchesto's internal name of the virtual bucket.
- Region: The regional placement of the backing bucket.
To see the content and manage a specific bucket, simply click on the bucket to get access to the objects and bucket-level management functionality. Specific objects can be uploaded, deleted, downloaded and viewed. On the bucket-level, Orchesto features selections and toggles that allow for:
- Versioning: Enable or Suspend versioning. This is an Orchesto feature that automatically provides versioning to all uploaded objects in a virtual bucket, even in instances where the backing bucket does not support versioning!
- Creation of bucket policies: Allows for bucket-level IAM policies. See information regarding policy definitions, please see Appendix Using policy definitions.
- Encryption: Automatically applies Gateway-Side Encryption (GSE) to all uploaded objects. When reading objects via the Orchesto management console, the APIs, or the CLI, decryption will automatically be applied. This means that any client that consumes Orchesto can consider information to be in clear text, whereas in reality it may be encrypted when it leaves Orchesto for placement in an upstream storage.
- Cache: If enabled, objects will automatically be placed in the read cache upon upload.
- Versioning view-ability: If enabled, users of the management console will be able to see the different versions of an object as individual objects in the bucket. If disabled, only the latest version of each object will be displayed in the management console.
Furthermore, it is possible to immediately in the management console view the content of an object as well as edit its meta-data. Simply click the checkbox for the specific object whereby a modal pops up that allows for viewing, provision of a signed, time-restricted URL for sharing purposes, general object information and edit of meta-data. Furthermore, selecting one or several objects via their checkboxes allows for bulk deletion or a zip-compressed bulk download.
Another unique, value-adding feature in Orchesto, is the possibility to live sync / live migrate objects from one backend provider and region to another backend provider and region. And all this without disrupting the application that feeds and consumes objects to and from Orchesto! To execute a live migration, select the bucket you wish to migrate and click the Create Sync button. This will display a modal where the new target region can be selected. The region selected will map to a specific backend provider. Hence, a migration can take place not only between regions in the same backend but also from one backend to another. The migration will delete all objects in the source bucket once it is completed.
The new region will be displayed in the bucket overview screen. New objects uploaded to the virtual bucket will be placed in the new region (and potentially also in a new backend).
A status view of live migrations can be accessed via Sync Jobs in the sidebar. This screen shows previous and ongoing migrations and provides information regarding which data sets that were targeted, source and target regions and when the jobs were executed.
Orchesto implements granular access control to identities and resources based on a subset of the AWS IAM API. To work with Orchesto IAM you can use the Orchesto management console or, for programmatic access, the IAM APIs or a command line tool.
The IAM section consists of 3 tabs - Users, Groups and Policies - and is accessed by clicking IAM in the sidebar.
The IAM section is disabled for non-admin users.
The User section allows for creation, revocation and editing of users. When creating a new user in Orchesto, an option is provided to duplicate permissions (i.e., policies) that are assigned to a pre-existing user. This "cloning" capability simplifies administration when new users with similar permissions need to be created. Furthermore, it is possible to assign specific, pre-exiting IAM policies to the user that is being created. Copying permissions and Attaching existing policies give both simplicity and flexibility during the creation process.
For management purposes it is also possible to introduce a hierarchy using policy groups. A policy group is simply a collection of policies. When creating new users whose permissions should be identical to pre-existing users, instead of copying policies during the creation process (see above), a policy group can be created. This provides better overview and also allows for changes in a group's policies to automatically be propagated to all users belonging to that group.
Policies are defined, versioned and assigned to users and groups by accessing IAM in the sidebar. A policy needs to be written in JSON format and follow strict rules. For more information regarding policies in general, as well as access to some illustrative policy samples, please see Appendix Using policy definitions.
The process of policy execution during command requests - via the Orchesto management console, the Orchesto API or the Orchesto CLI - is as follows:
When Orchesto receives the request, it completes several steps
- Objective: Determine whether to allow or deny the request.
- Authentication: Authenticates the principal that makes the request.
- Processing the Request Context: Processes the information gathered in the request to determine which policies apply to the request.
- Evaluating Policies: Evaluates all of the policy types, which affect the order in which the policies are evaluated.
- Determining Whether a Request Is Allowed or Denied: Processes the policies against the request context to determine whether the request is allowed or denied.
ZIDA is Orchesto's proprietary information dispersal algorithm. Configuration is accessed via the ZIDA button in the sidebar. In order to achieve strong and customisable dispersal, ZIDA provides parametrisation options for the user and also leverages a composite ZIDA region. The latter differs from simple, non-ZIDA regions by mapping a virtual bucket to several regions belonging to different backends. More specifically, a ZIDA region is defined by:
- Dispersal method: Selection of Replication or ZebEC. The former acts on objects by replicating them to the defined regions. The latter by applying a Zebware proprietary erasure-code algorithm that partitions the objects into shards for distributed placement in the defined regions.
- Regions: Definition of regions for Replication or ZebEC distribution. For Replication, a minimum of 2 regions are required. For ZebEC, a minimum of 3 regions are required. Each dispersal region must use a different backend.
For ZebEC, additional definitions are required:
- Shards (only required for ZebEC): Definition of object partition in terms of number of data shards and parity shards. A data shard is a slice of the payload holding original data. A parity shard is a slice of the payload holding redundant data. Hence, you can lose up to the number of parity shards and still reconstruct the object.
- Profiles: Selection of pre-defined performance profiles for ZebEC. Currently, the profiles Basic, Efficiency, Balanced and Performance are featured. The profiles make trade-offs between performance and storage overhead. General recommendation is to use Balanced.
- Basic: Minimum storage overhead. E.g., Archival, Backup. (Targeting maximum 1.5% larger parity shards than data shards.)
- Efficiency: Prioritises to reduce storage overhead over performance. E.g., Files. (Targeting maximum 10% larger parity shards than data shards.)
- Balanced: Balances storage overhead with performance. E.g., Databases. (Targeting maximum 15% larger parity shards than data shards.)
- Performance: Prioritises performance over reducing storage overhead. (Targeting maximum 25% larger parity shards than data shards.)
The Logs view is where you can monitor access and error events in the Orchesto server across all storage backends. Logging can be configured by accessing the System view in the sidebar.
Logs section is disabled for non-admin users.
The API view directs you to a new browser tab. This displays information regarding request methods as well as request and response samples for the Orchesto API.
The System view is where you manage general system aspects, such as encryption, caching, end-point access and logging.
System configuration is disabled for non-admin users.
The top section provides controls for configuration of general security and caching.
The mid section allows for configuration of the endpoints that Orchesto should be listening to.
The endpoints in the configuration can be overridden by providing a value at the time of Orchesto startup.
ORCHESTO_DSN="postgres://email@example.com:5432/postgres?sslmode=disable" ./orchesto --listen 127.0.0.1:9090
And at last, the bottom section of the System view gives access to configurations related to logging. Logs will always be available via the Logs view in the web management console. However, Orchesto can additionally be configured to write logs to a specific file as well as to the system / terminal console.
Configuration of general security and caching is vital to Orchesto and requires some elaboration.
Transport layer security (TLS) secures communication between clients and Orchesto to provide privacy and data integrity. Enabling this mode of communication requires a valid certificate. Please visit Chapter TLS for more detailed instructions on how to obtain certificates.
An additional control serving to reduce the number of attack vectors into Orchesto is File System Lockdown. Setting File System Lockdown to disabled limits access to the host filesystem since this, under some circumstances, can be considered as a security-sensitive operation. Both TLS and File System Lockdown can be controlled either at time of Orchesto launch or later via the System screen.
To further control the security, Orchesto can also lock down CLI access and be set to only respond to configurations and commands via the web interface. This is achieved by simply disabling S3 API by setting this toggle to off.
A critical security aspect of Orchesto is the support for Gateway-Side Encryption (GSE). Given its significance, this domain is more thoroughly discussed in Chapter Encryption. Before GSE can be used, Orchesto must be configured with a supported Key Management Service (KMS).
At the moment, Orchesto supports the following KMS providers which can be added either via the management console or the Orchesto API:
- Orchesto KMS
- HashiCorp Vault
For more information on how to how to set up, manage and utilise GSE, please review the discussion in Chapter Encryption.
A key functionality in Orchesto to improve performance is the read cache. Prior to requesting retrieval of an object from an upstream backend, Orchesto will query the cache to see if the object is available locally. This will both improve performance as well as reduce costs since no egress charges will be applied by the cloud service provider.
Configuration of the cache requires defining:
- Cache Drives: Pointing to the local area where the objects should be stored.
- Expiry: Maximum life length of objects. Once an object reaches the defined age, it will automatically be removed from the cache.
- Max Use: Defines the maximum percentage of the available disk partition that can be used by the cache.
- Default Backends: Objects stored in selected backends will by default be cached.
- Default Regions: Objects stored in selected regions will by default be cached.