Secret Store Tasks

This section provides tasks for managing cluster secrets using the secret store.

Security Overview

Apcera release 3.0.0 introduces the cluster secret store (Vault running on Consul backend) as the security foundation for the platform. Orchestrator version 2.x is enhanced to support the management of Vault and is the tool to manage your customer-owned secrets. A cluster passphrase is required for Vault to deploy and manage clusters. In addition, secrets previously stored in cluster.conf can now be stored securely in Vault.

Refer to the cluster secret store documentation for additional details.

Non-secure cluster deployment


orchestrator-cli deploy -c cluster.conf --release-bundle release-3.0.0.tar.gz --nopass [--dry]

You can use the --nopass flag to bypass secret store security. In this case the default cluster passphrase is used, and you will not be prompted for additional inputs to complete the deployment (--nopass assumes that --non-interative is true).

This workflow is identical to prior Apcera releases, with the exception of the --nopass flag.

Cluster deployment using prompts to collect customer secrets


orchestrator-cli deploy -c cluster.conf --release-bundle release-3.0.0.tar.gz [--dry]

Any time you run the orchestrator-cli deploy command using Orchestrator 2.x or later, without using the --nopass flag, you will be prompted for the following information:

  • Cluster Passphrase
  • Customer-owned secrets
    • Google Auth Client ID
    • Google Auth Client Secret
    • Google Auth Web Client ID

Note that if you are upgrading you will have to manually add the prompts to cluster.conf as described in the documentation.

Once you have entered data following a prompt, Orchestrator presents you with the following options:

  • Proceed
  • Back
  • Save
  • Quit
  • Explain

If you choose Save, you are prompted for the Config Passphrase. Once provided, the system generates a file named cluster.sec that is stored in your local working directory. This file contains your saved, encrypted, customer-owned secrets. When you redeploy the cluster you simply provide the Config Passphrase and the secrets are read from the cluster.sec file. Note that this is an optional user convenience.

Redeploy using saved encrypted config


orchestrator-cli deploy -c cluster.conf --release-bundle release-3.0.0.tar.gz

If you do save the prompt entries you will be prompted for:

  • Cluster Passphrase
  • Config Passphrase

You can either reuse the saved answers or leave them empty and prompt for new values.

Provision of cluster and config passphrases using stdin


cat pwd.json | orchestrator-cli deploy -c cluster.conf --release-bundle release-3.0.0.tar.gz --input-.json [--dry]

With the above command Orchestrator will read the values of stdin. This is used for scripting a secure cluster deployment to eliminate human involvement. In this case Orchestrator reads stdin, so cat, echo, and some processes are supported. Futher, the prompts entered are not visible in the command history.

Using the Orchestrator security command


orchestrator-cli security [args]

There is a new subcommand for Orchestrator 2.x called security that is used to manage the cluster secret store and other facets of security. By default it performs the status subcommand. With the refreshtokens subcommand, it will unseal vault and refresh the app tokens in the cluster which the components use in order to connect to Vault.

Note that you will required to provide the cluster passphrase to use the security commmand.


  changepwd     - Provide new cluster passphrase.
  unseal        - Unseal any sealed vault servers following a reboot.
  refreshtokens - Unseal any sealed vault servers and refresh the app tokens.
  reset         - Clear saved CA and vault certificates allowing deploy to recreate.
  status        - Check input passphrase and report status of vault server.

Command options

 -c, --config FILE     - The cluster configuration file to describe the settings
                         that the cluster should reflect.

 -f, --force           - Procced with reset subcommand even when errors occur in processing.

 --non-interactive     - Indicates that the user is not required to acknowledge certain
                         warning messages and permits the command to proceed (warning messages
                         are still displayed). This option implies --nopass so if the default
                         cluster passphrase should not be set then --nopass=false must
                         also be specified.

 --nopass              - Indicates that the system is using the default value as the cluster
                         passphrase. This option is a convinience for testing and demo systems,

 --input-json          - Indicates that stdin will provide a json structure with answers to prompts.
                         Supported attributes are "clusterPassphrase", "configPassphrase" and
                echo '{"clusterPassphrase":"pass@word1"}' | orchestrator-cli security --input-json ...

Using the security status command

Default usage:

orchestrator-cli security -c cluster.conf

Explicit usage:

orchestrator-cli security -c cluster.conf status

Verbose usage:

orchestrator-cli security -c cluster.conf --vv

The status flag is the default security subcommand. The status command allows for programmatic remote checking of the cluster passphrase. You will be required to provide the cluster passphrase any time you use the security command.

The --vv flag provides verbose output about the security status. Specifically, the system returns more information about the Vault server it is trying to connect to and the type of reply it is getting, including:

  • Check if Vault is initialized
  • The host IP the Vault instance is running on
  • Check if the secret store is ready

Change cluster passphrase


orchestrator-cli security changepwd -c cluster.conf

Using the changepwd subcommand prompts you to provide a new cluster passphrase that is at least 8 characters in length. The passphrase restrictions are minimal. The only requirement is a minimum of 8 characters.

Unseal Vault


orchestrator-cli security unseal -c cluster.conf

Restarting the host (Central, typically) where Vault is installed automatically unseals and restarts the Vault service (i.e. unattended restart). If you want to restart just the Vault service you'll need to first unseal it using the unseal subcommand.

Refresh platform component login tokens

orchestrator-cli security refreshtokens -c cluster.conf

Provides a way to manually renew tokens in case of expiration or compromise. Tokens expire if cluster is shut down for more than 30 days.

The command will generate a new vault token for each deployed component. Takes about 1 minute for this to occur. If you have existing tokens, the components will be able to continue serving requests.

Reset CA cert


orchestrator-cli security reset -c cluster.conf

When we set up Vault on the initial deployment of a 3.0 cluster, we set up SSL/TLS to be used between clients and Vault instances. So, the IM needs to talk to Vault, Orchestrator needs to talk to Vault, etc. This is done using SSL/TLS. Because when you go to Vault and retrieve secrets, we want the secrets to be encrypted on the wire, not delivered in plain text.

To implement SSL/TLS, we need a trusted root certificate so we can issue a server cert to Vault. Allow the trusted root cert created and distributed by first deploy to be replaced in case of compromise or expiration.

On the first deploy we create a CA trusted root cert for SSL communications. On each subsequent deploy we refresh the server cert with a new cert generated from the trusted root cert. We install the trusted root cert on each of the hosts in the cluster in the SSL certificates file which makes it trusted by any process on that host.

If one of the hosts where the trusted root CA is installed is compromised, or you simply want to update the trusted root CA cert, you can run this command to do so.

Note that this command does not generate the new trusted root CA cert, it simply removes the old one. To generate a new trusted root CA cert, you must redeploy the cluster.