Encrypting Cluster Traffic

You can enrypt the runtime traffic in your cluster using Internet Protocol Security (IPsec). This section describes how to enable IPsec encryption for your cluster, as well as considerations for using cluster encryption.

Using cluster encryption

Apcera supports the use of the Internet Protocol Security (IPsec) to encrypt cluster traffic. IPsec provides IP packet-level encryption between two IP endpoints (the source and destination). Apcera uses the OpenSWAN IPsec implementation.

When IPsec is enabled for your cluster, all runtime communications are encrypted. This includes traffic between cluster components, such as the Job Manager, Package Manager, and Instance Manager, both TCP and HTTP traffic. It is a mesh network, so there will be encrypted tunnels between all nodes in the cluster, Central to IM, Central to Singleton, Singleton to IM, etc.

IPsec encryption only applies at cluster runtime, not during cluster creation or deployment. This means that communication traffic between the Orchestrator Server and any cluster host or component is not encrypted.

IPsec encryption is enabled during deployment of the Apcera release. Once enabled, encryption exists only between cluster hosts/components but not with Orchestrator. Any traffic between the Orchestrator host and the other Apcera cluster components are unencrypted.

In addition, the following types of traffic are not encrypted:

Performance considerations

There is an inherent tension between security and performance. If you enable IPsec encryption for your cluster, plan for a noticeable performance impact dependent on the specific platform, hardware acceleration, etc.

Creating a job is not that resource-intensive and you may not notice a performance hit here. However, if there is a lot of traffic between jobs (job links, virtual networks), you are likely to notice the performance hit.

If you are deploying a hybrid cluster across two or more providers (such as on AWS with remote IMs on OpenStack) and using a VPN connection to bridge the hosts, traffic flowing across the VPN will be double-encrypted and you will incur a double performance hit.

Enabling cluster encryption

To enable IPsec encryption for your cluster you set the chef.continuum.ipsec_mesh_enabled key in your cluster.conf file to true and the ipsec_mesh_preshared_key to a random string of your choosing to generate your own pre-shared key (see Generating the Pre-shared Key).

To enable cluster encryption:

  1. In your cluster.conf file, set the chef.continuum.ipsec_mesh_enabled key to true and the ipsec_mesh_preshared_key to a random string of your choosing to generate your own pre-shared key, for example: 
    chef: {
  continuum: {
 ...
 "ipsec_mesh_enabled": true,
 "ipsec_mesh_preshared_key": abcdefghigklmnopqrstuvwxyz1234,
 ...
  }
}
  2. Redeploy your cluster with the modified cluster.conf file. 
    orchestrator-cli deploy -c cluster.conf --update-latest-release
  3. If the deployment is an update to an existing Apcera cluster you must also reboot each cluster host and restart all jobs.

Generating the pre-shared key

The pre-shared key you assign to the ipsec_mesh_preshared_key parameter is used to authenticate IPsec using ISAKMP. Once authenticated the IPsec protocol does dynamic key session allocation. The pre-shared key is not used for cluster encryption.

Apcera strongly recommends that the pre-shared key value contain a mixture of alphanumeric characters and be at least 30 characters in length. You must define your own key. Do NOT copy/paste the pre-shared key value we provide in the example. You should use a random string generator to create the pre-shared key, and encrypt and version control your cluster.conf file. See Considerations about IPsec Pre-Shared Keys for best practices concerning generating and maintaining pre-shared keys.

Reboot and job restart required

If you are updating an existing cluster to enable IPsec encryption, or removing (disabling) IPsec encryption, you must reboot each cluster host following the cluster host reboot procedure. During a normal reboot process, each Instance Manager (IM) host will experience a downtime of approximately 1 to 2 minutes during which job instances will not be available.

Note that after performing an upgrade/downgrade of IPsec, the system will automatically restart all job instances (containers). Job restart is required and performed automatically so that IPsec gets enabled for each job. Note that it may take up to a minute for the job restart to complete.

You do not need to reboot cluster hosts if you are deploying a new cluster for the first time that enables IPsec encryption.

Firewall rule requirements

All components except for the Orchestrator host must allow the following connections across all private networks components are assigned to:

  • Allow UDP from private_networks ports 1023-65535 to private_networks port 500
  • Allow UDP from private_networks ports 1023-65535 to private_networks port 4500
  • Allow ESP(50) from private_networks to private_networks
  • Allow AH(51) from private_networks to private_networks

ESP and AH are connectionless protocols like UDP. Tunnels must be allowed in cross-mesh.

These rules must be modified to fit your firewall environment. For example, in AWS with security groups, each security group would need an ingress rule and an egress rule for each of the above rules. In a standard cluster having "private_networks" ("10.0.0.0/8", for example), unless otherwise covered by a more general rule, some example egress rules could be "allow UDP ports ALL destination 0.0.0.0/0" or "allow protocol ALL destination 0.0.0.0/0" (which is more general than all four rules above).

© 2022 Apcera Inc. Last updated: 2017-09-13 21:51:51 +0200 Feedback