Encrypting Cluster Traffic
You can enrypt the runtime traffic in your cluster using Internet Protocol Security (IPsec). This section describes how to enable IPsec encryption for your cluster, as well as considerations for using cluster encryption.
- Using cluster encryption
- Performance considerations
- Enabling cluster encryption
- Generating the pre-shared key
- Reboot and job restart required
- Firewall rule requirements
Using cluster encryption
Apcera supports the use of the Internet Protocol Security (IPsec) to encrypt cluster traffic. IPsec provides IP packet-level encryption between two IP endpoints (the source and destination). Apcera uses the OpenSWAN IPsec implementation.
When IPsec is enabled for your cluster, all runtime communications are encrypted. This includes traffic between cluster components, such as the Job Manager, Package Manager, and Instance Manager, both TCP and HTTP traffic. It is a mesh network, so there will be encrypted tunnels between all nodes in the cluster, Central to IM, Central to Singleton, Singleton to IM, etc.
IPsec encryption only applies at cluster runtime, not during cluster creation or deployment. This means that communication traffic between the Orchestrator Server and any cluster host or component is not encrypted.
IPsec encryption is enabled during deployment of the Apcera release. Once enabled, encryption exists only between cluster hosts/components but not with Orchestrator. Any traffic between the Orchestrator host and the other Apcera cluster components are unencrypted.
In addition, the following types of traffic are not encrypted:
- Address Resolution Protocol (ARP)
- Dynamic Host Configuration Protocol (DHCP)
- Bootstrap Protocol (BOOTP)
- DNS Server
- ISAKMP used to negotiate (authenticate) the IPsec exchange
Performance considerations
There is an inherent tension between security and performance. If you enable IPsec encryption for your cluster, plan for a noticeable performance impact dependent on the specific platform, hardware acceleration, etc.
Creating a job is not that resource-intensive and you may not notice a performance hit here. However, if there is a lot of traffic between jobs (job links, virtual networks), you are likely to notice the performance hit.
If you are deploying a hybrid cluster across two or more providers (such as on AWS with remote IMs on OpenStack) and using a VPN connection to bridge the hosts, traffic flowing across the VPN will be double-encrypted and you will incur a double performance hit.
Enabling cluster encryption
To enable IPsec encryption for your cluster you set the chef.continuum.ipsec_mesh_enabled
key in your cluster.conf file to true
and the ipsec_mesh_preshared_key
to a random string of your choosing to generate your own pre-shared key (see Generating the Pre-shared Key).
To enable cluster encryption:
- In your cluster.conf file, set the
chef.continuum.ipsec_mesh_enabled
key totrue
and theipsec_mesh_preshared_key
to a random string of your choosing to generate your own pre-shared key, for example:chef: { continuum: { ... "ipsec_mesh_enabled": true, "ipsec_mesh_preshared_key": abcdefghigklmnopqrstuvwxyz1234, ... } }
- Redeploy your cluster with the modified cluster.conf file.
orchestrator-cli deploy -c cluster.conf --update-latest-release
- If the deployment is an update to an existing Apcera cluster you must also reboot each cluster host and restart all jobs.
Generating the pre-shared key
The pre-shared key you assign to the ipsec_mesh_preshared_key
parameter is used to authenticate IPsec using ISAKMP. Once authenticated the IPsec protocol does dynamic key session allocation. The pre-shared key is not used for cluster encryption.
Apcera strongly recommends that the pre-shared key value contain a mixture of alphanumeric characters and be at least 30 characters in length. You must define your own key. Do NOT copy/paste the pre-shared key value we provide in the example. You should use a random string generator to create the pre-shared key, and encrypt and version control your cluster.conf file. See Considerations about IPsec Pre-Shared Keys for best practices concerning generating and maintaining pre-shared keys.
Reboot and job restart required
If you are updating an existing cluster to enable IPsec encryption, or removing (disabling) IPsec encryption, you must reboot each cluster host following the cluster host reboot procedure. During a normal reboot process, each Instance Manager (IM) host will experience a downtime of approximately 1 to 2 minutes during which job instances will not be available.
Note that after performing an upgrade/downgrade of IPsec, the system will automatically restart all job instances (containers). Job restart is required and performed automatically so that IPsec gets enabled for each job. Note that it may take up to a minute for the job restart to complete.
You do not need to reboot cluster hosts if you are deploying a new cluster for the first time that enables IPsec encryption.
Firewall rule requirements
All components except for the Orchestrator host must allow the following connections across all private networks components are assigned to:
- Allow UDP from private_networks ports 1023-65535 to private_networks port 500
- Allow UDP from private_networks ports 1023-65535 to private_networks port 4500
- Allow ESP(50) from private_networks to private_networks
- Allow AH(51) from private_networks to private_networks
ESP and AH are connectionless protocols like UDP. Tunnels must be allowed in cross-mesh.
These rules must be modified to fit your firewall environment. For example, in AWS with security groups, each security group would need an ingress rule and an egress rule for each of the above rules. In a standard cluster having "private_networks" ("10.0.0.0/8", for example), unless otherwise covered by a more general rule, some example egress rules could be "allow UDP ports ALL destination 0.0.0.0/0" or "allow protocol ALL destination 0.0.0.0/0" (which is more general than all four rules above).