As businesses move their data to the public cloud, one of the most pressing issues is how to keep it safe from illegal access.

Using a tool like HashiCorp Vault gives you greater control over your sensitive credentials and fulfills cloud security regulations.

In this blog, we'll walk you through HashiCorp Vault High Availability Setup.

Hashicorp Vault

Hashicorp Vault is an open-source tool that provides a secure, reliable way to store and distribute sensitive information like API keys, access tokens, passwords, etc. Vault provides high-level policy management, secret leasing, audit logging, and automatic revocation to protect this information using UI, CLI, or HTTP API.

High Availability

Vault can run in a High Availability mode to protect against outages by running multiple Vault servers. When running in HA mode, Vault servers have two additional states, i.e., active and standby. Within a Vault cluster, only a single instance will be active, handling all requests, and all standby instances redirect requests to the active instance.

Integrated Storage Raft

The Integrated Storage backend is used to maintain Vault's data. Unlike other storage backends, Integrated Storage does not operate from a single source of data. Instead, all the nodes in a Vault cluster will have a replicated copy of Vault's data. Data gets replicated across all the nodes via the Raft Consensus Algorithm.

Raft is officially supported by Hashicorp.

Architecture

Prerequisites

This setup requires Vault, Sudo access on the machines, and the below configuration to create the cluster.

• Install Vault v1.6.3+ent or later on all nodes in the Vault cluster 

In this example, we have 3 CentOs VMs provisioned using VMware. 

Setup

1. Verify the Vault version on all the nodes using the below command (in this case, we have 3 nodes node1, node2, node3):

CODE: https://gist.github.com/velotiotech/98ccfd52655426dc08000d1db2d5811c.js

2. Configure SSL certificates‍‍

‍Note: Vault should always be used with TLS in production to provide secure communication between clients and the Vault server. It requires a certificate file and key file on each Vault host.

We can generate SSL certs for the Vault Cluster on the Master and copy them on the other nodes in the cluster.

Refer to: https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engine#scenario-introduction for generating SSL certs.

• Copy tls.crt tls.key tls_ca.pem to /etc/vault.d/ssl/ 
• Change ownership to `vault`

CODE: https://gist.github.com/velotiotech/d4367d86748b09edde96d390060cba7c.js

• Copy tls* from /etc/vault.d/ssl to of the nodes

3. Configure the enterprise license. Copy license on all nodes:

CODE: https://gist.github.com/velotiotech/921f3b8556a0a9b80ff6914ade265e82.js

4. Create the storage directory for raft storage on all nodes:

CODE: https://gist.github.com/velotiotech/0539c4aad5563146b193096a86146ace.js

5. Set firewall rules on all nodes:

CODE: https://gist.github.com/velotiotech/136a3c1732bab687b055a9ee7a43d89f.js

6. Create vault configuration file on all nodes:

CODE: https://gist.github.com/velotiotech/147ddbacb2b61eb146c0c5c2ba0cbba7.js

CODE: https://gist.github.com/velotiotech/ab6c864d37e29c49b7a1d3523851a3d0.js

CODE: https://gist.github.com/velotiotech/5e9bdfcd08881758b415698a37523d55.js ‍

7. Set environment variables on all nodes:

CODE: https://gist.github.com/velotiotech/f3f75b6d1f4a7b080871d20c1bd9757a.js

8. Start Vault as a service on all nodes:

You can view the systemd unit file if interested by: 

CODE: https://gist.github.com/velotiotech/d06dfff9d28b4071c9503d617a9483b8.js

9. Check Vault status on all nodes:

CODE: https://gist.github.com/velotiotech/788b90b7d66326531e79f51e2584e6b1.js

10. Initialize Vault with the following command on vault node 1 only. Store unseal keys securely.

CODE: https://gist.github.com/ASangave/7c5cc379bc161da648e98881c2c4ea26.js

11. Set Vault token environment variable for the vault CLI command to authenticate to the server. Use the following command, replacing <initial-root- token> with the value generated in the previous step.

CODE: https://gist.github.com/velotiotech/0959280e51efad1e5dc375c444336fb2.js

12. Unseal Vault1 using the unseal key generated in step 10. Notice the Unseal Progress key-value change as you present each key. After meeting the key threshold, the status of the key value for Sealed should change from true to false.

CODE: https://gist.github.com/velotiotech/e292a4d9e23a503f22fc2bb6bc2a115d.js

13. Unseal Vault2 (Use the same unseal key generated in step 10 for Vault1):

CODE: https://gist.github.com/velotiotech/2472add4b340d015086b00293a004c37.js

14. Unseal Vault3 (Use the same unseal key generated in step 10 for Vault1):

CODE: https://gist.github.com/velotiotech/ef0ccce55aa3f1aae1cc877cad27dafb.js

15. Check the cluster's raft status with the following command:

CODE: https://gist.github.com/ASangave/5d2670f30fa09380e3681f085a8afdc3.js

16. Currently, node1 is the active node. We can experiment to see what happens if node1 steps down from its active node duty.

In the terminal where VAULT_ADDR is set to: https://node1.int.us-west-1-dev.central.example.com, execute the step-down command.

CODE: https://gist.github.com/velotiotech/b227701fab072aedefbb8477e495c81a.js

In the terminal, where VAULT_ADDR is set to https://node2.int.us-west-1-dev.central.example.com:8200, examine the raft peer set.

CODE: https://gist.github.com/ASangave/68959954784f5b5fed9e5a31ae544e97.js

Conclusion 

Vault servers are now operational in High Availability mode, and we can test this by writing a secret from either the active or standby Vault instance and see it succeed as a test of request forwarding. Also, we can shut down the active vault instance (sudo systemctl stop vault) to simulate a system failure and see the standby instance assumes the leadership.