After the last post where we create a Consul cluster, it is time to use it as a backend for a Vault cluster.

In this lab I will guide you through deploying and configuring a Vault cluster of 3 Vault servers (HA) and we will use Consul as a Vault backend storage.

I want to point out that this steeps can be used on Vault and Consul on their open source or enterprise version

Steeps:

  • Verify the health of the Consul cluster
  • Install Vault
  • Generate and review the Vault configuration
  • Generate a self-sign certificate
  • Add a Consul ACL policy and an ACL token for Vault
  • Generate the Consul certificates
  • Setup the Vault servers
  • Initialized the Vault servers
  • Unseal the Vault Servers

For this lab, you should completed

Infrastructure:

You can use any cloud instance, virtual or physical computer to do the lab, HashiCorp recommends to have 3 to 5 consul servers dedicated to be the backend storage for vault, and at least 3 dedicated VM's, cloud instances o physical computers dedicated to run Vault

The instances where we going to install vault need to have the Consul agent in client mode, so if you do the last labs making a Consul cluster, you have 3 consul servers and 1 client.

You will need to add two more clients, this is highly recommend to run in production.

In this lab as Consul is not going to play a relevant role, I going to use 1 consul server and 3 consul agents where we going to install the vault cluster.

The infrastructure will be like this

Hostname Consul Role Vault Role IP
raspb-ubuntu-1 Consul Server Vault Storage Backend 192.168.1.1
raspb-ubuntu-2 Consul Agent 1 Vault Server 1 192.168.1.2
raspb-ubuntu-3 Consul Agent 2 Vault Server 2 192.168.1.3
raspb-ubuntu-4 Consul Agent 3 Vault Server 3 192.168.1.4

Hands On

Verify the health of the Consul cluster

First we need to verify the health of the Consul cluster, you can put this command in any computer

consul members
#Output
Node            Address           Status  Type    Build  Protocol  DC   Segment
raspb-ubuntu-1  192.168.1.1:8301  alive   server  1.9.2  2         dc1  <all>
raspb-ubuntu-2  192.168.1.2:8301  alive   client  1.9.2  2         dc1  <default>
raspb-ubuntu-3  192.168.1.3:8301  alive   client  1.9.2  2         dc1  <default>
raspb-ubuntu-4  192.168.1.4:8301  alive   client  1.9.2  2         dc1  <default>

if you are working with 3 consul servers maybe you want to know who is the leader, you can check this out with

consul operator raft list-peers

If you want, you can check the Consul UI in client using the port 8501 for example https://192.168.1.4:8501/

Install Vault

We need to install vault on the Consul agents, each steep is needed to execute in each Consul client.

Download the vault binaries from the official page from HashiCorp in the next link

Downloads | Vault by Hashicorp
Vault secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing. Vault handles leasing, key revocation, key rolling, auditing, and provides secrets as a service through a unified API.

Select the right package for you and right click and select copy link address

use curl to download the file and save it in /tmp/vault.zip (Consul Agents/Vault Servers)

curl -fsSL -o /tmp/vault.zip https://releases.hashicorp.com/vault/1.6.3/vault_1.6.3_linux_amd64.zip

unzip the file and copy the vault binary in /user/local/bin (Consul Agents/Vault Servers)

unzip -o -d /user/local/bin/ /tmp/vault.zip

Give de capability to Vault to block memory with cap_ipc_lock  (Consul Agents/Vault Servers)

sudo setcap cap_ipc_lock=+ep /usr/local/bin/vault

Give the capability to Vault to use privileged ports with cap_net_bind  (Consul Agents/Vault Servers)

sudo setcap cap_net_bind_service=+ep /usr/local/bin/vault

Add the vault user

sudo useradd --system --home /etc/vault.d --shell /bin/false vault

Create the directories and new configuration for vault  (Consul Agents/Vault Servers)

mkdir --parents /etc/vault.d
touch /etc/vault.d/vault.hcl
chown --recursive vault:vault /etc/vault.d
chmod 640 /etc/vault.d/vault.hcl

Now lets create the unit service for systemctl, this configuration will provide Vault with a method to work as a service and start with the system

First lets create the file  (Consul Agents/Vault Servers)

sudo nano /etc/systemd/system/vault.service

And paste this data  (Consul Agents/Vault Servers)

[Unit]
Description="Vault Server"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK CAP_NET_BIND_SERVICE
Capabilities=CAP_IPC_LOCK+ep CAP_NET_BIND_SERVICE+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK CAP_NET_BIND_SERVICE
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d
ExecReload=/bin/kill --signal HUP
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=60
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

The following parameters are set for the [Unit] stanza:

  • Description - Free-form string describing the vault service
  • Documentation - Link to the vault documentation
  • Requires - Configure a requirement dependency on the network service
  • After - Configure an ordering dependency on the network service being started before the vault service
  • ConditionFileNotEmpty - Check for a non-zero sized configuration file before vault is started
  • StartLimitIntervalSec, StartLimitBurst - Limit vault to three start attempts in 60 seconds (support for systemd version 230+)

The following parameters are set for the [Service] stanza:

The following parameters are set for the [Install] stanza:

  • WantedBy - Creates a weak dependency on vault being started by the multi-user run level

Now enable the service  (Consul Agents/Vault Servers)

systemctl daemon-reload

Generate and Review the Vault configuration

Edit the vault.hcl file  (Consul Agents/Vault Servers)

sudo nano /etc/vault.d/vault.hcl

And paste this data and save it  (Consul Agents/Vault Servers)

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_cert_file = "<VAULT_CERT_CRT>"
  tls_key_file  = "<VAULT_CERT_KEY>"
}
api_addr = "https://192.168.1.2:8200"
cluster_addr = "https://192.168.1.2:8201"
storage "consul" {
  address = "127.0.0.1:8501"
  path = "vault/"
  token = "<CONSUL_TOKEN>"
  scheme = "https"
  tls_ca_file   = "/etc/consul.d/tls/consul-agent-ca.pem"
  tls_cert_file = "<CONSUL_CERT_PEM>"
  tls_key_file  = "<CONSUL_CERT_KEY>"
}
ui = true
vaut.hcl

It makes the vault servers do the following

  • address: We bind 127.0.0.1 (localhost) to respond on port 443, you can also use a PrivateIP or 0.0.0.0 if you need to Vault respond on any private IP (more insecure)
  • tls_cer_file and tls_key_file: We going to create this certs further in the post and this allows TLS encrypted communication between the Vault servers.
  • api_addr and cluster_addr: These are the address Vault servers will use to advertise to other servers for client redirection and request forwarding respectively, each server will have his own private IP address put in, and this address needs to be reachable within the servers.
  • storage: Use Consul as the storage backend by including the consul stanza, in this stanza we include the address of the consul agent (localhost) the path we going to use for the storage and the ca pem file to communicate with consul, we also going to generate further in the post an individual cert and key file for each of the vault servers and this will be used to use TLS encrypted communication with consul.

Generate a self-sign certificate

We need to create an X509 certificate for Vault, you can obtain this from a Certificate Authority, but as I'm using a private IP block (192.168.1.0/24) I can use a self-sign certificate, create the certification configuration file (Any Vault Server)

nano selfsign.cfr

and paste this data and save it (Any Vault Server)

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = US
ST = state
L =  city
O = company
CN = *

[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
IP.1 = 192.168.1.2
IP.2 = 192.168.1.3
IP.3 = 192.168.1.4
IP.4 = 127.0.0.1

Change the IP.(1-4) for your own vault servers IP's

Maybe want to remove DNS.1 and IP.4, this will prevent Vault can respond from localhost, and you can make a cert for each vault server, in my case i want to generate a cert that I can be use in any vault server

This configuration file does the following

The configuration options are specified in the req section of the configuration file.

  • distinguished_name: This specifies the section containing the distinguished name fields to prompt for when generating a certificate or certificate request.
  • x509_extensions: This specifies the configuration file section containing a list of extensions to add to certificate generated when the -x509 switch is used.
  • prompt: If set to the value no this disables prompting of certificate fields and just takes values from the config file directly.

Create the certificate with Openssl (Any Vault Server)

openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout vault-server.key -out vault-server.crt -config selfsign.cfr -days 9999

Usually in a production server you will set -days to 365, this is up to you.

You will get an output

Generating a RSA private key
........................................................................................................+++++
.............+++++
writing new private key to 'vault-server.key'
-----

If you want to see the key structure you can use

openssl x509 -text -in vault-server.crt

Copy the cert to the other two servers

scp vault-server.crt ubuntu@192.168.1.3:/home/ubuntu/
scp vault-server.key ubuntu@192.168.1.3:/home/ubuntu/
Change your user and server IP
scp vault-server.crt ubuntu@192.168.1.4:/home/ubuntu/
scp vault-server.key ubuntu@192.168.1.4:/home/ubuntu/
Change your user and server IP

Now add the cert to your ca-bundle to do not get the "invalid CA" message (Consul Agents/Vault Servers)

cat vault-server.crt | sudo tee -a /etc/ssl/certs/ca-certificates.crt

make a tls folder in the vault.d directory (Consul Agents/Vault Servers)

sudo mkdir /etc/vault.d/tls

Move the certificates to the tls folder (Consul Agents/Vault Servers)

sudo mv vault-server.* /etc/vault.d/tls/

And give permissions to Vault over the files (Consul Agents/Vault Servers)

sudo chown -R vault:vault /etc/vault.d/tls
sudo chmod 0664 /etc/vault.d/tls/vault-server.key

Edit the vault.hcl (Consul Agents/Vault Servers)

sudo nano /etc/vault.d/vault.hcl

And change the <VAULT_CERT_CRT> and <VAULT_CERT_KEY> and put the certs we just create under the tcp stanza (Consul Agents/Vault Servers)

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_cert_file = "/etc/vault.d/tls/vault-server.crt"
  tls_key_file  = "/etc/vault.d/tls/vault-server.key"
}
api_addr = "https://192.168.1.2:8200"
cluster_addr = "https://192.168.1.2:8201"
storage "consul" {
  address = "127.0.0.1:8501"
  path = "vault/"
  token = "<CONSUL_TOKEN>"
  scheme = "https"
  tls_ca_file   = "/etc/consul.d/tls/consul-agent-ca.pem"
  tls_cert_file = "<CONSUL_CERT_PEM>"
  tls_key_file  = "<CONSUL_CERT_KEY>"
}
ui = true

Export the CERT environment variables and add to the bashrc (Consul Agents/Vault Servers)

export VAULT_CACERT=/etc/vault.d/tls/vault-server.crt
echo "export VAULT_CACERT=/etc/vault.d/tls/vault-server.crt" >> ~/.bashrc
export VAULT_CLIENT_CERT=/etc/vault.d/tls/vault-server.crt
echo "export VAULT_CLIENT_CERT=/etc/vault.d/tls/vault-server.crt" >> ~/.bashrc
export VAULT_CLIENT_KEY=/etc/vault.d/tls/vault-server.key
echo "export VAULT_CLIENT_KEY=/etc/vault.d/tls/vault-server.key" >> ~/.bashrc

Export the VAULT_ADDRESS to tell vault where it need to respond and add it to the bashrc (Consul Agents/Vault Servers)

export VAULT_ADDR=https://127.0.0.1:8200
echo "export VAULT_ADDR=https://127.0.0.1:8200" >> ~/.bashrc

Add a Consul ACL policy and get an ACL token for Vault

Lets create a policy for the vault servers can write and use the consul kv storage, create it using nano

nano vault-policy.hcl

And paste this information and save it

key_prefix "vault/" {
  policy = "write"
}
node_prefix "" {
  policy = "write"
}
service "vault" {
  policy = "write"
}
agent_prefix "" {
  policy = "write"
}
session_prefix "" {
  policy = "write"
}

This policy do the following:

  • Gives the ability to read and write data on the vault/ path in Consul's key/Value store, for Vault to store the backend.
  • The Ability to register all nodes with the Consul catalog API, register on the service discovery using the Consul Health API, and filter results for all nodes with the Consul Agent API.
  • The ability to register the "vault" service with the Consul Catalog API and monitor its health with the consul Health API.
  • The ability to write operations agains the Consul Agent in all the nodes.
  • The ability to write operations in the Consul Session API

All this privileges will give Vault what it needs to storage and manage its data.

Add the policy to your Consul server

consul acl policy create -name vault -rules @/path/to/vault-policy.hcl

Changing /path/to/ for the current path where the policy is

This will return a Success message, with the policy and rules.

Now create a new token that will be use the ACL policy

consul acl token create -description "vault token" -policy-name vault

This will return an AccessorID and a SecretID, the SecretID is the actual token please copy the token and paste it in the <CONSUL_TOKEN> in the /etc/vault.d/vault.hcl file

Your file most seem something like this

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_cert_file = "/etc/vault.d/tls/vault-server.crt"
  tls_key_file  = "/etc/vault.d/tls/vault-server.key"
}
api_addr = "https://192.168.1.2:8200"
cluster_addr = "https://192.168.1.2:8201"
storage "consul" {
  address = "127.0.0.1:8501"
  path = "vault/"
  token = "872014d2-c422-3575-354c-a57b70bc5e9f"
  scheme = "https"
  tls_ca_file   = "/etc/consul.d/tls/consul-agent-ca.pem"
  tls_cert_file = "<CONSUL_CERT_PEM>"
  tls_key_file  = "<CONSUL_CERT_KEY>"
}
ui = true

Generate the Consul certificates

Great, don't lose your mind, we are almost there, the only thing left is generate certs to communicate with Consul via TLS.

We can make this lab in separated steeps, but is best to do it in one run, because we don't want to launch vault and generate the backend storage until we have all the consul backend configuration ready.

Go to the consul tls folder in the consul server (Consul Server)

cd /etc/consul.d/tls/

And generate the tls certs for the vault clients (Consul Server)

consul tls cert create -client

You will get something like this (Consul Server)

#Output
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-0.pem
==> Saved dc1-client-consul-0-key.pem

In my case I have the vault servers in raspb-ubuntu-2, raspb-ubuntu-3 and raspb-ubuntu-4, so I'm going to execute this command 4 more times (Consul Server)

sudo consul tls cert create -client
#Output
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-1.pem
==> Saved dc1-client-consul-1-key.pem

sudo consul tls cert create -client
#Output
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-2.pem
==> Saved dc1-client-consul-2-key.pem

sudo consul tls cert create -client
#Output
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-3.pem
==> Saved dc1-client-consul-3-key.pem

sudo consul tls cert create -client
#Output
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-4.pem
==> Saved dc1-client-consul-4-key.pem

I will delete the cert I don't going to use (Consul Server)

sudo rm dc1-client-consul-0* dc1-client-consul-1*

Copy the certs to the first vault server (change IP, username and location) (Consul Server)

scp dc1-client-consul-2.pem ubuntu@192.168.1.2:/home/ubuntu/
scp dc1-client-consul-2-key.pem ubuntu@192.168.1.2:/home/ubuntu/

Copy the certs to the second vault server(change IP, username and location) (Consul Server)

scp dc1-client-consul-2.pem ubuntu@192.168.1.2:/home/ubuntu/
scp dc1-client-consul-2-key.pem ubuntu@192.168.1.2:/home/ubuntu/

Copy the certs to the third vault server(change IP, username and location) (Consul Server)

scp dc1-client-consul-3.pem ubuntu@192.168.1.3:/home/ubuntu/
scp dc1-client-consul-3-key.pem ubuntu@192.168.1.3:/home/ubuntu/

delete the cert from the Consul server (Consul Server)

sudo rm dc1-client-consul-2* dc1-client-consul-3* dc1-client-consul-4*

On the Vault servers, copy the pem files to the /etc/vault.d/tls folder and change the permissions (Consul Agents/Vault Servers)

sudo mv dc1-client-consul-* /etc/vault.d/tls
sudo chown -R vault:vault /etc/vault.d/tls

Edit the vault.hcl and modify the <CONSUL_CERT_PEM> and <CONSUL_CERT_KEY> and use the cert we just copy  (Consul Agents/Vault Servers)

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_cert_file = "/etc/vault.d/tls/vault-server.crt"
  tls_key_file  = "/etc/vault.d/tls/vault-server.key"
}
api_addr = "https://192.168.1.2:8200"
cluster_addr = "https://192.168.1.2:8201"
storage "consul" {
  address = "127.0.0.1:8501"
  path = "vault/"
  token = "872014d2-c422-3575-354c-a57b70bc5e9f"
  scheme = "https"
  tls_ca_file   = "/etc/consul.d/tls/consul-agent-ca.pem"
  tls_cert_file = "/etc/vault.d/tls/dc1-client-consul-2.pem"
  tls_key_file  = "/etc/vault.d/tls/dc1-client-consul-2-key.pem"
}
ui = true

Change the number of the certificate on each Vault Server

Finally export the environment variable CONSUL_CLIENT_CERT and CONSUL_CLIENT_KEY on each Vault server  (Consul Agents/Vault Servers)

export CONSUL_CLIENT_CERT=/etc/consul.d/tls/dc1-client-consul-2.pem
echo "export CONSUL_CLIENT_CERT=/etc/consul.d/tls/dc1-client-consul-2.pem >> ~/.bashrc
export CONSUL_CLIENT_KEY=/etc/consul.d/tls/dc1-client-consul-2-key.pem
echo "export CONSUL_CLIENT_KEY=/etc/consul.d/tls/dc1-client-consul-2-key.pem >> ~/.bashrc

Changing the number of the cert on each server

Setup the Vault servers

Enable the vault service

systemctl enable vault.service

Now we can start our Vault servers (Consul Agents/Vault Servers)

systemctl start vault

You won't see any output from these commands unless something goes wrong. But just to check, please wait 30 seconds and then run the following command on each server: (Consul Agents/Vault Servers)

ps -ef | grep vault

This should show 2 lines, one of which includes a line including "/usr/local/bin/vault server -config=/etc/vault.d".

If the above command does not show 2 lines on one of the Vault servers, please run cat /etc/vault.d/vault.hcl and verify your configuration

If everything is right, check the vault status like this

vault status

You will get something like this

Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            n/a
HA Enabled         true

The fact that you no longer get an error indicates that you have set all your Vault environment variables correctly. The actual response indicates that the Vault server has not yet been initialized and is sealed.

Initialized the Vault servers

Initialize your vault cluster with the following command on one of the vault servers

vault operator init -key-shares=3 -key-threshold=2 | tee /home/ubuntu/initialization.txt

You will get 3 keys and the Root Token

Normally, you would not save these into a file, but we want to make sure you can find them later in the lab if needed. You should only do this once; running the init command against an initialized cluster will give an error.

Unseal the Vault Servers

After you initialize a Vault cluster, you then have to unseal each of its servers. This can be done manually using the unseal keys that you obtained when initializing the cluster or in an automated fashion using one of Vault's supported Auto Unseal(https://www.vaultproject.io/docs/concepts/seal#auto-unseal) methods which are also known as seals. You can see all of the Auto Unseal methods here(https://www.vaultproject.io/docs/configuration/seal).

We will start out with manual unsealing

vault operator unseal

Provide one of your 3 unseal keys the first time and a different unseal key the second time you run this for each server. It does not matter which keys you use or what order you use them in as long as you do use 2 different keys for each server.

For each server, after running the command the first time, you will see the status of the server. It should look like this:

Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       3
Threshold          2
Unseal Progress    1/2
Unseal Nonce       528621e4
Version            1.6.1
HA Enabled         true

Vault is still sealed, but the results confirm that your first key has been accepted and that you only need one more.

After running the command a second time on each server, you will see slightly different output:

Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           3
Threshold              2
Version                1.6.1
Storage Type           consul
Cluster Name           vault-cluster-7028157a
Cluster ID             985db9a6-fb16-f62e-f04a-eeab3c5e5e36
HA Enabled             true
HA Cluster             https://192.168.1.2:8201
HA Mode                standby
Active Node Address    https://192.168.1.2:8200

The vault status is telling us the following:

  • The Seal Type of shamir indicates that we are using Shamir's Secret Sharing Algorithm to split the cluster's master key into separate shares (the unseal keys).
  • The fact that HA Enabled is set to true means that our cluster is a High Availability cluster. That is because Consul is a High Availability storage backend for Vault.
  • HA Cluster and Active Node Address give the URLs of the cluster_addr and api_addr settings for the current leader of the Vault cluster.
  • HA Mode indicates whether the server is currently a standby or an active server. Only one server in a Vault cluster, the leader, is marked as active.
  • Performance Standby Node indicates that the server is a performance standby server that can service read-only requests from applications.
  • There are also Last WAL and Performance Standby Last Remote WAL keys which we did not show above. These refer to "write-ahead-logs" which are used by Vault to track updates. They are not significant within a single cluster, but should be close to each other when comparing between Vault clusters.

Next, we recommend that you inspect the vault service on the "Services" tab of the Consul UI.

You can open the consul UI in your browser and calling any of the clients https://192.168.1.X/8501/ui

It should have 3 instances, two of which have the "performance-standby" tag while the other has the "active" tag. The one with the active tag should have a name that includes the same IP shown in the "HA Cluster" and "Active Node Address" keys displayed by the vault status command.

Please run the following commands to see how the Vault servers are exposed by Consul's DNS interface:

Run this command to see all the Vault servers:

nslookup vault.service.consul -port=8600

This should list all 3 Vault servers and their IPs.

Then run this command to see the active Vault server:

nslookup active.vault.service.consul -port=8600

This should only list one server, and its IP should match the one shown in the "HA Cluster" and "Active Node Address" keys displayed by the vault status command.

Then run this command to see the standby Vault servers:

nslookup standby.vault.service.consul -port=8600

This should list the other two Vault servers.

Consul will automatically update the Vault service in its service catalog as servers are started and stopped and as their active/standby status changes. To see how Consul updates the Vault service, please follow this process:

  • Run the consul members command on any server to determine which Vault server has the same IP as the current active server.
  • Run systemctl stop vault on the corresponding to the active Vault server.
  • After 10 seconds, run nslookup active.vault.service.consul on the consul server or any other vault server
  • You should see a different IP listed in the answer section than what was listed when you ran this command the first time.

Well done, you complete this lab!

References: