Integrating HashiCorp Vault with Terraform for Secrets Management

When working with sensitive data in cloud infrastructure, managing secrets securely is essential. HashiCorp Vault, a powerful tool for managing secrets, integrates smoothly with various providers, including AWS, to enhance security and control over access to sensitive information. In this blog, we'll explore how to set up and configure Vault on an AWS EC2 instance running Ubuntu, integrate it with Terraform, and implement secrets management efficiently.

Prerequisites

• Basic understanding of AWS and Terraform

• An AWS account with permissions to create EC2 instances

• Terraform CLI installed

Step 1: Set Up an AWS EC2 Instance with Ubuntu

To get started, launch an EC2 instance in AWS with Ubuntu:

1. Navigate to AWS Management Console > EC2 service.

2. Click Launch Instance and select the Ubuntu Server AMI.

3. Choose an instance type (e.g., t2.micro for testing).

4. Configure settings as needed, then Launch the instance.

After launching, connect to your instance via SSH.

Step 2: Install HashiCorp Vault on Ubuntu

Vault is not included by default on Ubuntu. To install Vault, first add HashiCorp’s package repository:

Add HashiCorp Repository: This tells Ubuntu’s package manager (apt) where to find Vault.

Install Vault via Package Manager: Once the repository is added, you can use Ubuntu’s package manager (apt) to install Vault. This makes installing, updating, and managing Vault easier since apt handles all the details.

1. Update package list and install GPG:

sudo apt update && sudo apt install -y gpg

2. Add the HashiCorp repository:

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update

Verifying key’s fingerprint is not mandatory for installation. However, it’s generally recommended as a security best practice to confirm that the key is valid, which ensures that the repository and its packages are from a trusted source.

Verify the key's fingerprint

gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint

Add the HashiCorp repo

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update

3. Install Vault:

sudo apt install vault

Step 3: Start Vault in Development Mode

To start Vault, you can use the following command:

vault server -dev -dev-listen-address="0.0.0.0:8200"

For a proof of concept (POC) setup, you can use HashiCorp Vault's development server mode. This is a quick setup ideal for testing, but it’s not secure for production because it doesn’t persist data and lacks critical security measures.

Note: In a production environment, you’ll need to set up a production-grade instance of Vault.

Production Server Setup:

• Run Vault in server mode with a secure backend like Consul or AWS S3 to store data safely.

• Use TLS (Transport Layer Security) to encrypt all communication.

• Set up SSL/TLS certificates to authenticate and secure connections, which involves getting or creating certificates and configuring Vault to use them

Configure Environment Variable

To access Vault from outside the instance, set the VAULT_ADDR environment variable to the public IP of your EC2 instance:

export VAULT_ADDR='http://<EC2_PUBLIC_IP>:8200'

Access Vault using http://<EC2_PUBLIC_IP>:8200 and copy the root token displayed in the terminal output.

If you encounter an error like:

"Error Authentication failed: permission denied invalid token"

If you don’t have access to the initial root token, you can stop the existing Vault server and restart it in development mode to generate a new root token.

1. Stop the Current Vault Process: Find the process running on port 8200 and terminate it. You can use the following commands:

    sudo fuser -k 8200/tcp
   

2. Restart Vault in Development Mode: Start Vault again with the -dev flag:

    vault server -dev -dev-listen-address="0.0.0.0:8200"
   

   This will output a new root token.

Note: In Vault’s development mode, if you reset or lose the token, all stored data, roles, and secrets are lost, as this mode doesn't persist data.

Step 4: Configure the Key-Value (KV) Secret Engine in Vault

In HashiCorp Vault, you can create and store different types of secrets using secret engines. Secret engines are plugins or modules that allow Vault to manage different types of secrets, from simple key-value pairs to more complex secrets like database credentials or Kubernetes tokens.

HashiCorp Vault's KV secret engine allows for storing secrets like API keys, passwords, and configurations:

1. Enable the KV Secret Engine:

Select KV

Key-Value (KV) Secret Engine for Regular Key-Value Pairs

The KV (Key-Value) secret engine is commonly used to store secrets like API keys, passwords, and configuration data. This engine makes it easy to save, retrieve, and manage simple secrets securely.

Enable the KV Secrets Engine with a Custom Path and Versioning Settings:

• Set the path to kv (mount).

• Configure the maximum number of versions to retain per key (default is 10 versions if unset or set to 0).

• Enable Check and Set (CAS) if you want to enforce conditional writes on all keys, which requires a version check to prevent overwrites.

2. Store Secrets: Navigate to the KV path in Vault and add your secrets there. Vault will encrypt each key-value pair as soon as you store it, keeping only the encrypted version. This means that only Vault can access and decrypt the data, ensuring that your secrets stay secure.

Click "Create Secret," then provide the path for the secret, the secret data name, and the password, and then save it.

Step 5: Create AppRole Authentication for Terraform

To allow Terraform or Ansible to access a secret in HashiCorp Vault, you indeed need to:

• Create a role within Vault.

• Assign policies to that role to define the necessary permissions for accessing specific secrets.

This concept is similar to IAM roles in AWS, where you define the permissions that external entities (like users, applications, or services) have for accessing resources. In both cases, roles and policies control access based on predefined permissions.

Go to access→ Click Enable new Method

Click App role and Enable

It tells Vault to enable the AppRole authentication method.

You authenticate Terraform or Ansible using this AppRole.

2. Create an AppRole:

We need to create a policy first (to access Terraform).

Define permissions for Terraform to read secrets by creating a custom policy:

vault policy write terraform - <<EOF
path "*" {
  capabilities = ["list", "read"]
}

path "secrets/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "kv/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}


path "secret/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "auth/token/create" {
capabilities = ["create", "read", "update", "list"]
}
EOF

Error: http: server gave HTTP response to HTTPS client

Error uploading policy: Put "https://127.0.0.1:8200/v1/sys/policies/acl/terraform": http: server gave HTTP response to HTTPS client

When encountering this error while uploading a policy, it's likely due to Vault being accessed over HTTP instead of HTTPS.

Ensure to Set VAULT_ADDR to the EC2 Public IP:

Since you're accessing Vault from outside the EC2 instance, you should set VAULT_ADDR to the public IP of the EC2 instance. In your case, your EC2 public IP is 3.83.121.14.

export VAULT_ADDR=http://3.83.121.142:8200

Now Terraform has access to the secret folder and the kv path.

Now you'll need to create an AppRole with appropriate policies and configure its authentication settings. Here are the steps to create an AppRole:

Enable Role Authentication

1. Enable AppRole:

vault auth enable approle

2. Create the AppRole:

vault write auth/approle/role/terraform \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40 \
    token_policies=terraform

In a Production environment, read the HashiCorp documentation and configure the parameters according to your requirements.

3. Generate Role ID and Secret ID:

After creating the AppRole, you need to generate a Role ID and Secret ID pair. The Role ID is a static identifier, while the Secret ID is a dynamic credential.

a. Generate Role ID:

You can retrieve the Role ID using the Vault CLI:

vault read auth/approle/role/terraform/role-id

Save the Role ID for use in your Terraform configuration.

Generate Secret ID:

To generate a Secret ID, you can use the following command:

vault write -f auth/approle/role/terraform/secret-id

This command generates a Secret ID and provides it in the response. Save the Secret ID securely, as it will be used for Terraform authentication.

Step 6: Integrate Vault with Terraform

Now that Vault is set up, configure Terraform to use it as a secrets provider:

Below the demo terraform configuration demonstrates how to integrate HashiCorp Vault with AWS to retrieve secrets and use them to configure an EC2 instance*.*

AWS Provider Configuration:

provider "aws" {
  region = "us-east-1"
}

This section configures the AWS provider to use the us-east-1 region. It allows Terraform to interact with AWS resources.

Vault Provider Configuration:

provider "vault" {
  address = "http://3.83.121.142:8200" # # Replace with your Vault server's IP
  skip_child_token = true

  auth_login {
    path = "auth/approle/login" #This tells Vault to use the AppRole authentication method

    parameters = {
      role_id = "3b1aedf0-32eb-b0c9-105b-f4f236df3399" # # Replace with your Vault AppRole Role ID
      secret_id = "a6359ab4-02e7-99cd-daa0-33cf3608f0bd" # Replace with your Vault AppRole Secret ID
    }
  }
}

This configuration tells Terraform where your Vault server is located and how to authenticate with it (via AppRole) using the Role ID and Secret ID you provide.

Vault Secret Retrieval:

data "vault_kv_secret_v2" "example" { 
  mount = "kv" // change it according to your mount
  name  = "test-secret" // change it according to your secret
}

This block retrieves a secret from Vault using the KV (Key-Value) secret engine. It fetches the secret named test-secret from the kv path in Vault.

Refer to the Terraform document for syntax: https://registry.terraform.io/providers/hashicorp/vault/latest/docs/data-sources/kv_secret_v2

AWS EC2 Instance Creation:

resource "aws_instance" "my_instance" {
  ami           = "ami-0866a3c8686eaeeba"
  instance_type = "t2.micro"

  tags = {
    Name = "test"
    Secret = data.vault_kv_secret_v2.example.data["username"]
  }
}

This section creates an EC2 instance in AWS. The Secret tag is populated using the username value from the Vault secret (data.vault_kv_secret_v2.example.data["username"]). The EC2 instance is created with the specified AMI and instance type.

When you run terraform init and see that Vault plugins are downloading, it's part of the initialization process. Terraform uses plugins to interact with different providers, and in this case, it's downloading the Vault provider plugin to interact with HashiCorp Vault.

With the Vault provider plugin, Terraform can securely interact with HashiCorp Vault to fetch secrets and manage resources like policies, roles, and secrets. The plugin allows Terraform to securely access Vault and automate the retrieval of secrets as part of your infrastructure provisioning.

Note: If you encounter a 400 error, it might indicate an expired or invalid Role ID/Secret ID. Regenerate the IDs in Vault as needed.

Once this configuration is applied (terraform apply), Terraform will:

1. Authenticate with Vault.

2. Retrieve the secret from Vault.

3. Create an EC2 instance in AWS, with the Vault secret used as part of the instance’s tags.

If you have two instances — one for the Vault server and another for the instance created using Terraform (e.g., an EC2 instance) — here’s how they interact:

Vault Server Instance: This is the instance running your HashiCorp Vault server, where secrets are stored and managed. It is accessible via its public IP.

Terraform-managed Instance: This is an instance (created and managed by Terraform) that interacts with Vault to fetch secrets and use them during provisioning (e.g., using secrets to configure EC2 instance tags).

Quick Tips:

The Vault Server stores and manages secrets.

The Terraform-managed instance interacts with the Vault server (via the Vault provider plugin) to fetch secrets or manage Vault resources as part of your infrastructure automation.

Conclusion

Integrating HashiCorp Vault with Terraform provides a robust solution for securely managing secrets in cloud infrastructure. While we used a simple POC setup here, be sure to explore production-grade configurations for Vault to ensure data persistence and secure communication. With Vault, you can improve the security and efficiency of your Terraform-based deployments by keeping sensitive data safe.

+--------------------------+
|  Vault Server (EC2)      |
|  (Stores secrets securely)|
+--------------------------+
             |
             v
+--------------------------+
| Configure Vault          |
| - Launch AWS EC2 for Vault|
| - Install and Configure   |
|   Vault                   |
| - Start Vault in Dev Mode |
+--------------------------+
             |
             v
+--------------------------+
| Enable KV Secrets Engine |
+--------------------------+
             |
             v
+--------------------------+
| Write Vault Policy       |
| - Define permissions     |
|   for AppRole            |
+--------------------------+
             |
             v
+--------------------------+
| Configure Vault AppRole  |
| - Set up AppRole for     |
|   Terraform Authentication|
+--------------------------+
             |
             v
+--------------------------+
| Terraform Configuration  |
| - Configure AWS and Vault|
|   providers              |
+--------------------------+
             |
             v
+--------------------------+
| Authenticate Terraform   |
| - Use AppRole to access  |
|   Vault secrets          |
+--------------------------+
             |
             v
+--------------------------+
| Retrieve Secrets in      |
| Terraform Configuration  |
+--------------------------+
             |
             v
+--------------------------+
| Vault Provider           |
| - Fetches secrets from   |
|   Vault for Terraform use|
+--------------------------+
             |
             v
+--------------------------+
| Terraform-managed EC2    |
| Instance Provisioned     |
| - EC2 launched with tags |
|   or configs using secrets|
+--------------------------+
             |
             v
+--------------------------+
| Deploy AWS Infrastructure|
| - Additional resources   |
|   with retrieved secrets |
+--------------------------+