A database password in the Git repository, an API key in a .env file on the web server, a root password in plain text in an Ansible variables file – we come across something like this in almost every project we take over for operations. Secrets management is about keeping this data in one place, encrypted, with logged access, so that only those who need it get it. In this article we go through the options, from simple file-based encryption to HashiCorp Vault, and also cover Vault’s licence change and the OpenBao fork.
What’s wrong with Git, .env and plain text?
- Git doesn’t forget. A password pushed once stays in the history even after the commit is deleted, in every clone, on the developers’ machines and in the cache of the CI runners. The only sure fix is to change the password.
- The
.envfile ends up everywhere. It gets into backups, accidentally into a container image or a public web directory, and on the server anyone with access to the application’s user can read it. - No logging and no rotation. You don’t know who looked at the password and when, and after a colleague leaves, nobody changes it, because nobody knows in how many places it appears.
In the cloud the stakes are especially high: a leaked access key can cause an expensive bill or a data leak within minutes. We have already written about common cloud security mistakes.
The options in brief
| Tool | How does it work? | When is it a good fit? |
|---|---|---|
| Ansible Vault | Encrypts files or individual variables with a password | If you configure with Ansible anyway, few secrets, small team |
| SOPS | Encrypts only the values in YAML, JSON and ENV files; the key can be age, PGP or a cloud KMS | Encrypted configuration in Git, GitOps |
| HashiCorp Vault | A separate service with an API, policies, logging and dynamic secrets | Several teams, many systems, mixed environment |
| AWS Secrets Manager | Managed service with IAM permissions and automatic rotation | If the application runs on AWS |
| Google Secret Manager | Managed service with IAM permissions and versioning | If the application runs on Google Cloud |
Ansible Vault
The smallest step. You can encrypt a variable like this:
ansible-vault encrypt_string 'S3cr3t-password' --name 'db_password'
The output can be pasted into the variables file, and when you run the playbook, you provide the key with the --ask-vault-pass option or a password file. The downside is that a single shared password protects everything, there is no logging, and changing the password means re-encrypting all the files.
SOPS
SOPS leaves the structure of the file readable and encrypts only the values, so changes can also be tracked in Git. You don’t have to manage the keys yourself if you use AWS KMS or Google Cloud KMS: access is controlled by IAM. It fits well with Kubernetes and GitOps environments.
Cloud secret managers
AWS Secrets Manager costs $0.40 per secret per month, plus $0.05 per 10,000 API calls. Google Secret Manager charges $0.06 per active secret version per location per month and $0.03 per 10,000 access operations, with six secret versions per month free. If your system runs in one cloud, these are the most convenient: there is nothing to operate, and permissions can be tied to existing IAM roles. On AWS, the SecureString parameters of Systems Manager Parameter Store are a cheaper alternative for simpler needs; we wrote about SSM in this article.
HashiCorp Vault basics
Vault is a standalone server that you access through an API, from the command line or via a web interface. The current stable branch is 1.15, with 1.15.5 released at the end of January. The most important concepts:
Secrets engines. The simplest is the KV (key–value) engine, which in version 2 also keeps earlier versions of secrets. Storing and reading a secret:
export VAULT_ADDR='https://vault.example.com:8200'
vault login
vault kv put -mount=secret myapp/db username=app password='S3cr3t-password'
vault kv get -mount=secret myapp/db
vault kv get -mount=secret -field=password myapp/db
Dynamic secrets. Vault doesn’t just store credentials, it also generates them. On request, the database engine creates a temporary user, for example in PostgreSQL or MySQL, which is deleted automatically when the configured time (TTL) expires; the AWS engine likewise issues short-lived access keys. This way a leaked password is only worth something for a few hours, and every application instance has its own access.
Authentication and policies. People can log in with, for example, LDAP or OIDC, and machines with AppRole, a Kubernetes service account or a cloud role. Who can access what is decided by an HCL policy. This one, for example, grants only read access to the application’s secrets (with the KV version 2 engine, the path contains data/):
path "secret/data/myapp/*" {
capabilities = ["read"]
}
vault policy write myapp-read myapp-read.hcl
Seal/unseal. When it starts, Vault is in a sealed state: its data is encrypted, and it doesn’t store the key on disk. At initialisation (vault operator init), five key shares are created by default, and three of them must be entered to unseal it (vault operator unseal). This is secure, but it requires manual intervention at every restart, so in production auto-unseal with a cloud key management service (AWS KMS, Google Cloud KMS) is common.
From an operations point of view, Vault is a critical service: if it goes down, applications don’t get their secrets. So in production it’s worth running it as a cluster of at least three nodes (with the built-in Raft storage), with regular snapshot backups and audit logging enabled.
The licence change and OpenBao
On 10 August 2023, HashiCorp announced that its products – including Vault and Terraform – would be released under the Business Source License 1.1 instead of the Mozilla Public License 2.0, starting with future releases. The BSL restricts building products and services that compete with HashiCorp; you can still use the products freely in your own infrastructure, for internal purposes. Vault 1.15, released in September 2023, already came under the BSL.
In response, the community started forks. OpenTofu, forked from Terraform, released its first stable version, 1.6.0, under the Linux Foundation on 10 January this year. The Vault fork, OpenBao, started last November under the Linux Foundation’s LF Edge project, led mainly by developers from IBM, from Vault’s last MPL-licensed code. OpenBao is still at an early stage and has no released version yet, so it is not an option for production use yet. It is worth watching, though, especially if you are unsure because of the licence terms. We also wrote about a similar upheaval caused by a licence change in the VMware–Broadcom case.
Which one should you choose?
- Small project, a few servers, Ansible: Ansible Vault, with a separate password file that is not in Git.
- Configuration in Git, GitOps, Kubernetes: SOPS with a cloud KMS key.
- An application running in one cloud: the cloud’s own secret manager, with an IAM role, without keys.
- Several teams, mixed environment, dynamic access: HashiCorp Vault – but only if there is someone to run it as a cluster and back it up.
Whichever you choose, the first step is the same: find the existing secrets in Git and on the servers, change those that have already leaked, and make it the rule that new secrets can only go into the chosen tool.
If you need help choosing a secrets management solution, deploying Vault, or reworking existing pipelines and Ansible roles, get in touch; you can read more about this on the Cloud and DevOps page.