A team rotates their Ansible Vault password every quarter with ansible-vault rekey. Why might their secrets still be compromised, and what does a real rotation involve?

## What rekey actually does ```bash ansible-vault rekey \ --vault-password-file .old_pass \ --new-vault-password-file .new_pass \ group_vars/prod/vault.yml ``` Ansible decrypts the file with the old password, re-encrypts the identical plaintext with the new one, and writes it back. The variable values are byte-for-byte unchanged. With labelled identities the same operation is expressed with `--new-vault-id prod@prompt`, which also updates the label recorded in the header. So rekey rotates the *key*, not the *secret*. That distinction is the whole question. ## Why rotating only the password leaves you exposed The threat model that makes it obvious: a contractor had the vault password and a clone of the repository, and has now left. You rekey. Your `HEAD` commit is now encrypted under a password they do not have. But the commit from last month is still in the history they cloned, still encrypted under the password they do have, and its plaintext is the same production database password you are using right now. Their access to your live credentials was never interrupted. Rekeying is not even a speed bump, because git never forgets and encrypted blobs in history cannot be un-published. The same reasoning applies to any past exposure: a repository briefly made public, a CI log that printed a value, a laptop backup. Once a *value* may have been seen, only changing that value helps. ## What real rotation looks like Rotation is a property of the credential's issuing system, not of the file it is stored in. The order matters, because doing it backwards causes an outage: 1. **Create a new credential at the source** — a second database user or password, a second API key. Most systems allow two valid credentials at once precisely for this. 2. **Put the new value in the vault file** with `ansible-vault edit`, commit, and review. 3. **Deploy** — run the playbook so every consumer is using the new value, and confirm it. 4. **Revoke the old credential** at the source. This is the step that actually ends the exposure, and it is the step teams skip. 5. **Rekey the vault password separately**, on its own schedule or when a password holder leaves. Step 4 is what converts every historical copy of the old ciphertext — in git history, in backups, in someone's clone — into worthless bytes. Steps 1 and 4 being separate is what keeps the service up while the change rolls out. ## Two mechanical traps in rekey itself Even when rekeying is the right operation, it is easy to do incompletely. **It only touches the files you name.** There is no "rekey the repository" mode. A real estate has `group_vars/*/vault.yml`, `host_vars/*/vault.yml`, and secrets inside roles, so the operation is a sweep: ```bash find group_vars host_vars roles -name 'vault*.yml' -print0 \ | xargs -0 ansible-vault rekey \ --vault-password-file .old_pass \ --new-vault-password-file .new_pass ``` Miss one and the next run fails to decrypt it — or worse, keeps working because CI still supplies the old password too, and the incomplete rotation goes unnoticed until the old password is finally withdrawn. **It ignores inline strings.** A `!vault` scalar embedded in an otherwise plaintext YAML file is not an encrypted file, and `rekey` will not find or rewrite it. Every one has to be regenerated by hand with `ansible-vault encrypt_string` under the new password. A repository that leans on inline strings has a materially more expensive rotation, which is a genuine argument for concentrating secrets in whole encrypted files. ## Rolling the password without an outage Because Ansible can be given several identities in one run, you can cut over gradually: add the new identity to the run alongside the old, rekey files in batches (each batch keeps working because one of the supplied passwords opens it), and remove the old identity only once nothing decrypts with it. That is also how you verify completeness — the moment you withdraw the old password, any file you forgot fails loudly rather than silently. ## What to say in the interview Lead with the distinction: rekey rotates the password, not the secret; git history makes the difference decisive. Then give the ordered rotation procedure with revocation as the terminal step, mention the sweep and the inline-string gap, and close with the honest limitation — Vault has no per-user keys and no audit trail, so "who still has the old password?" is a question the tool cannot answer for you. That last point is usually what the interviewer was probing for.