skip to content

A teammate committed a production database password as an ordinary value in `Pulumi.prod.yaml` rather than with `--secret`. What do you do, and what does `pulumi stack change-secrets-provider` actually re-encrypt?

level: seniorimportance: nice to knowfreq 25%

answer

  1. rotate before you re-encrypt
  2. encryption afterwards protects the next value
  3. re-set it with --secret, then deploy
  4. change-secrets-provider swaps the key, not the classification
  5. git history and CI logs still have it

basics

~20 s

Treat the credential as compromised and rotate it at the source first, then re-set it with pulumi config set --secret. Changing the stack's secrets provider re-encrypts the stack's secret config and the secrets held in its state under a new key — it does nothing about a plaintext value already in the repository's history.

solid answer

~50 s

Order matters. The value was written in the clear, committed, and pulled by everyone with the repo, so nothing you do inside Pulumi makes it secret again — rotate the actual database password first. Then set the new one properly with `pulumi config set --secret dbPassword`, so `Pulumi.prod.yaml` carries only a `secure:` ciphertext, and deploy. Separately, purge or at least record the exposure in git history and check CI logs, because a plaintext config value happily appears in a pipeline's diff output. `pulumi stack change-secrets-provider <url>` is a different tool for a different problem: it re-encrypts the stack's existing secret config values and the secret properties in its state under a new key — moving a stack from a shared `passphrase` to `awskms://`, or rotating away from a key someone who left could still use. It cannot retroactively protect a value that was never marked secret.

code

bash · 10 lines
bash
# 1. Rotate the credential at the source first (outside Pulumi).

# 2. Store the replacement encrypted.
pulumi config set --secret dbPassword "$NEW_PASSWORD" --stack prod
pulumi config get dbPassword --stack prod          # prints [secret]
pulumi up --stack prod

# 3. Separately: move the stack off a shared passphrase onto a KMS key.
pulumi stack change-secrets-provider \
  "awskms://alias/pulumi-prod?region=eu-west-1" --stack prod

go deeper

for a junior

Know that --secret is what makes a config value encrypted in Pulumi.<stack>.yaml, and that a value committed without it has to be treated as exposed rather than quietly fixed.

for a middle

Explain the difference between marking a value secret and changing the key that encrypts secrets, and be able to say what pulumi stack change-secrets-provider covers: existing secret config and secret state properties, nothing else.

for a senior

Lead with rotation at the source, then containment — git history, CI logs, wherever else the value was pasted — and finish with the prevention that actually holds: scanning stack config files and generating credentials instead of typing them.

for a principal

Frame it as a control failure, not a mistake: decide who may decrypt a production stack, whether that right is revocable per person, and whether humans should ever handle the credential at all. Be ready to justify the cost of moving to generated or store-fetched secrets.

## First: the value is burnt A plaintext credential in a committed file has been distributed to everyone who cloned the repository, every CI cache, and every fork or mirror. Encryption applied afterwards protects the *next* value, not this one. So the first action is at the source system: rotate the database password. Nothing in the tool changes that ordering, and an interviewer asking this question is largely checking that you say it before you say anything about Pulumi. ## Second: put the replacement in correctly ```bash pulumi config set --secret dbPassword "$NEW_PASSWORD" --stack prod pulumi up --stack prod ``` Now `Pulumi.prod.yaml` holds: ```yaml config: myapp:dbPassword: secure: v1:...:ciphertext... ``` and the value is encrypted both there and, once it flows into the resource, in the stack's checkpoint. Confirm with `pulumi config get dbPassword --stack prod`, which prints `[secret]` unless you add `--show-secrets`. ## Third: contain the exposure - **Git history.** The old commit still contains the plaintext. Rewriting history is possible but disruptive and never complete once the repository has been cloned; do it if policy requires, and treat rotation as the real remedy. - **CI logs.** A non-secret config value shows up in preview output and diffs. Check whether the pipeline printed it, and whether those logs are retained and readable by a wider group than the repository. - **Anywhere the value was copied.** A password set in the clear tends to have been pasted elsewhere too. ## What `change-secrets-provider` is for ```bash pulumi stack change-secrets-provider "awskms://alias/pulumi-prod?region=eu-west-1" --stack prod ``` This re-encrypts the stack's secrets under a new provider: the secret config values in `Pulumi.<stack>.yaml` and the secret properties recorded in the stack's state. The stack file's key material is rewritten accordingly — the `encryptionsalt` of a passphrase provider gives way to the new provider's settings. The legitimate uses are: - Migrating from a shared `passphrase` to a cloud KMS key so decryption becomes an auditable, revocable IAM grant. - Rotating after someone with the passphrase leaves, since a shared string cannot be revoked from one person. - Moving a stack to a different backend or account where the old key is not reachable. The limits are the point of the question. It re-encrypts what is already *marked* secret; a value stored as ordinary config stays ordinary. And it operates on the current stack state and config — it does not reach into git history, CI logs, or the copy in someone's shell history. ## Fourth: stop it recurring The durable fixes are cheap: - **Review stack-config diffs.** Any new key in `Pulumi.<stack>.yaml` whose value is not under `secure:` and whose name looks credential-shaped should stop a pull request. - **Secret scanning in pre-commit and CI.** The same scanner you run over source code should cover stack config files. - **Prefer never having the value at all.** Where the platform supports it, generate the credential in the program or read it from a secret store at deploy time, so no human ever types it into config and no plaintext version exists to commit. - **Watch `--show-secrets` in pipelines.** It is the other common way a properly stored secret ends up in a log. ## The summary an interviewer wants Rotate first because the tool cannot un-leak a value; re-set it with `--secret` so the file carries ciphertext; use `change-secrets-provider` for key hygiene — moving off a shared passphrase or rotating after a departure — and be clear that it re-encrypts existing secrets rather than reclassifying values that were never secret.

  • Does running `pulumi stack change-secrets-provider` help with the value that already leaked?
    No. It re-encrypts the stack's already-secret config and state properties under a new key; the plaintext sitting in a previous commit is untouched and still readable by anyone with the repository. It is a key-hygiene tool — useful for moving off a shared passphrase or rotating after someone leaves — not an incident remedy.
  • How would you stop this happening again?
    Review stack-config diffs for any credential-shaped key whose value is not under `secure:`, and run the same secret scanner over `Pulumi.<stack>.yaml` that you run over source. Better still, avoid the human step: generate the credential in the program or fetch it from a secret store at deploy time so no plaintext version ever exists to commit.
  • A properly stored secret still appeared in a CI log. How?
    Usually `--show-secrets` on `pulumi config get` or `pulumi stack output` in a pipeline step, or program code that pulled the value out of an Output and logged or interpolated it into a shell command. Encryption at rest doesn't survive code that deliberately prints the value, so treat those calls as review items.

saying these in an interview costs you the question

  • Re-encrypting the config is treated as fixing the leak
  • Assumes deleting the commit removes the exposure
  • Thinks change-secrets-provider marks existing values as secret
  • Ignores CI logs where the plaintext was printed
  • Rotates the key but never the database password

context