Explain how rotation works in AWS Secrets Manager: what the AWSCURRENT, AWSPENDING and AWSPREVIOUS staging labels are for, and what the rotation function is expected to do at each of its four steps.
answer
- immutable versions, movable labels
- three reserved labels, one default read
- four invocations, one function
- the outside world changes at one step only
- one user or two, and why it matters
basics
~20 sSecrets Manager versions a secret and moves staging labels between versions. AWSCURRENT is what readers get by default, AWSPENDING is the candidate being rotated in, AWSPREVIOUS is the last good one. The rotation function runs createSecret, setSecret, testSecret and finishSecret.
solid answer
~50 sA secret is a set of immutable versions with movable **staging labels** on top. `GetSecretValue` with no version returns whatever carries `AWSCURRENT`; `AWSPENDING` marks a candidate mid-rotation; `AWSPREVIOUS` marks the version that `AWSCURRENT` just left. Secrets Manager invokes the rotation function four times with the same `SecretId` and `ClientRequestToken` but a different `Step`. **createSecret** generates the new credential and stores it as a new version labelled `AWSPENDING`. **setSecret** changes the credential in the target system — the database user's password, say — to match that pending version. **testSecret** opens a real connection with the pending credential to prove it works. **finishSecret** moves `AWSCURRENT` onto the pending version, which implicitly demotes the old one to `AWSPREVIOUS`. Because readers only see the new value at the last step, and the old one stays valid until then, in-flight clients are not cut off mid-rotation.
go deeper
Know that a secret can rotate on a schedule and that applications keep calling the same secret name rather than being redeployed with a new value. You are not expected to write a rotation function.
Explain that versions are immutable and labels move, that AWSCURRENT is the default read, and name the four rotation steps in order with one sentence each about what changes where.
Show the failure reasoning: why the old credential must stay valid until the label moves, why alternating users protects pooled connections, and what a client must do on an auth failure so rotation is not an incident.
Own the estate-level position: which credentials get managed rotation at all, how rotation failures are alarmed and who owns the pager, and how you avoid a fleet of bespoke rotation Lambdas whose testSecret steps nobody reviews.
## Versions and labels, not overwrite The first thing to get right is that AWS Secrets Manager does not overwrite a secret. Every write creates a new **version**, identified by a `VersionId`, and versions are immutable. What moves is a small set of **staging labels** — pointers you can relocate from one version to another. Three labels are reserved: - **`AWSCURRENT`** — the version returned by `GetSecretValue` when the caller does not name a `VersionId` or `VersionStage`. This is what every application gets. - **`AWSPENDING`** — the candidate version created during a rotation, not yet promoted. - **`AWSPREVIOUS`** — the version that `AWSCURRENT` most recently moved away from, kept so you can fall back. You can also attach your own custom labels, but the rotation machinery is defined entirely in terms of these three. Understanding that rotation is *label movement over immutable versions* is what makes the rest obvious. ## The four steps Rotation is configured on the secret with `RotationRules` (a `ScheduleExpression` or a day interval) and a rotation Lambda function. When the schedule fires, Secrets Manager invokes that function **four separate times**, passing the same `SecretId` and the same `ClientRequestToken` — which is the `VersionId` of the pending version — and a different `Step` each time. ``` createSecret -> setSecret -> testSecret -> finishSecret ``` **createSecret.** Generate the new credential (typically with `GetRandomPassword`) and store it as a new version carrying `AWSPENDING`. This step must be idempotent: if a version already exists for the given token, do nothing. Nothing outside Secrets Manager has changed yet. **setSecret.** Go to the target system and make the new credential real — issue the `ALTER USER` against the database, call the third party's key-rotation API, whatever applies. This is the only step that touches the outside world, and it is where a rotation function most often needs elevated credentials of its own (a master user, or a second application user). **testSecret.** Use the `AWSPENDING` value to actually do the thing the credential is for — open a connection and run a trivial query. This is the step teams are tempted to stub out, and it is the one that prevents promoting a credential that does not work. **finishSecret.** Move the `AWSCURRENT` label from the old version onto the pending one. Secrets Manager automatically puts `AWSPREVIOUS` on the version that just lost `AWSCURRENT`, and removes `AWSPENDING`. Only now do ordinary readers see the new value. If any step throws, Secrets Manager retries; a rotation that keeps failing leaves `AWSPENDING` in place and raises the secret's rotation-failed state, which is exactly what you want to alarm on. ## Why the ordering avoids an outage The sequence is built so that at no point is there a window where the credential readers hold is invalid. Between `setSecret` and `finishSecret` the new credential works but `AWSCURRENT` still points at the old one — so a client that fetched before rotation is still fine *if the old credential is still accepted*. That last clause is the crux, and it is why Secrets Manager offers two rotation strategies: - **Single user.** One database user whose password is changed in place. Simple, one identity to manage, but the old password stops working the instant `setSecret` runs. Any client holding a cached value will fail until it refetches. Acceptable when clients open connections per request and refetch on auth failure. - **Alternating users.** Two users that take turns; rotation changes the password of the *inactive* one and then promotes it. The credential currently in `AWSCURRENT` keeps working throughout, so long-lived connections and caches ride out the rotation. It costs you a second identity and a clone-permissions step. This is the tradeoff an interviewer is usually fishing for: managed rotation does not by itself make rotation non-disruptive; the *strategy* plus the client's refetch behaviour does. ## What clients must do Even with alternating users, a client that caches a secret forever will eventually hold something stale. The durable pattern is: cache the value with a bounded lifetime, and on an authentication failure invalidate the cache and refetch once before giving up. That single retry converts a rotation from an incident into a blip, and it works regardless of which strategy the secret uses. For RDS, Aurora, DocumentDB and Redshift, AWS supplies the rotation function so you pick a strategy rather than write Lambda code. For anything else — a third-party API key, an internal service credential — you write the four steps yourself, and `testSecret` is where you earn your salary. ## Operational notes `RotateSecret` triggers a rotation on demand, which is how you exercise the path before you need it. `DescribeSecret` reports the rotation configuration, the last rotated date and the version-to-label mapping, which is the first thing to look at when a rotation is stuck. And a secret whose `AWSPENDING` label has been sitting on a version for days is a failed rotation nobody noticed — worth an alarm.
- What breaks if the rotation function skips the testSecret step?You lose the only check that the pending credential actually works before it becomes the one everybody reads. A `setSecret` that silently half-succeeded — wrong host, insufficient grants, a password the target quietly truncated — gets promoted by `finishSecret`, and the failure surfaces as a production-wide authentication outage instead of a failed rotation you could retry.
- A client holds a cached secret and rotation just completed. How should it recover?On an authentication failure it should invalidate its cached copy, call `GetSecretValue` again to pick up the new `AWSCURRENT`, and retry the operation once. Bounded cache lifetimes alone are not enough because the window between rotation and expiry is exactly when the failure happens. One refetch-on-auth-error turns rotation into a blip.
- When would you pick the single-user strategy over alternating users?When the credential is used by short-lived callers that open a connection per request and already retry on auth failure, and when creating a second identity in the target system is awkward — some third-party APIs only allow one active key. Alternating users is the safer default for databases with long-lived pooled connections, at the cost of a second user to keep permissions in sync.
saying these in an interview costs you the question
- Thinking rotation overwrites the secret in place
- Believing rotation alone guarantees zero client disruption
- Assuming AWSPENDING is what applications read
- Skipping testSecret because setSecret returned success
- Calling finishSecret before the target actually accepts the credential