Why does putting a function's database password in a deploy-time environment variable make rotation harder than fetching it from a secret store at runtime?
answer
- where the value is frozen
- a rotation becomes a release
- cache with a lifetime
- both credentials valid for a while
basics
~20 sA deploy-time environment variable is part of the deployed configuration, so changing it means a new version or a restart. A secret read from a store at runtime and cached briefly changes when the store does, with no redeploy.
solid answer
~50 sOn platforms with immutable versions, environment variables are frozen into the published version, so a new password means publishing a version and moving the alias: a full release, run by whoever owns the pipeline, and a later rollback can bring the revoked password back. Even where settings are mutable, changing one usually restarts instances. With a runtime fetch, the function's identity can read one secret, the code reads it at start or first use and caches it for a limited lifetime, and rotation is just an update in the store. Two rules make that safe: cache with a lifetime rather than fetching per invocation, and refetch once if the database rejects the cached credential. Because instances refresh at different times, keep the old and new credentials both valid until every cached copy has expired, then revoke the old one.
go deeper
Recall that a deploy-time environment variable only changes with a new deployment, while a value read from a secret store can change without one.
Explain caching with a lifetime, refetching on authentication failure, and why both credentials must stay valid until every cached copy has expired.
Show the incidents you would prevent: a rollback restoring a revoked password, a store outage failing every invocation, and instances that never refresh a value read once at start.
Discuss the standard a team sets for which values live in the version and which live in a store, and who owns rotation across many functions.
## Two ways a function gets a secret A function that talks to a database needs a password or token. There are two common places to put it: - **A deploy-time environment variable.** The value is written into the function's configuration when it is deployed, and the platform hands it to each sandbox at start. - **A runtime fetch from a secret store.** The function's identity is granted read access to one secret, and the code reads it from the store at start or on first use, keeping it in memory for a limited time. Some platforms offer an in-between form: the configuration holds a *reference* to a secret, and the platform resolves it when an instance starts. For rotation it behaves much like an environment variable, because a running instance keeps the value it started with. ## Why a deploy-time value turns rotation into a release On platforms with immutable versions, environment variables are part of the published version. Changing one means publishing a new version and moving the alias to it: a full release with its own rollout and risk. On platforms without versions, changing a setting usually restarts the function's instances. Either way: - the change has to pass through the deployment pipeline, often owned by a different team than the database - running instances keep the old value until they are replaced - rolling back the release can bring the old, now-revoked password back with it - the value is visible to anyone allowed to read the function's configuration ## Why a runtime fetch avoids the redeploy With a runtime fetch, rotating the secret means updating it in the store. No version is published and no traffic moves. Each instance picks up the new value when its cached copy expires. Two rules make this work: - **Cache with a lifetime, never fetch per invocation.** A fetch on every invocation adds latency, may add per-request charges, and turns a store outage into a failure of every call. A short cache lifetime bounds how long a stale value can survive. - **Refresh on authentication failure.** If the database rejects the cached credential, refetch once and retry, so a rotation that lands mid-cache recovers instead of piling up errors. Reading the secret only once at instance start is the weaker form: the value then lives as long as the instance, which the platform decides and the operator cannot predict. ## Rotating without an outage Cached copies expire at different times, so there is a window in which some instances hold the old value and some hold the new one. A safe rotation keeps both valid across that window: 1. Create the new credential in the database while the old one still works. 2. Write the new value to the secret store as the current one. 3. Wait at least one full cache lifetime, or, with start-only reads, until the instances have been replaced. 4. Confirm in the database's login records that nothing still authenticates with the old credential. 5. Revoke the old credential. Skipping step 3 is the classic outage: the old password is revoked while warm instances still hold it, and their next queries fail until their caches expire. ## The options side by side | | Deploy-time variable | Reference resolved at start | Runtime fetch with cache | |---|---|---|---| | New value reaches running code | After a new version or restart | When new instances start | When the cache expires | | Rotation needs a release | Yes | No, but needs instance turnover | No | | Extra call at start or first use | No | No | Yes | | Rollback can restore an old secret | Yes | Only if the reference pins an old secret version | No | ## Configuration that is not secret Feature flags, endpoint names and timeouts follow the same logic. Values that change together with the code belong in the version, so a rollback brings them back in step with it. Values that change on their own schedule, such as credentials, signing keys and kill switches, belong in a store read at runtime, with read permission granted in the same deployment template that declares the function.
- Why might reading the secret only once, when an instance starts, not be enough?A warm instance can live for a long and unpredictable time, and it never sees the new value until the platform replaces it. Add a cache lifetime so the value is reread periodically, and a refetch on authentication failure so an instance holding a revoked credential recovers on its next query.
- How do you keep a rotation from causing a burst of failed database logins?Keep the old and new credentials both valid until every cached copy has expired, and make the function refetch the secret once and retry when the database rejects it. Then check the database's login records for the old credential before revoking it.
- Updating the value in the secret store is seen at once by every running instance.
- Fetching the secret on every invocation is the safest default.
- Changing an environment variable needs no rollout because nothing is republished or restarted.
- Revoking the old password right after writing the new one is safe.
- An environment variable is encrypted at rest, so it rotates as easily as a secret store.