skip to content

One image, two environments: production fails at start-up on a configuration value staging has had for weeks — how did that happen?

level: seniorimportance: should knowfreq 45%

answer

  1. values may differ, keys may not
  2. the set was edited where it was needed
  3. nothing declared what the artifact requires
  4. diff key names across stages, not values
  5. fail at start-up naming every missing key

basics

~20 s

The value sets are maintained per environment, so a key added where it was first needed never travelled to the others. Nothing declared the key set as a contract, so the gap stayed invisible until the code path that reads it ran somewhere nobody supplied it.

solid answer

~50 s

This is configuration drift, and it is the failure mode that one-artifact delivery makes possible: because the bytes are identical, the only way the two environments can differ is the values, and those are edited per environment. Someone added the key in staging alongside the code change that reads it, production's set was never updated, and nothing compared the two. Start-up failure is the lucky version — the unlucky one is a key read by a code path that runs at three in the morning. The fixes are a contract and a comparison: have the artifact declare the keys it requires and exit non-zero naming any that are missing; keep the key set in one shared document so adding a key forces an answer for every stage; render each stage's resolved set in the pipeline and treat a key-name difference between environments as an error, while value differences are expected.

code

pseudocode · 16 lines
pseudocode
requiredKeys = ["storageEndpoint", "reportRetentionDays"]
optionalKeys = ["exportFormatBeta"]

resolved = resolveValuesForThisStage()

missing = every key in requiredKeys where resolved[key] is absent or null
unknown = every key in resolved where key is not in requiredKeys + optionalKeys

for each key in unknown:
    log("supplied key not read by this version: " + key)   # typo or a newer/older set

if missing is not empty:
    log("missing required values: " + missing)             # all of them, not the first
    exit(non-zero)                                         # fail before serving

serve(resolved)

go deeper

for a junior

Take away the distinction: across environments the values are supposed to differ, the key names are not. A key that exists in only one place is a bug waiting for a code path.

for a middle

Explain why it hides — nothing declares which keys the artifact needs, and a whole-set diff is drowned in legitimate value differences — and what start-up validation changes about when you find out.

for a senior

Design the guardrails and say what each one catches: declared required keys with a loud non-zero exit, a key-name comparison in the pipeline, and one staged path for value changes so nothing is applied where it was needed.

for a principal

Decide that value sets are under the same review and promotion discipline as code, and that hand edits in an environment are drift by definition, then make the platform re-render rather than preserve them.

## How the gap opens Drift is not carelessness so much as the natural shape of per-environment value sets. The usual sequence: 1. A change adds a code path that reads a new key. 2. The engineer adds the key to the environment they are working in, because that is where they needed it and where they verified it. 3. The change is promoted. The artifact moves forward correctly — the bytes are the same everywhere — but the value set does not, because it is a different object maintained per environment. 4. In the new environment the key is absent. If it is required at start-up, the deployment fails. If it is read later, nothing happens at all until that path runs. There is a second route to the same place: a hand edit. Someone fixes a value directly in one environment during an incident, or adds a key to unblock a test, and the source of truth is never updated. The environment now has a value nothing will reproduce the next time it is rebuilt from that source. ## Why it stays invisible - **Nothing declares the contract.** The artifact knows which keys it reads, but if that knowledge lives only in code paths, the delivery system has nothing to check a supplied set against. - **Nothing compares the sets.** Values are supposed to differ per environment, so a naive diff is all noise; teams stop looking, and the signal that matters — a difference in the *key names* — is buried in it. - **Absence is quiet.** A missing key usually reads as empty or as whatever default the code happens to have, and an unused default causes no symptom at all until the path is taken. - **The lower environments do not exercise everything.** The path guarded by the new key may only run under production-shaped data or scale. ## Making the key set a contract The values are meant to differ. The **keys** are not. That single distinction is what makes drift detectable: - Declare, in the artifact, the keys it requires and the keys it optionally reads. Required means no safe default exists. - Validate the supplied set once at start-up, before serving, and exit non-zero naming every missing required key — not the first one, all of them, so one deployment fixes the whole gap. - Keep the key set in one shared document with per-stage values or overlays, so that adding a key is a change to the shared structure and every stage must either answer it or inherit an explicit default. - Log unknown supplied keys rather than refusing to start on them, so a typo is discoverable without making staged rollouts brittle. ## Guardrails, roughly in order of payoff 1. **Start-up validation with a loud failure.** Turns the worst case (a silent gap discovered by a user) into the visible case (a deployment that never took traffic). Note what it catches: the keys the artifact declares, and only those. 2. **Key-set diff in the pipeline.** Render every stage's resolved set and compare key names across stages. Different keys is an error; different values is the design. 3. **One path for value changes.** A value change travels the same staged route as a code change instead of being applied where it was needed. That alone removes most of the drift. 4. **Re-render rather than edit.** Treat hand edits in an environment as drift by definition: fix the source of truth and re-render, so the next deployment does not quietly undo the fix. 5. **Safe defaults where they exist.** A new key with a genuinely safe default degrades gracefully in environments that have not adopted it. A key with no safe default must be required — the mistake is inventing a default that is merely plausible. ## Values may differ; keys may not | Difference between two stages | Expected? | What it should trigger | |---|---|---| | Same key, different endpoint or account | yes | nothing; this is the point of one artifact | | Same key, different sizing or verbosity | yes | nothing | | Key present in one stage only | no | a failed comparison in the pipeline | | Required key missing in one stage | no | a non-zero exit at start-up, naming the key | | Key spelled differently in one stage | no | the unknown-key log plus the missing-key failure | The last row is worth dwelling on, because it produces both symptoms at once: the misspelled key is logged as unknown and the real key is reported as missing. A team that only implemented one of the two checks sees half the story. ## The interview point The answer that lands is not "someone forgot". It is the observation that one-artifact delivery deliberately concentrates *all* environment difference into the value sets, which makes those sets the thing that must be under the same discipline as code — declared, compared, and changed through one path — and that start-up failure is the good outcome you should be engineering for, not the incident.

  • Diffing two environments' value sets produces mostly noise, since values are supposed to differ. What do you actually compare?
    Compare key names, not values. Render each stage's fully resolved set, reduce it to its sorted key list, and compare those. A key present in one stage and absent from another is a finding; a value difference is the design working. Where a value's shape matters you can compare types or non-empty-ness rather than contents, which keeps credentials out of the comparison entirely.
  • A new key has a default that works fine in the lower environments. Should the artifact make it required anyway?
    Only if no value is safe everywhere. If a real default exists, ship it and let environments that have not adopted the key keep working. If the default is merely plausible — an empty endpoint, a zero timeout, a guessed limit — make it required, because a plausible default converts a configuration error into wrong behaviour that nobody is alerted about.
  • How do you clean up an estate where several environments have been hand-edited for months?
    Render every stage's resolved set, take the union of the key names, and work through the differences one row at a time: either the key belongs everywhere and the other stages get an answer, or it belongs nowhere and it goes. Then reconcile each surviving value back into the source of truth, re-render, and redeploy each stage so what is running is what the source produces.

saying these in an interview costs you the question

  • Calls it carelessness and proposes a checklist as the fix
  • Compares whole value sets and gives up on the noise
  • Invents a plausible default for a key with no safe value
  • Fixes the value in the environment and not the source
  • Assumes identical artifacts imply identical configuration
  • Reports only the first missing key and exits