skip to content

In what order should a test run resolve configuration from defaults, files, process environment and explicit overrides, and why fix that order?

level: middleimportance: must knowfreq 58%

answer

  1. One object, built before the first case
  2. Layers stack from general to specific
  3. Each later layer replaces earlier keys
  4. Defaults, file, process environment, invocation override

basics

~20 s

Resolve configuration once at startup in one fixed, documented order: built-in defaults, then a checked-in file, then process environment variables, then explicit overrides, each layer replacing the last. A fixed order makes every run's effective settings predictable.

solid answer

~40 s

A run should build **one resolved configuration object** before the first case, then never consult a layer again. The conventional order runs from least specific to most specific: built-in defaults compiled into the harness, then a checked-in file, then variables in the launching process's environment, then explicit overrides passed on the invocation. Each later layer replaces the keys it sets and leaves the rest alone. Which order you pick matters less than the fact that it is documented and identical from every entry point, so an engineer can say what a given run used without re-running it. Wide-reaching layers sit low because they change rarely and affect everyone; the per-invocation layer sits on top because it is what a person reaches for when they need to deviate exactly once.

code

pseudocode · 10 lines
pseudocode
config = builtin_defaults()
config = overlay(config, read_file("suite-config"))
config = overlay(config, read_process_env(prefix = "SUITE_"))
config = overlay(config, read_invocation_arguments())

require_present(config, ["target_tenant", "worker_count", "artefact_dir"])
freeze(config)

# nothing below this point consults a layer again
run_suite(config)

go deeper

for a junior

Be ready to say that a run should read its settings once, at the start, and that a value given directly on the invocation beats one written in a shared file. Knowing the layers exist and that later beats earlier is enough here.

for a middle

Explain the mechanics: the layer order, that each later layer replaces the keys it sets rather than merging into them, and that types are parsed once at the boundary so a value's type never depends on which layer supplied it.

for a senior

Show that you would make the order identical from every entry point and would validate and freeze the resolved object before the first case. Expect to be asked what breaks when a helper reads a layer directly instead.

for a principal

Own the trade between a chain rich enough that teams can deviate and a chain simple enough that anyone can predict a value. Be ready to argue where structured keys may merge, who may add a layer, and how the estate keeps ordering uniform across many suites.

## What "configuration" means for a run Configuration is every value that is true of a **whole run** rather than of one case: where the suite points, how many parallel workers it starts, where it writes artefacts, which optional behaviours are switched on. It is not the data a case operates on, and it is not one case's own parameters. The distinction matters because run-wide values are the ones a person changes from outside the code, and the only way to stop that becoming guesswork is to give it a shape. The shape is one sentence: **resolve every value exactly once, at startup, into a single object, and hand that object to everything downstream.** A run then has an answer to "what did you actually use?" that is a lookup rather than an archaeology exercise. ## The layers, least specific first Layering exists because the same suite runs in several situations, and each situation wants to change a different amount. A conventional stack, ordered from the layer that changes least to the layer that changes most: | Layer | Comes from | Changes | What it is for | |---|---|---|---| | Built-in defaults | values compiled into the harness | almost never | the value that is right in most situations | | Checked-in file | a file committed beside the suite | per release | the team's agreed settings, reviewed like code | | Process environment | variables set by whatever launches the run | per machine or per job | what differs by *where* the run happens | | Invocation overrides | arguments on the command that starts the run | per run | one deliberate deviation, right now | Each later layer **replaces** the keys it sets and leaves every other key alone. So a suite with sensible defaults needs no file at all; a team that disagrees with one default writes one line in the file; an engineer chasing a problem sets one override on their own invocation and changes nothing for anybody else. ## Why this order rather than another Two principles pick it, and they agree with each other: 1. **Specificity wins.** A layer that speaks for one invocation is a narrower statement than a layer that speaks for every run everywhere, and the narrower statement should win. Inverting this — letting the shared file beat an explicit override — makes the override silently do nothing, which is the single most confusing failure in this whole area, because the person is looking straight at the value they set. 2. **Audience width.** Defaults are authored by whoever wrote the harness and affect everyone. The file is authored by the team and reviewed. The process-environment layer is set by whoever configured a machine or a job. The override is typed by one person for one run. Reach narrows as you climb, so blast radius shrinks as authority grows. Which order you pick matters less than the fact that it is **fixed, documented, and identical from every entry point**. A suite whose stacking differs between a hand-started run and an automated job has no reproducible behaviour, and that difference is always discovered the expensive way. ## Replacement, not silent deep merging When two layers both set a key whose value is a structure rather than a single scalar, there are two possible rules: replace the whole structure, or merge it field by field. Whole replacement is the safer default because it keeps the prediction rule trivial — *the value in force is the one from the highest layer that mentions this key* — so a reader answers "what is this value?" by looking in one place. Deep merging produces composite values that exist in no layer at all, and nobody can point at where a given field came from. If one key genuinely needs merging, make that the declared behaviour of that key rather than the ambient rule for everything. ## What a fixed chain buys you - One place to **validate**: every required key is checked after resolution and before the first case, so a missing value is one loud failure instead of a scatter of confusing ones. - One place to **echo**: the resolved object is printed at the top of the run's output, which is what makes a past run explainable at all. - One place to **freeze**: after resolution nothing writes to the object, so two cases in the same run cannot disagree about a value. - One thing to **hand down**: cases and helpers receive the resolved object instead of each reaching for a mechanism of its own, which is also what makes them exercisable in isolation. - One thing to **diff**: two runs that behaved differently compare as two resolved objects, which usually ends the argument in seconds. ## Where teams lose it - A helper reads a raw process variable directly, bypassing the chain entirely, so the resolved object is no longer the whole truth. - The file layer is allowed to win over an explicit override, so overrides appear broken and people stop trusting them. - Two entry points into the same suite stack the layers in different orders. - Types that depend on which layer supplied the value: a number in the file, text from the process environment, and a comparison inside a case that quietly does the wrong thing. Parse into declared types at the boundary, once.

  • Two layers set the same key with different types, a number in the file and text from the process environment. What should resolution do?
    Parse against a declared type at the boundary, in one place, for every layer, and fail if the text cannot be parsed. Layers that arrive as text otherwise deliver values whose type depends on which layer supplied them, and a comparison inside a case then quietly does the wrong thing. Declaring the type beside the key also gives validation something to check before the first case runs.
  • Where should a single case that needs a larger batch size than the rest of the suite express that?
    In the case itself or its declared metadata, not in the run-wide chain. Run-wide configuration answers what this whole run points at and how it behaves; one case's requirement belongs where a reader of that case can see it. Pushing it into a global layer makes every other case's behaviour depend on which case happened to run, and removes the local reason from the place that needs it.

Four people write on the same whiteboard in an agreed order: the last one to touch a line owns it, and everybody knows in advance who goes last.

saying these in an interview costs you the question

  • Says every layer merges, so no layer ever wins
  • Lets the shared file override an explicit invocation argument
  • Reads layers again mid-run, whenever a value is needed
  • Treats the ordering as arbitrary and per-entry-point
  • Assumes a missing value should silently become empty text