A deployed service is using a configuration value nobody put in its configuration file - how do you find which source supplied it?

"Where did this value come from?" is the signature incident of layered configuration. It is answerable mechanically, and the discipline is to enumerate rather than theorise. ## Establish what the process actually sees Before hunting sources, confirm the symptom. The value you are chasing must be read **from the running process** - an operational endpoint that reports effective settings, a boot log line, or a diagnostic that dumps the merged model with each entry's origin, with sensitive values redacted. Two mistakes happen here constantly: reading the value from a config file and assuming that is what the process uses, and reading it from a *different instance* than the one misbehaving. If the service cannot report its own effective configuration with origins, that absence is the root cause of the incident, not a detail of it. Adding that view converts every future occurrence from an investigation into a lookup. ## Reconstruct the source order Write down the sources this service registers, lowest to highest, from its startup path rather than from memory: - code defaults; - the file packaged in the artifact; - any environment-specific override set, and which one is active; - external files supplied by the deployment; - remote or secret-store sources fetched at boot; - environment variables; - command-line arguments. Then check the key in each, top down. The first layer that defines it is your answer - and if the top layer that defines it is not the one you expected to win, you have found either a precedence misunderstanding or a stray definition. ## The usual culprits | Symptom | Likely source of the surprise | |---|---| | Value differs per instance, same deployment | Environment variable injected per instance, or a command-line argument | | Value is right locally, wrong in one environment | Wrong profile resolved, or an environment-only external file | | Override "does nothing" at all | Key name does not match after translation, or the layer ranks below the one that wins | | Value reverts after a restart | A one-off command-line override that was never persisted | | Value is a plausible default nobody set | No source defines the key; a code default is in force | ## Five checks that resolve most cases 1. **Inspect the environment of the running process**, not the deployment manifest. Manifests describe intent; the process environment is fact, and platforms inject variables of their own. 2. **Confirm the resolved profile**, by name, from the process. A misspelled or unset profile name loads no override set and produces exactly the "base defaults in production" symptom. 3. **Re-derive the key name across namespaces.** Hierarchical keys map onto flat environment-variable names by a convention - case, separator substitution, sometimes a prefix. A name that misses the convention by one character defines a brand-new key that nothing reads, while the old value keeps winning. 4. **Look for sources outside the repository**: a mounted file, a value fetched from a remote store at boot, a platform-supplied variable. These are invisible to anyone reading the codebase, which is why they produce the most confident wrong answers. 5. **Check whether the value is read once or re-read.** A setting bound at boot cannot have changed since; a setting re-read on each use may have been altered by a dynamic source after startup, which changes the question from "which source" to "which source, when". ## Distinguish precedence from binding A value that is present but not *in effect* has three distinct explanations, and conflating them wastes time: a higher layer overrode it (precedence), the key was never claimed by any bound field so nothing reads it (binding), or it was read into a field whose default replaced it because conversion failed silently in a permissive implementation. Checking whether the key appears in the effective model *at all* separates the first case from the other two immediately. ## Close the loop Fixing the value is half the job. The durable outcome is: - an effective-configuration view with origins, redacted, available in every environment; - one documented source order shared across services, so intuition transfers; - strict rejection of unknown keys under bound prefixes, so the silent-typo class stops existing; - a boot log line naming the active profile and the sources that were actually loaded. Every one of those turns a multi-person investigation into a single lookup, which is the only real defence against a mechanism whose failure mode is a value that is simply *there*.