How does an infrastructure-as-code tool actually detect drift — what does it compare against what, and how does that differ between tools that keep a recorded state and tools that do not?

## The three states Drift detection is a comparison between three descriptions of the same resource: - **Declared** — what the configuration files say should exist. - **Recorded** — what the tool believes it last created or last saw, kept in a state file, in a backend, or on the platform's own servers. - **Observed** — what the provider API returns when asked right now. Detection begins with the observation. The tool walks the resources it manages, calls the provider's read APIs for each one, and refreshes its picture of reality. Only then can it compare. ```json { "declared": { "instance_type": "t3.medium", "monitoring": true }, "recorded": { "instance_type": "t3.medium", "monitoring": true }, "observed": { "instance_type": "t3.large", "monitoring": true } } ``` Here `recorded` and `observed` disagree on one attribute: that is drift, and nobody edited the code. If instead `declared` and `recorded` disagreed while `recorded` matched `observed`, that would be an ordinary pending change waiting to be applied. Distinguishing those two is the entire practical value of keeping a record. ## Why the record earns its keep A recorded state buys three capabilities that pure declaration cannot provide: 1. **Identity.** The record maps a name in your code to a concrete provider identifier, so the tool knows *which* object to read. 2. **Deletion tracking.** If the record lists an object and the provider says it does not exist, the tool knows it was deleted out of band. Symmetrically, if you delete a block from the code, the record still names the object, so the tool knows to destroy it. Without a record there is nothing to compare a removal against. 3. **Change attribution.** As above — telling "you changed this" apart from "the world changed this". ## Stateless tools Configuration-management tools in the Ansible style keep no record between runs. Each run gathers the current condition of the target directly and converges only the things the playbook names. This is a perfectly coherent design: the machine itself is the source of observed truth, so there is nothing to lose or corrupt, no locking problem, and no adoption step. The costs follow directly from the missing record: the tool cannot report drift as a standalone read-only question separate from a run, cannot tell you about resources you never mentioned, and cannot clean up something you removed from the playbook — you must write an explicit "absent" instruction instead. Platform-side state is a third shape: the service keeps the record for you on its servers, so drift detection is an API call to the platform rather than a local file read, and the record cannot be lost by an engineer's laptop. Continuously reconciling controllers are a fourth: they run the same comparison on a loop and act on it within seconds instead of when a human happens to run a command. ## What limits detection Detection is only as good as the read path, and every limitation here shows up as a real-world complaint: - **Write-only and unreadable attributes.** Passwords, secret values and some generated fields are never returned by the provider. The tool cannot see whether they drifted, so those attributes are structurally invisible. - **Attributes the tool does not model.** If the provider gained a field the plugin does not know about, changes to it are not drift as far as your tooling is concerned. - **Normalisation and semantic equality.** A policy document reordered by the provider, a value case-folded, a duration expressed differently — all produce differences that are textually real and operationally meaningless. This *phantom drift* is the biggest source of distrust in drift reports. - **Cost and rate limits.** Refreshing thousands of resources is thousands of API calls. On a large estate, detection is slow and can trip provider throttling, which is why scans are often scheduled rather than run on every command. - **Eventual consistency.** A resource read moments after a change may report an intermediate value and look like drift when it is simply not settled yet. - **Scope.** Detection covers what the tool manages. A resource created by hand and never adopted is invisible: there is no record of it, so nothing compares it to anything. Finding *unmanaged* resources is a different job, usually done by an inventory or cloud-security tool scanning the account. ## The point to make in an interview Say the three states out loud, name which pair defines drift, and then show you know that detection is a *read* — its accuracy is bounded by what the provider will tell you and how faithfully the tool compares it. That framing explains phantom drift, invisible secret fields, and slow scans in one move, without depending on any particular tool's command surface.