A configuration-management tool runs successfully against every host every night, yet two hosts that should be identical behave differently. Why does repeated in-place convergence still leave divergent servers, and how does replacing hosts instead of patching them remove that class of problem?

## What "green" actually asserts A configuration-management run compares the host against the resources it *currently declares* and fixes any that differ. That is a genuinely useful guarantee, and it is much narrower than "this host matches that host". Green means: every declared resource is now in its declared state. It says nothing about the far larger surface of the machine that the code has never mentioned. Divergence lives entirely in that gap, and it arrives through three doors. ## Door one: the removal problem Deleting code does not delete its effects. If a role once installed a debugging package, wrote a config file and added a cron entry, removing those tasks from the repository stops *new* hosts from getting them — while every existing host keeps them forever, because nothing in the run asserts their absence. Converging tools are one-directional by default: they assert presence, not absence, unless you explicitly declare the removal. That leaves a trap. To actually clean up, you must add an explicit "absent" declaration and then keep it in the codebase indefinitely, because you cannot tell which hosts have already run it. Estates that have been managed this way for years accumulate a graveyard of removal tasks that exist only to undo things nobody remembers adding — and if one is deleted too early, the divergence comes back on the hosts that never saw it. ## Door two: time is an undeclared input Anything not pinned to an exact version resolves at the moment of execution. "Install the latest" on a host built in March and on a host built in September gives two different builds, both reported green. The same applies to anything fetched from a moving reference — a branch instead of a tag, a floating base image tag, an artifact endpoint that always serves "current". The host's build date becomes a hidden input to its configuration, which is why the phrase "but the code is identical" is never sufficient evidence that two hosts are identical. Partial failure compounds it. A run that fails halfway leaves a host in a state neither the old nor the new code describes, and if the failure is retried later the intermediate state may persist under the successful run that follows. ## Door three: everything nobody declared The hot-fix applied at 3am during an incident. A kernel parameter tuned by hand and never written back. A file left in place by a support engineer. A package pulled in as a transitive dependency of something else. A log directory that filled a disk on one host and not another. None of it is visible to a run that only inspects what it declares, and all of it can change behaviour. ## How replacement removes the class of problem The immutable answer does not detect divergence better — it removes the accumulation that causes it. A new instance boots from one artifact with no history. There is nothing left over from a previous configuration, because there was no previous configuration. Three properties follow: - **Identity becomes checkable.** "Are these two hosts the same?" collapses to comparing image identifiers, instead of diffing file trees on live machines. - **Divergence is time-bounded.** Whatever an operator does to a running host disappears the next time that host is replaced, so the maximum age of any undeclared change is the fleet's replacement cadence. - **Removal is free.** Deleting the line that installed a package genuinely removes it from the next image, because the image is rebuilt from scratch rather than converged forward from its own past. ## The honest caveat: the problem moves to the build Immutability does not delete the issue, it relocates it. If the image build itself is not reproducible, two builds of the same source produce different images and you have snowflake *images* instead of snowflake hosts — the same failure, one layer up, where at least it is visible in a pipeline rather than hidden on a machine. So the same discipline applies at build time: pin package and dependency versions, prefer immutable references (a digest or tag over a branch or a floating tag), and record a manifest of what actually went into each image so you can answer "what changed between these two builds?" without guessing. And note that replacement changes what the on-host tooling is *for*: instead of converging long-lived servers forever, the same code runs once during the image build, in a controlled environment, where a failure is a red build rather than a partially-configured production host.