A host that passed a baseline control now fails it: how do you tell whether the host moved or the baseline moved?

## Two sides, one finding A control failing today after passing last quarter tells you a comparison came out differently. It does not tell you which operand changed. There are exactly three possibilities and a good engineer eliminates them in order: 1. The **recorded baseline** moved — the control was tightened, a new rule was added to the profile, a threshold changed, or the control's applicability now covers this host when it did not before. 2. The **measured state** moved — the machine drifted, through a hotfix, a package upgrade or an agent writing state back. 3. The **measurement** moved — the check now reads a different place (the running kernel ruleset instead of the file on disk, the effective merged configuration instead of one file), or it reads something non-deterministic. Most teams debug only the second and lose an afternoon when the answer was the first or the third. ## Eliminating the baseline side The cheap discriminator is provenance on the result. Every measurement should carry the identity of the content that produced it: which revision of the control set ran, and ideally a hash of the specific control's logic. With that, *did the rule change* is a lookup rather than an argument. Without it, you are reduced to weak signals — for example, a whole fleet re-failing the same control in the same run is far more consistent with a content change than with every machine drifting simultaneously, whereas a single host failing a control the rest of the fleet still passes points at that host. Beware the softer forms of a baseline move. The rule text can be untouched while its *scope* changed: a host newly labelled as in-scope for a stricter role, an input or variable that supplies the allowed egress destinations updated to a shorter list, a control whose impact was raised so a result that used to be informational is now a failure. All of those move the recorded side without anyone editing the logic, and all of them produce a finding that looks exactly like decay. ## Eliminating the host side If the content is stable, diff the two states. What you want is not the pass/fail but the *observed values*: the egress ruleset the check saw last time against the one it sees now, the audit rule set as recorded against as measured. That diff names the field that moved, which is what makes the next conversation possible. Then attribute it. The three decay sources leave different traces: - A **human hotfix** usually correlates with an incident. The modification time on the ruleset, the shell history on the host, and the incident timeline will bracket the same window. This is where the on-call engineer who widened an egress rule at 03:00 shows up. - A **package upgrade** correlates with the patch window. The package database records when the owning package changed; a vendor copy of the file left alongside yours is a strong tell that the local file was preserved while the vendor's moved on. - An **agent write-back** correlates with a convergence run. Its log will say it enforced the resource, and the file's modification time will land on a convergence boundary rather than an incident. ## The measurement trap Host state is layered, and the two layers disagree more often than people expect. Audit rules are compiled from rule files and loaded into the kernel's audit subsystem; a rule loaded at runtime is in force but absent from disk, and a rule written to disk does nothing until it is loaded. Firewalls have the same split between the running ruleset and the saved one restored at boot. Proxy configuration can be set in a service's environment, in a system-wide file, and in the user's shell, with only one of them actually in effect for the daemon you care about. So a re-failure on a host whose files are bit-identical to last quarter is not a paradox. Either the check changed which layer it looks at, or a reboot flushed a runtime change into or out of existence, or the effective configuration is assembled from more inputs than the one file being diffed. When the file and the finding disagree, trust the layer that is actually in force and ask why the check and the machine disagree about where truth lives. ## What the answer should end with Both sides can move in the same interval, so *the rule changed* does not clear the host. Confirm each side independently rather than accepting the first explanation that fits, and record enough with each result — content revision, observed values, timestamp — that next quarter's version of this question takes minutes instead of a day.