Host baseline controls across your fleet re-fail every quarter after legitimate work: do you engineer the decay out or accept it?

## Start by refusing the binary *Engineer it out or accept it* is the wrong shape for a fleet decision, because decay is not uniform. A few controls will re-fail on most hosts within weeks, and the rest will hold for a year. Spending the same effort on both is how programmes burn a quarter and move nothing. So the first move is measurement: for each control, how long after a host is built or rebuilt does it re-fail, and which of the decay sources caused it. That gives a short list, and the short list is the entire conversation. ## Then choose per control, from three options **Relocate the control.** The strongest answer for a fast decayer is to stop asking a host to hold a setting that operations must be able to change under pressure. If egress restriction is enforced at the network boundary rather than in each host's local firewall, an engineer fixing an outage on the host cannot widen it, because it is not theirs to widen. Decay for that control drops to near zero. The costs are real and you should name them: the control now lives with a different team, the coarser enforcement point may not express per-host intent, and the incident that used to be fixed in one minute now needs someone who can change the boundary. **Remove the ability to hand-edit.** Rebuild rather than repair: the host's state is the output of a build, interactive change is unavailable or short-lived, and any fix goes through the same path as a normal change. This genuinely eliminates the human-hotfix source and shortens the tail of the other two, and it is the option platform teams reach for. The bill is incident response: whatever your mean time to repair is, it grows by the length of the build-and-roll path, and the organisation may not accept that for its worst outages. Do not present this option without the incident cost attached. **Accept the decay and change the claim.** Perfectly legitimate, and often correct for controls whose decay rate is low or whose consequence is small. What is not legitimate is accepting decay while continuing to describe the control as continuously enforced. A control that is measured periodically and repaired after the fact is a detective control with a repair loop, and saying so plainly is stronger than a claim that the next re-failure will contradict. Precision here also protects the engineers: it stops every ordinary decay event being reported as a lapse. ## Who decides, and with whose budget This is why it is a lead's call rather than an engineer's. Three parties hold different halves of it. The **platform team** would fund rebuild tooling and image work and has its own roadmap. The **on-call rota** pays the incident-response cost of anything that removes hands-on repair, and they are the ones who will route around a control they cannot live with. Whoever **owns the claim** carries the consequence of the weaker wording. A decision made by only one of the three either does not get built, gets bypassed quietly, or produces a claim nobody can support. Bring the decay data to all three and let the tradeoff be explicit: this control costs X of platform work to make durable, or Y minutes of response time, or we say something weaker about it. ## Signals that you chose wrong Two failure modes are worth naming because interviewers probe for them. The first is a hardening programme that raises decay: every control pushed onto the host, none relocated, and the fleet now generates hundreds of quarterly re-failures that nobody triages, which trains everyone to ignore the results. The second is over-locking: interactive access removed on machines whose incidents genuinely need it, so engineers acquire an unofficial route back in and you have lost both the control and the visibility. If either shows up, the answer is not more enforcement, it is re-running the decay measurement and moving the specific controls that keep losing. ## The short version of a good answer Measure decay per control; expect a short list to dominate; for those, prefer relocating the control to a layer operations cannot edit, then rebuild-not-repair with the incident cost stated; accept the rest and downgrade the wording to match reality; and make the call jointly with platform, on-call and the claim owner rather than unilaterally.