You are building automation that terminates and replaces any instance whose health check is failing. A monitoring bug briefly reports every instance in the fleet as unhealthy. What safeguards stop that from destroying the fleet?

## The failure mode being defended against A human executing a replacement on a bad signal replaces one instance, sees no improvement, and stops to think. A control loop replaces one instance, observes that the fleet is still unhealthy, and replaces another — forever, at machine speed. The judgement that bounded the damage was never written down anywhere, and closing the loop deleted it. Every safeguard below is an attempt to write that judgement back into code. ## Safeguard 1: distrust an implausible signal The most valuable guard is a precondition on the *shape* of the input rather than on any single reading. One instance failing its check is an ordinary event. Ninety percent of instances failing simultaneously is not a fleet event at all — it is overwhelmingly likely to be a broken check, an expired credential the health endpoint depends on, a DNS failure in the checker, or a bad deploy of the check itself. So the loop's first decision is: *is this signal believable?* If the unhealthy fraction exceeds a threshold, the correct action is to take no action and escalate. This inverts the naive design, where more evidence of failure means more aggressive remediation; here, too much evidence of failure means stop. ```bash if [ "$(( bad * 100 / total ))" -ge 20 ]; then logger -p daemon.err "remediator: $bad/$total unhealthy - signal implausible, refusing to act" exit 1 fi ``` ## Safeguard 2: rate limit the actor Even with a believable signal, cap how fast the loop may act: at most N replacements per hour, or per rolling window. The limit is derived from how quickly replacements can genuinely be needed in normal operation — if the fleet has never lost more than two instances an hour, two per hour is a generous ceiling and anything beyond it deserves a human. A rate limit converts a runaway from an outage into a page. ## Safeguard 3: cap the blast radius Separately from the rate, cap the *simultaneous* impact: never take out more capacity than the fleet can lose while still serving peak demand. If the service needs six instances at peak and runs eight, the loop may have at most one instance out of service at a time and still keep a spare. This is the same reasoning as redundancy headroom — running N+1 means one unit may be absent, N+2 means two — and the remediator must be aware of the headroom it is consuming, not just of the instance it is fixing. ## Safeguard 4: a circuit breaker on the automation itself The loop should watch its own effectiveness. If it has replaced three instances and the unhealthy count has not improved, its model of the world is wrong; continuing is destructive. Tripping a breaker — stop acting, hold the state, page — is the mechanical version of the human who stopped to think. A breaker that only resets on human acknowledgement is stronger than one that resets on a timer, because a timed reset resumes the runaway at 3am with nobody watching. ## Safeguard 5: verification between actions A loop that fires on a stale reading acts on a world that no longer exists. Re-read the health signal after each action, wait for the replacement to become healthy before starting another, and require the condition to persist for a duration before acting at all. A required persistence window is cheap and eliminates the entire class of transient blips — the monitoring bug in this scenario might not even survive it. ## Safeguard 6: an audit trail and an off switch Every action, with a timestamp, the instance affected, and the reading that triggered it, written somewhere a human can read during an incident. Without it, the incident begins with an unanswerable question about who or what terminated the instances. And the off switch must be documented in the same place on-call already looks, and must have been exercised — an untested kill switch is not a control, it is a belief. ## Safeguard 7: earn the authority gradually Run the loop in observe-only mode first, logging the action it would have taken. Compare against reality for a few weeks. That measures the false-positive rate on live signals, which is the number that decides whether this automation is safe at all — and it is a number no staging environment can produce. ## How to answer in an interview Lead with the signal-plausibility check, because it is the guard that specifically defeats the scenario as posed, and many candidates only reach for rate limiting. Then layer rate limit, blast-radius cap, breaker, audit trail and off switch. Finish with observe-only rollout. The through-line to state explicitly: the remediator is only as good as its input, so it must be built to doubt its input.