A watchdog restarts your API process whenever its health check fails. It has been quietly restarting the process about three times a night for two months; last night the restarts could not keep up and the service was down for 40 minutes. What went wrong in the way that automation was built?

## Self-healing defers a problem; it does not solve one A watchdog that restarts a failing process is a legitimate and valuable control. What it is not is a repair. It converts a long outage into a short blip, and it converts an obvious failure into an invisible one. That trade is worth making — as long as the invisibility is deliberately undone by instrumentation. When it is not, the automation becomes a mechanism for accumulating hidden risk at a steady rate. The pattern in this scenario is the textbook one. Something in the process degrades over hours — a leak, a file-descriptor exhaustion, a connection pool that never drains. Restarting resets it. The reset is cheap enough that nobody sees it, so the underlying defect is never diagnosed, and it gets worse: the interval between restarts shrinks from eight hours to four to twenty minutes, and eventually the process cannot stay up long enough to be useful. ## What the automation should have emitted A remediator has three outputs, not one: 1. **The action itself** — restart the process. 2. **A record of the action** — a counter, incremented every time, with a label saying which condition fired. This is the difference between "three restarts a night" being a fact somebody could have noticed and being nothing at all. 3. **An escalation when the record looks wrong** — this is the part teams skip. The escalation has two tiers. A *ticket* tier: the first time a night's restart count exceeds the normal baseline, file work for the owning team; this is not urgent, but it must not be silent. A *page* tier: when the rate crosses a threshold that means the remediation is losing — restarts an hour rather than a night, or a failure to come back healthy after the restart. ## Alert on the remediation, not only on the fault The key inversion is that once you automate a fix, the fault stops being a good signal — the automation is designed to make it go away. The signal that carries information is now the *rate of remediation*. A team that only alerts on "the API is down" will never fire an alert, because the API is only down for four seconds at a time. A team that alerts on "the watchdog restarted the API more than three times in an hour" would have been paged in week one, at leisure, with a diagnosable process still running. ## Give the loop a ceiling A remediator should be allowed to give up. Encoding a limit — after N actions in a window, stop acting and escalate — does two things. It prevents the automation from grinding indefinitely against a fault it cannot fix, and it forces a human decision at exactly the moment when human judgement is worth more than another restart. In this incident, a ceiling would have converted a 40-minute unexplained outage into a page an hour earlier with a clear message: the watchdog has given up. ```bash #!/usr/bin/env bash set -euo pipefail state=/var/lib/watchdog/restarts hour=$(date +%Y%m%d%H) count=$(grep -c "^$hour" "$state" 2>/dev/null || echo 0) if [ "$count" -ge 3 ]; then logger -p daemon.err "watchdog: $count restarts this hour, escalating" exit 1 fi echo "$hour" >> "$state" systemctl restart api ``` ## The restarts were not free A process restart drops whatever it was doing. Three restarts a night means three bursts of failed requests, retries and elevated latency that real users experienced. Those errors belong in the service's own reliability measurement — if they never appeared there, the measurement is not capturing what users saw, which is a second defect hiding behind the first. ## Diagnosing it now Before the next restart, capture state: heap or memory profile, open file descriptors, connection counts, thread dumps. A watchdog that restarts on failure destroys the evidence every time, which is another reason silent remediation is expensive — two months of restarts produced zero diagnostic artefacts. A mature remediator captures a snapshot *before* it acts, so the loop that buys time also collects the material needed to stop needing it. ## How to answer in an interview Say plainly that the automation was not instrumented, and that the missing alert is on the remediation rate rather than on the fault. Then add the two senior touches: the ceiling that makes the loop escalate instead of grinding, and the diagnostic capture that makes the borrowed time useful. Avoid the weak answer — "they should have fixed the root cause" — which is true of everything and describes no mechanism.