A page configured as 'average error ratio over the last hour exceeds 14.4 times the budget rate' fires during a ten-minute total outage of a 99.9% service. The outage is mitigated, but the page keeps firing for most of the following hour. Explain why, and how the standard fix works.

## The mechanism: a sliding window remembers An alert on "the average bad-event ratio over the last hour" is evaluated against a window that slides forward in time. The bad traffic does not leave that window when the incident ends; it leaves when it becomes older than the window length. Until then it keeps contributing to the average. Work the numbers for the scenario. The threshold is `14.4 * 0.001 = 1.44%` bad events. The outage contributes ten minutes at 100% errors. Assume steady request volume, so the one-hour average immediately after mitigation is roughly `10/60 = 16.7%` — more than ten times over the line. As minutes pass, less of the outage remains inside the window. The average falls below 1.44% only when the surviving bad minutes fall under `0.0144 * 60 ≈ 0.86` minutes, i.e. about 52 seconds' worth. That happens roughly 59 minutes after mitigation. The responder has fixed the problem and the pager still says it is broken. ## Why this is worse than an annoyance Three concrete costs. First, the alert stops carrying information: nobody looking at it can tell whether the burn is live or historical, which matters most during handoffs and escalations. Second, if the alerting system re-notifies on an interval, the responder is paged repeatedly for an outage they already fixed. Third — and this is the damaging one — people learn to silence or acknowledge that alert reflexively, and the next real burn arrives at a suppressed alert. ## The fix: require recency as well as significance The multi-window design ANDs a second condition onto the same threshold, evaluated over a much shorter window — conventionally a twelfth of the long one, so 5 minutes against 1 hour, or 30 minutes against 6 hours: ``` fire when avg_bad_ratio(1h) > 1.44% AND avg_bad_ratio(5m) > 1.44% ``` After mitigation the five-minute window is emptied of bad traffic within five minutes, the conjunction becomes false, and the alert resolves — even though the one-hour term is still overwhelmingly true. **Reset time is governed by the short window.** Detection is not slowed. During a real burn the short window crosses the threshold first, not last, because it is not diluting the current traffic with the previous fifty-five quiet minutes. The conjunction is therefore satisfied at essentially the moment the long window crosses, which is what makes the pairing free. ## Choosing the short window The twelfth is a heuristic balancing two failure modes. Too short — say 30 seconds — and the short term flickers across the threshold on ordinary jitter while the long term stays true, so the conjunction flaps: fire, resolve, fire again, minutes apart. That is a distinct pathology from the one you set out to fix and it trains people to distrust the alert just as effectively. Too long — say 30 minutes against a 1-hour window — and you have barely improved reset time; the alert still hangs around for half an hour after recovery. If the short window still flaps at a twelfth, the usual levers are a small pending or "for" duration in whichever alerting engine evaluates the rule, and a minimum bad-event count for low-volume services, rather than lengthening the short window back out. ## Related recovery behaviour worth knowing A partial mitigation produces a genuinely ambiguous state: the short window drops below threshold while the long window remains far above it. The alert resolves, which is correct — the burn has stopped — but the budget damage is done and the incident is not over. This is why burn-rate pages are a signal to respond, not the record of the incident: closing the alert is not the same as closing the incident, and the remaining budget is what tells you how much room you have left before the next one. Similarly, an intermittent burn that stops and restarts every few minutes will fire, resolve, and fire again, because that is literally what is happening to the service. The right response is to treat the repeated firing as the signal it is, not to lengthen the short window until the pattern is hidden.