You applied a mitigation five minutes ago and the dashboards look calmer. How do you decide the incident is actually mitigated, and what commonly makes an incident look recovered when it is not?

## Verify against the user, not the dashboard The dashboard you were staring at during the incident was chosen for diagnosis, and it is usually full of causes: CPU, queue depth, connection counts, error logs. Those going quiet is suggestive, not conclusive. The question that decides whether the incident is mitigated is narrower: **are users getting correct responses in acceptable time again?** That means the service level indicator — the proportion of valid requests served successfully, and the latency distribution of real user traffic — read at the granularity that failed. Granularity is where most false recoveries hide. If a single shard, one region, or one very large tenant is still failing, a global success rate can look almost perfect while a real population remains completely down. Whatever dimension the failure was scoped to — region, shard, customer, endpoint, client version, device platform — verify along that same dimension. And prefer a signal measured as close to the user as you can get: an independent synthetic probe or client-reported success beats a server-side counter, especially when the failure is in the network path or in the telemetry pipeline itself. ## The four classic false recoveries **1. The denominator collapsed.** Error ratio is errors divided by requests. If clients time out, retries exhaust, a mobile app stops retrying, or an upstream starts shedding, the request count falls and the ratio improves without a single user being better off. The fix is a habit: never read an error ratio without reading throughput beside it. A sudden drop in traffic during an incident is a symptom, not a relief. **2. The window has not turned over.** A rate or ratio computed over a five-minute window carries five minutes of history. Ninety seconds after the mitigation, most of what you are seeing is still the incident. The same lag applies to the alert that fires and clears on those metrics. Wait past one full evaluation window before treating an improvement as real — and remember that any alert clearing tells you the alert's condition is no longer met, which is a weaker claim than users being fine. **3. The backlog is still draining.** Mitigation often restores the front door while the damage sits behind it: a queue that grew to millions of messages, replication lag measured in minutes, a retry backlog that is about to arrive all at once, a consumer group far behind the tip. New requests succeed, so the SLI looks healthy, while the work accepted during the incident is still unprocessed or is about to re-overload the recovered service. Verify that backlog metrics are not merely bounded but **decreasing**, and estimate the drain time from the rate of decrease before you close anything. **4. The system is still cold or fragile.** After a rollback, restart or scale-up, caches are empty, connection pools are refilling and JIT-style warmup has been lost. Latency can remain well above baseline, sometimes badly enough to breach the objective on its own, and the system is more likely to fall over again under a load it used to handle. Watch latency percentiles, not just error rate, and expect a recovery curve rather than a step change. ## Say what "mitigated" means before you declare it A usable bar, agreed out loud in the incident channel: the user-facing SLI is back inside its objective, measured on the affected dimension, sustained for longer than one full metric window, with backlog trending down and no lever still masking the problem in a way that will expire. Naming the criterion before you look at the graph keeps optimism from doing the deciding. Then keep the two states distinct. **Mitigated** means impact has stopped — this is what ends the urgency, the paging and the customer-facing severity. **Resolved** means the cause has been addressed, which usually happens days later through the postmortem's action items. Between the two sits a third obligation people forget: **reconciliation**. Requests that failed during the incident may have left side effects — unsent notifications, half-completed orders, payments captured but not recorded, duplicated writes from retries. Those need identification and repair, and the incident is not finished just because the graphs are green. ## And keep the budget in view The error budget consumed during the incident does not come back when the mitigation lands. That number — how much of the allowance for the window the incident spent — is what turns "we recovered" into a decision about what the team does next, and it belongs in the incident record alongside the recovery time.