A rollback of a bad release finished two minutes ago and the server-side error-rate graph is falling. What do you check before you call the rollback successful?

## Why "the graph is falling" is not enough A falling error rate two minutes after a rollback is weak evidence. It is consistent with recovery, and it is equally consistent with a partial rollback, with clients giving up, with a metric computed over a window that has not caught up, and with the aggregate hiding a cohort that is still broken. Verification is a short checklist, and running it is the difference between an incident that ends and one that reopens twenty minutes later. ## 1. Did the rollback actually land everywhere? Partial rollbacks are the single most common reason a rollback "didn't work": - one instance or node failed to cycle and is still running the bad version; - a second region, cluster or cell was never targeted by the rollback command; - an autoscaling group is still launching instances from a template that points at the bad release; - an edge or CDN cache is still serving the bad client bundle, so users get the new front end against the old backend; - a controller reconciling against source control is quietly putting the bad version back because the revert has not landed. The check is mechanical: emit the running version as a label on your request metrics and confirm that traffic on the bad version has gone to zero, rather than trusting the deployment tool's "succeeded". ## 2. Are you reading a signal that represents users? Measure as close to the user as you can. Server-side error rate misses everything that failed before reaching your handler - a load balancer returning 5xx, TLS failures, timeouts where the client gave up first. If the release broke something upstream of your application, your own error rate may look fine throughout. And always read **request rate alongside error rate**. A falling error *count* with falling traffic often means clients stopped trying: retries exhausted, a mobile client backing off, an upstream circuit opening. Recovery looks like errors falling while traffic returns to its normal shape. Where you have them, success-rate ratios rather than raw counts protect you from this, but the traffic curve is the sanity check. ## 3. Compare against the baseline, not the peak During an incident everything is measured against the worst moment, and everything is an improvement on the worst moment. The question is not "is it better than five minutes ago" but "is it back to where it was before the release". Pull up the same window from the previous day or the pre-release period and compare shape, not just level. A useful framing here is the burn rate: at a burn rate of 1 you are consuming your error budget exactly as fast as the window allows, so recovery means the burn has come back to its normal baseline - not merely that it has come down from the incident. Watching burn return to baseline is more honest than watching a raw error graph flatten. ## 4. Give it a window with meaning How long you watch should be tied to how long the problem took to appear. If the regression only showed up after five minutes of traffic - a slow leak, a cache filling, a periodic job - then a two-minute observation proves nothing. Watch for at least the original detection latency, and preferably across one cycle of whatever periodic work the service does. ## 5. Look for residue The rollback restores your code, not the state the bad release created: - caches holding objects in the wrong format keep failing until invalidated; - messages already published in a bad shape sit in queues and will be redelivered; - rows written incorrectly during the window stay written; - alerts may still be firing on stale evaluation windows, and silencing them is not the same as fixing them. ## 6. Slice by the dimension the incident lived on Aggregates hide cohorts. If the failure was specific to one region, one tenant, one client version or one endpoint, the global success rate can look completely healthy while that cohort is still at zero. Re-run the check on the specific slice where the symptom appeared, and on the specific request path, before declaring it over. ## What "successful" should mean A defensible bar: every instance serving traffic is on the intended version; the user-facing SLI is at its pre-release baseline over a window at least as long as the original detection latency; request volume is normal; budget burn is back to baseline; the specific failing slice is healthy; and any residual bad state is either cleared or has a named owner and a follow-up item. Anything less and you are hoping, not verifying.