A canary deploy caught a fatal bug before any customer saw an error. Do you analyze it like an incident, and what would that analysis look at?

## Why near misses are worth the time A near miss is an event where the conditions for a serious incident were present and something stopped it. In an outage you pay for the information with customer impact. In a near miss you get most of the same information free: the defect existed, the path to production was real, the failure mode is observable — only the impact is missing. Aviation and medicine formalized near-miss reporting decades ago for exactly this asymmetry, and software teams mostly have not, because the trigger for a review is usually "did the SLO burn?" rather than "how close was it?" The practical consequence is that organizations learn about their controls only when the controls fail. ## The selection criterion is margin, not outcome Reviewing every save would be its own kind of toil, and review capacity competes directly with real incidents. The useful filter is how much room there was: - **Designed control, wide margin.** The canary failed automated analysis at 2% of traffic within four minutes, exactly as intended. Log it, move on. This is the system working. - **Designed control, thin margin.** It failed analysis at minute 29 of a 30-minute bake. The control held, but a slightly slower manifestation walks straight past it. - **Undesigned save.** Someone happened to be looking at a dashboard. The canary sample happened to include the one shard with the affected data. A dependency happened to be slow, which throttled the rollout. None of these repeat on demand — and treating them as evidence the system is safe is how a fluke gets recorded as a control. An undesigned save should be reviewed with the same seriousness as a real outage, because the next occurrence has no protection at all. ## What the analysis asks The questions invert an incident postmortem: 1. **How close was it?** Quantify. What fraction of traffic was exposed, for how long, and how far from the failing threshold did the SLI sit? "It was fine" is not an answer; "errors reached 0.4% against a 0.5% abort threshold" is. 2. **What actually caught it — and was that its job?** Distinguish the control that was designed to catch this class of thing from whatever incidentally did. 3. **Would it have caught a variant?** Slower onset, a bug affecting 1 in 500 requests instead of 1 in 5, a failure only visible in a region the canary does not cover. Canary detection power depends on the exposed population and the bake duration; a bug rarer than your canary's sample size will not show up. 4. **What was the counterfactual impact?** If this had reached full production, what would it have cost — how many users, how much of the error budget? This is the number that funds any follow-up work, and without it a near-miss review produces findings nobody prioritizes. 5. **Would the next line of defense have held?** If the canary had passed it, would monitoring have paged quickly, and was the rollback path ready? ## Where near misses come from Beyond canaries: an alert that fired and self-resolved before anyone acknowledged it, a failover that worked but took far longer than expected, a disk that reached 92% before a cleanup job ran, a retry storm that stayed just under the point where the database would have tipped, a deploy rolled back on a hunch. Each carries the shape of a real incident minus the impact. A cheap way to surface them systematically is to ask, at the end of any routine review, whether anything nearly went wrong this week — because near misses have no alert of their own and nobody is paged when disaster does not happen. ## The failure mode to name Repeated near misses with no review are how a margin quietly disappears. Each save is read as proof the system is resilient, the acceptable operating point drifts toward the edge, and the population that would have flagged the drift is exactly the reviews nobody ran. The point of counting near misses is to notice the drift while it is still free to correct. ## In an interview Give a concrete near miss, state the margin as a number, and say what you changed as a result — the canary population, the bake duration, an alert that should have fired first. Saying "we would write a postmortem" is a policy statement; saying "errors peaked at 0.4% against a 0.5% abort threshold, so we shortened the analysis window" is an answer.