Three outages this quarter had different proximate causes, but all three involved the same shared configuration service. How do you approach the analysis across them?

## The blind spot of per-incident analysis A postmortem is scoped to one event, and that scoping is deliberate: it keeps the analysis concrete and the findings actionable. It also guarantees that a class of problem is invisible. If a shared component contributes a little to many incidents but is the headline cause of none, every individual postmortem will correctly attribute the outage to something else, and the pattern will be discussed only in hallway conversations where nobody can fund anything. Seeing it requires a second, slower analysis whose unit is the set of postmortems rather than the incident. ## Making the corpus analyzable This only works if factors were recorded in a comparable form, which is a standard you have to set before you need it: - **Consistent tagging.** Contributing factors carry tags — the affected service, the class of factor (rollout policy, alert coverage, dependency behaviour, capacity), and the incident phase. Free-text-only postmortems cannot be aggregated without re-reading them all. - **Comparable durations.** Time-to-detect and time-to-mitigate defined the same way across incidents, so they can be summed and compared across quarters. - **Impact recorded in one currency.** Usually error-budget minutes or affected-user-minutes, so three incidents can be added together into a number that means something. Then the quarterly question is not "what caused each of these?" but "which factor appears in the most incidents, and which factor accounts for the most impact?" Those are often different components, and the second one is usually the better investment. ## The base rate trap The most common error here is confusing exposure with fault. A configuration service that every request path touches will appear in a large share of incident timelines by construction — it is on the path, so it is in the story. Two corrections: 1. **Normalize by exposure.** Incidents per dependent service, or per million requests served through it, rather than raw counts. A component involved in three of twelve incidents while serving every request may be more reliable than one involved in two of twelve while serving a tenth of traffic. 2. **Distinguish involvement from contribution.** Being present in the timeline is not the same as being a contributing factor. The test is the materiality one applied per incident: would a different behaviour from this component have prevented or reduced the impact? A component that merely propagated someone else's failure faithfully is not the finding — though "we have no fallback when it is unavailable" might be. ## The decision, and what it costs Assume the pattern survives. Now the choice is real: | Option | Cost | Risk | |---|---|---| | Continue per-incident fixes | Days each, already budgeted | The pattern continues; you pay repeatedly and learn nothing new | | Reduce blast radius (fail-static cache, last-known-good, per-tier isolation) | Weeks | Often the best value; bounded work, breaks the shared-fate link | | Harden the component itself | Weeks to months | Fixes the source but the dependency remains a single point of failure | | Remove the shared dependency from the critical path | A quarter or more | Highest cost, hardest to fund, sometimes the only real answer | The aggregate framing is what makes the middle options fundable. One incident of eleven minutes justifies nothing structural. Three incidents totalling ninety minutes of impact, a large fraction of the quarter's error budget, plus the on-call hours and the review time, is an argument a business can act on. Where a budget policy exists, exhausted budget attributable to one dependency is the strongest lever available, because the consequence is already agreed in advance rather than negotiated after the fact. ## Being honest about the limits Three data points is a weak sample and you should say so. Deciding to fund structural work on three incidents is a judgment call about downside risk, not a statistical conclusion, and it is more defensible when there is a mechanism that explains the pattern — a genuine shared-fate coupling — rather than only a correlation in the tags. Look for corroborating evidence too: near misses involving the same component, and pages that never became incidents. ## The organizational half The pattern often crosses team boundaries, which is exactly why nobody owns it: each team's own postmortem quality is fine. Somebody has to hold the cross-incident view — a periodic reliability review, a designated owner of the postmortem corpus — and be able to fund work outside a single team's roadmap. Naming that gap is usually a stronger interview answer than any particular technical fix, because it is the reason the pattern persisted for three incidents in the first place.