A shared database primary fails. Its own alert fires, and so do the alerts of thirty dependent services, paging thirty teams at once. How would you suppress the downstream pages, and what goes wrong with dependency-based suppression?

## The problem: alerts in a dependency relationship Grouping collapses alerts that are *peers* — the same symptom on many instances. It does nothing for alerts in a *dependency* relationship, where a single failure at the bottom of the stack causes correct, distinct, differently-owned alerts to fire above it. Thirty teams paged for one database is not thirty incidents; it is one incident and twenty-nine notifications that will produce no useful action. The cost of getting this wrong runs both ways. Leave it alone and every shared-infrastructure failure becomes an org-wide pager storm in which the one team that can fix it is drowning in cross-talk. Suppress too eagerly and a team sits unaware through an outage that was genuinely theirs. ## The mechanism: inhibition Dependency suppression is usually expressed as an inhibition rule with three parts: - a **source matcher** — which firing alert does the suppressing (for example, the database-down alert at critical severity); - a **target matcher** — which alerts get suppressed while it fires (the dependent services' error-rate alerts, or everything at warning severity from services labelled as depending on that datastore); - a set of **labels that must be equal** on both — cluster, region, environment. That third part is the one candidates forget, and it is the one that keeps the rule safe. Without it, a source alert anywhere suppresses targets everywhere: a database failure in staging silences production symptoms. With it, suppression is scoped to the blast radius the dependency actually has. Crucially, inhibition suppresses **notification**, not the alert. The downstream alerts still evaluate, still fire, still appear in the alert console and still land in the incident timeline. That matters during the postmortem, when you want to know which services were affected and when. ## Failure modes worth naming in an interview **1. The dependency graph drifts.** An inhibition rule is a hand-written model of who depends on whom, and it is correct on the day it is written. Services move off the shared database, new ones arrive, a cache is inserted in front. Nobody updates the inhibition rules, because nothing fails when they are stale — the failure is silent and shows up only as a page that never arrived. Any inhibition rule set needs an owner and a review cadence, and ideally a source of truth (service catalogue, dependency labels emitted by the services themselves) rather than a hand-maintained list. **2. The race on the firing edge.** Inhibition only works if the source alert is *already firing* when the target fires. If the database alert carries a two-minute pending duration and the downstream error-rate alerts carry thirty seconds, the downstream pages go out ninety seconds before there is anything to inhibit them. The fix is to make the source detect at least as fast as its targets — which sometimes means giving the cause alert a shorter pending duration than you would otherwise choose. A related trap is that inhibition works only where the alerts meet: alerts flowing through separate notification pipelines, or a per-service tool that notifies directly, never reach a common point where suppression can apply. **3. Over-broad matchers hide independent failures.** If the target matcher is 'any warning in production', then during any database incident *every* warning in production is muted, including one that has nothing to do with the database. This is the noise-reduction equivalent of a stuck-open valve, and it is worst precisely when it matters — during an incident, when a second, unrelated problem is more likely, not less. **4. The team that needed to know.** Suppressing a downstream team's page removes their notification but not their responsibility: their users are seeing errors. The suppression only works socially if there is a broadcast path — an incident channel, a status page, an automatic notification of affected owners — that reaches them without paging thirty people. Inhibition changes *how* they find out, and you have to design that replacement path deliberately. ## Alternatives and complements - **Alert on the symptom higher up.** If one objective-based alert on user-facing failure would have fired anyway, thirty component alerts add nothing at page severity and could be tickets. This is often a cleaner answer than a large inhibition rule set. - **Dependency-aware correlation** from a service graph, where the pipeline derives suppression from live dependency data rather than static rules. Better hygiene, more moving parts, and it fails in more interesting ways. - **Routing rather than suppression.** Route all downstream alerts of a known cause to one aggregated channel instead of thirty pagers — the information survives, the pages do not. ## What good looks like A small, owned set of inhibition rules for genuine shared-infrastructure dependencies — datastore, cluster control plane, network, a critical upstream — each scoped by equality labels, each with a source alert that detects at least as fast as the targets it suppresses, reviewed when the architecture changes, and paired with a broadcast path that tells affected owners what is happening without paging them. And an explicit acceptance that inhibition trades a small chance of a missed independent alert for a large reduction in cross-team noise — a trade worth naming rather than glossing over.