Alert Noise Reduction

## The failure mode One underlying fault almost never produces one alert. A failing database primary produces a connection-error alert on every application instance that talks to it, and if you have 200 instances you get 200 notifications describing one problem. This is the single largest source of raw pager volume in most estates, and — importantly — it is not fixed by raising thresholds or by deleting rules. The rules are all correct. The problem is that the notification pipeline is treating instance-level facts as if they were separate incidents. ## Deduplication Deduplication answers a narrower question: are these two alerts *the same alert*? An alert's identity is its full label set. If the same alert arrives twice — typically because you run the alert evaluators in a redundant pair for availability, and both send — the notification layer keeps one and drops the other. Deduplication is what lets you run a highly available alerting stack without doubling the pager. It does nothing about our 200 alerts, because they differ in the `instance` label and are therefore genuinely distinct alerts. ## Grouping: the key and the timers Grouping is the mechanism for the 200. You declare a grouping key — a subset of labels — and every firing alert whose values match on that subset lands in the same group, and one group produces one notification. Choosing `alertname, service, cluster` puts all 200 instance alerts in one bucket; choosing `alertname, instance` puts each in its own bucket and changes nothing. Three timers usually surround the group. Alertmanager, for example, calls them `group_wait`, `group_interval` and `repeat_interval`, and other systems expose the same three ideas under other names: - an **initial wait** after the first alert of a new group arrives, so its siblings have time to show up and be included in the first notification; - a **batching interval** before a notification is re-sent because new members joined an existing group; - a **repeat interval** before re-notifying about a group that has not changed at all. A common shape is tens of seconds for the initial wait, a few minutes for the batching interval, and hours for the repeat. ## Choosing the grouping key The useful rule is: **group by the thing one responder would handle in one action.** If a single human fixing the database resolves all 200, they belong in one notification. If two alerts in the group would be worked by different people, or would be fixed by different actions, the key is too coarse. That pushes you toward keys built from ownership and blast-radius labels — service, team, cluster, region, environment, alert name — and away from identity labels like instance, pod or container, which are exactly the labels that multiply. Keep the multiplying labels *in* the alert (the notification should still list which 200 instances) but out of the *key*. ## What aggressive grouping costs Three costs, and an interviewer wants you to name them unprompted: 1. **Detection delay.** The initial wait is added directly to time-to-notify. If your severe-incident target is a five-minute response, a three-minute wait window is most of your budget. Different routes can carry different waits: a short one for the page path, a long one for the ticket path. 2. **A real alert hiding inside an acknowledged group.** Once a group has notified and the on-call has acknowledged it, an unrelated alert that matches the same coarse key joins that group silently until the batching interval elapses — and if the responder is already deep in the database problem, it is read as more of the same. This is why grouping on a single label like `cluster` is dangerous. 3. **Loss of routing fidelity.** A group is delivered to one destination. Group across services and you have to send the whole batch to whoever owns the group, not to the owner of each alert. ## Where grouping is the wrong tool Grouping collapses alerts that are *peers*. When the alerts are in a *dependency* relationship — the database's own alert plus every downstream service's alert — suppression based on that dependency is the better instrument, because it removes the downstream noise entirely rather than batching it. And when a single alert flaps on and off, neither helps; that is a rule-level tuning problem. ## The practical answer in an interview Say what you would actually do: keep the per-instance rule (it is correct and you want the detail), set the grouping key to alert name plus service plus cluster, keep the page-path initial wait short enough to protect your response target, and verify by replaying a week of real firings through the proposed key to count how many notifications it would have produced.