Once a Grafana alert is firing, how does it reach a particular destination? Explain the role of notification policies and contact points, and how grouping and timing settings shape what actually arrives.

## Labels are the interface The single most important idea is that notification routing never mentions rules. An alert arrives at the notification pipeline as a label set plus annotations, and everything downstream — routing, grouping, silencing, muting — matches on those labels. That indirection is what lets a platform team change destinations without editing hundreds of rules, and it is why label vocabulary (team, severity, service, environment, cluster) must be agreed before the alert estate grows. ## The policy tree Notification policies form a tree rooted at a default policy that matches everything. Evaluation is depth-first and first-match-wins among siblings: the alert is tested against each child policy's matchers in order, and the first child whose matchers *all* match takes it; the search then continues among that child's own children, and so on, until no child matches and the current policy handles the alert. Two consequences follow. Ordering matters — a broad matcher placed above a narrow one shadows it, which is the most common routing bug. And an alert that matches nothing lands on the default policy, so the default must be a real destination someone reads, not a placeholder. A policy can be marked to **continue matching** subsequent sibling policies. That is the deliberate fan-out mechanism: send everything to a long-term archive webhook while also routing by team, or copy every critical alert to an incident channel in addition to the owning team's channel. Without it, routing is strictly one destination per alert. Child policies **inherit** unset settings from their parent — grouping, timers and contact point — so a tree usually sets sensible defaults at the root and overrides only what differs. Inheritance is per-field, not all-or-nothing. ## Contact points A contact point is a named bundle of one or more integrations: an email address, a chat webhook, a paging service, a generic webhook, and so on. One contact point with three integrations means one routing decision delivering to three places. Each integration has its own message template, so the same alert can be terse in a page and verbose in a chat message. Contact points also carry the choice of whether resolved notifications are sent — worth turning off for high-volume informational routes and on for anything where humans track closure. ## Grouping and the three timers Grouping is what stops a fleet-wide failure from sending five hundred messages. - **Group by** names the labels that define a batch. Alerts sharing those values are one notification. Grouping by alertname and cluster turns "forty nodes disk-full in cluster A" into one message listing forty instances. Grouping by nothing puts every alert in one giant batch; grouping by an instance-unique label defeats grouping entirely. - **Group wait** is how long to hold a *new* batch before the first send, so alerts that erupt together arrive together. Tens of seconds is typical: long enough to collect the storm, short enough not to delay a page unacceptably. - **Group interval** is the minimum time before sending an updated message for a batch whose membership has changed — more instances joined, or some resolved. - **Repeat interval** is how long an unchanged, still-firing batch waits before being re-sent as a reminder. Set it short and it becomes noise people mute; set it long and a forgotten alert stays forgotten. Hours is the usual range, and it should be longer than the group interval. These are per-policy, so a paging route can be aggressive and an informational route lazy. ## Simplified routing Recent Grafana versions let a rule name a contact point directly, bypassing the tree for simple cases. It is convenient for a small setup and it undermines the indirection the tree provides — the destination now lives on the rule again. Use it for genuinely one-off rules; keep team- and severity-based routing in the tree. ## How to debug a route When an alert goes to the wrong place, do not read the tree top-down and guess. Take the alert's actual label set from the alert list, then walk the sibling order at each level asking which matcher matches first; the shadowing sibling is almost always the answer. If it goes nowhere, check whether a silence or a mute timing swallowed it before delivery, and whether the contact point's integration is actually healthy — a webhook returning errors produces no message and no obvious alert of its own.