How do you decide which Datadog monitors notify a human, and keep one failure from becoming a notification storm?

## Two different questions A platform that can monitor anything invites a specific failure: teams create monitors because they can, point them all at the on-call handle, and within a quarter nobody reads the notifications. Good monitor design separates two questions that look like one. **Should this condition be detected?** Almost always yes. Detection is cheap and a recorded state history is what you read during an investigation. **Should this condition interrupt a person?** Much more rarely. Interruption is expensive, and its cost is paid by everyone who is later trained to ignore the channel. Datadog makes the distinction concrete. A monitor evaluates a query on a schedule, compares it with its thresholds, and transitions between states. Notification is a separate act, driven by **the recipients addressed in the monitor's notification message**. A monitor with no recipient in its message is a fully working monitor that records state and appears on the monitor list without waking anyone. This is a feature, not an incomplete configuration, and using it deliberately is most of what separates a curated alerting estate from a noisy one. ## What to check before a monitor is allowed to notify - **Is it a symptom or a cause?** Notify on what users experience. A cause-level condition - a queue depth, a saturated cache - is better recorded, then read during the investigation the symptom triggered. - **Is there an action?** If the responder's honest first move is to look at a dashboard and wait, the monitor is telling them something they cannot act on. - **Is it already covered?** Five monitors that all fire on the same underlying failure are one monitor and four notifications. - **Does it distinguish severity?** A warning threshold alongside the alert threshold lets a condition be visible before it is urgent, without a second monitor. - **What does it do when the data stops?** A monitor that treats missing data as a breach fires constantly on a sparse or batch-driven metric; one that ignores missing data stays silent through the outage that stopped the data arriving. Choose explicitly, per monitor. ## The three storm controls **1. Grouping.** A Datadog metric monitor whose query groups by a dimension becomes a multi-alert monitor: each group is evaluated and alerts independently. This is powerful and it is the most common self-inflicted storm. Group by host on a condition caused by a shared dependency and one failure produces one notification per host. The rule is to group by the smallest unit a responder would treat **differently**: grouping by service when each service has a different owner is useful; grouping by container instance almost never is, because nobody responds per container. **2. Composite monitors.** A composite evaluates the states of other monitors together, so a symptom monitor can be made to notify only while its known dependency monitor is healthy. This is how you stop a database outage generating a notification from every service that talks to it, without giving up per-service detection. **3. Scheduled downtimes.** A downtime mutes notification for monitors matching a scope - typically a tag such as an environment or a service - for a window, optionally recurring. The essential detail is that a muted monitor **keeps evaluating**: it still transitions state and still recovers, so you get an accurate history of what happened during a maintenance window rather than a hole in it. Scope the downtime by tag rather than by listing monitors, so monitors created next month are covered without anyone remembering to add them. ## Putting it together on a real estate | Condition | Notify? | Why | |---|---|---| | Booking confirmations fall below expected rate | Yes, grouped by service | User-visible symptom with a clear owner | | One node's disk filling | No, recorded only | Handled by automation; read during investigation | | Payment provider returning errors | Yes, single alert, ungrouped | One cause, one notification, however many callers | | Latency above target on a canary build | Warning threshold only | Visible without interrupting; rollback is a decision | | Nightly reconciliation job did not report | Yes, with missing-data handling on | Silence is the failure mode for a batch job | For a 41-service estate, the discipline that holds up over time is that every notifying monitor has a named owner and a stated response, everything else records, deploy windows are covered by a recurring downtime scoped by tag, and a monthly review deletes monitors that have notified without ever changing what anyone did. Notification volume is a design output, and it is the number worth watching.