Explain the states a Grafana-managed alert instance moves through — including its pending period — and what happens when the underlying query returns no data or fails outright.

## The normal path Each alert instance — remember, one per label set, not one per rule — carries its own state, and the rule's evaluation group drives it forward on a fixed interval. 1. **Normal.** The condition evaluated false at the last evaluation. 2. **Pending.** The condition evaluated true, but it has not yet been true for long enough. Nothing is notified. The instance is visible in the UI, which is useful when debugging a rule that never seems to fire. 3. **Alerting (Firing).** The condition has been continuously true for at least the pending period. This is the point at which the instance is handed to the notification pipeline. 4. **Resolved / back to Normal.** The condition evaluated false; the instance leaves the firing set and, if the contact point is configured to send them, a resolved notification is delivered. The pending period (historically written as a "for" duration) exists to convert a momentary spike into a non-event. Its interaction with the evaluation interval is the part people get wrong: the condition must be true at *every* evaluation during the window, and a single false evaluation resets the timer to zero. A pending period shorter than the evaluation interval therefore has no effect at all, and one that is not a multiple of the interval effectively rounds up to the next evaluation. If your data arrives every 60 seconds and you evaluate every 60 seconds, a 90-second pending period behaves like 120. ## No data An empty result is not a false condition; it is an absence of evidence, and treating the two as identical is how monitoring goes quiet exactly when the monitored thing dies. Grafana therefore makes it an explicit per-rule choice: - **No Data** (dedicated) — produces a distinct alert for the rule so a human sees "this rule stopped receiving data" rather than "everything is fine". - **Alerting** — treat absence as a breach. Appropriate when the series is guaranteed to exist while the system is healthy. - **Normal** — treat absence as fine. Appropriate for sparse, event-driven series where gaps are expected and would otherwise page constantly. - **Keep Last State** — hold whatever the instance was before. Useful across a known-flaky exporter, dangerous as a default because it can pin an instance firing or silent indefinitely. A subtlety: a no-data result usually has no labels to attach to, so the resulting alert is at rule level rather than per instance. And an instance that simply *disappears* from the query result (a host that no longer exists) is a different situation from the whole query returning nothing — Grafana stops tracking series that vanish, which is why "the alert stopped without resolving" sometimes has no notification behind it. ## Error A data-source timeout, an authentication failure or a malformed query produces an error rather than a value. The configurable outcomes mirror the no-data case: alerting (loud, correct when a silent monitoring system is worse than a false page), normal (quiet, acceptable only when someone else watches the data source), or keep last state. Errors are also where evaluation cost shows up: a query too heavy for the interval will time out under load precisely when the system is stressed, so a rule that is loud on error will page for the monitoring system's own overload. That is an argument for making the query cheap, not for making the rule quiet. ## Recovering and hysteresis A breach that oscillates around the threshold produces alternating firing and resolved notifications. Two mechanisms address it. Threshold expressions can carry **hysteresis** — a separate, lower recovery bound so an instance must clearly improve before resolving. And newer Grafana versions add a **Recovering** state with a keep-firing duration, so an instance that stops breaching remains firing for a configured period before resolving, absorbing brief dips. Both are preferable to simply raising the pending period, which delays the first real alert as well. ## Where the state ends up Instance state is persisted so it survives a restart rather than everything resetting to Normal and re-firing, and the state history is queryable, which is the fastest way to answer "was this flapping or genuinely down for an hour?". A paused rule is different from all of the above: it stops evaluating entirely and holds no state, which makes pausing a blunt tool compared with a silence.