An alert on a reader group's record lag fires on a single sample and pages nightly on write bursts; what does a usable condition need instead?

## An alert condition is three choices Teams talk about an alert as though it were a number. It is three separate decisions, and each one can be wrong on its own: - **The signal** - which measurement the rule reads, named precisely enough that two people would pick the same series: a reader group's record lag, the age of its oldest unread record, the copies-behind count, the wait time a request spends in a node's queue. - **The threshold** - the value that counts as a breach, together with its unit. `2,000,000 records` and `90 seconds of age` are different conditions on the same reader, and they fail at different moments. - **The sustained window** - how long the breach has to hold continuously before the condition fires at all. A rule with a signal and a threshold but no window is not a weaker alert; it is a different and mostly useless one. ## Why one sample proves nothing here Messaging signals move in steps, and how sharply depends on how the flow batches. Writers accumulate records and publish them together; readers fetch a batch, process it, and only then record their progress. Between two fetches, a reader's unread backlog grows by everything written in the gap, then collapses when the next batch lands. Plotted per second, a perfectly healthy reader draws a staircase whose peaks are the batch size. There is a second, more mechanical reason. Where the reader owns a stored position, lag is not measured directly - it is derived from the distance between the stream's newest record and that stored position, and the two are refreshed on different schedules. A sample taken between those refreshes can show a step that no reader ever experienced. So a single-sample rule fires on the phase of a staircase. The on-call learns within a week that the page means nothing, and the rule is then worse than no rule: it occupies the attention budget that a real condition would need. ## Choosing the window Derive it, do not round it to five minutes because five minutes looks tidy: 1. Measure the writer's burst interval - how long between the batches that arrive on this stream. 2. Measure the reader's cycle - fetch, process, record progress, fetch again. 3. Set the window longer than one full cycle of the two together, and preferably around two, so a breach must survive a complete drain to count. 4. Replay a week of history against the candidate condition and count how many times it would have fired. A condition nobody has back-tested is a guess. The same arithmetic sets the threshold's unit. A record-count threshold has to be re-chosen every time throughput changes; an age-based threshold survives a traffic change because it already asks the question the business asks - how stale is the oldest thing we have not handled. ## The window is detection delay | Window | What it does | What it costs | |---|---|---| | Shorter than one cycle | Fires on every normal batch | Trust; the page is ignored inside a week | | Around one to two cycles | Fires only on a breach that survives a drain | A detection delay equal to the window | | Much longer than the cycle | Very quiet | A reader that stopped dead goes unreported for the whole window | There is no setting that is both instant and quiet, and pretending otherwise is how a team ends up with a rule that fires at three in the morning and a rule that never fires, both watching the same signal. Pick the delay you can live with and write it down as a deliberate choice. Two practical details belong to the condition, not to the dashboard. First, decide what a gap in the data means: if the collector misses samples, does the breach count as broken, or does the condition hold? A condition that treats missing data as not-breaching goes silent exactly when the cluster is in trouble. Second, decide whether the window resets when a single sample dips below the threshold. Most evaluators require the breach to be continuous, which means a reader flapping either side of the line can stay under the alert forever; where that pattern matters, express the condition as a fraction of breaching samples inside the window instead. ## What varies across platforms The shape carries; the signal does not. Where readers own a durable position, lag comes in records and in age. Where the broker deletes a record once it is acknowledged, there is no position to subtract from, and the equivalent signal is the depth and the age of the unread backlog - but it is still a step-shaped number that needs a sustained window. Some platforms publish the number per reader group and per partition, others only a single coarse figure per queue; with a per-partition signal, the condition must also say whether it fires on any one part or on the total, and those two rules behave completely differently when one part is stuck.