A nightly reader's unread backlog legitimately grows to millions of records and drains by morning, so no fixed record threshold works; how would you phrase the alert condition?

## Why a fixed level cannot work on this stream The fixed-threshold rule assumes the signal has one normal. This reader's normal is a cycle: a batch job writes for hours, the reader is deliberately behind by design, and by morning it has caught up. Any single number sits in one of two useless places. - **Above the nightly peak** - the condition is silent all night by construction, which also means it is silent on the night the reader dies at the peak. The team believes it is monitored and it is not. - **Below the nightly peak** - the condition fires every night on healthy behaviour, and within a fortnight the page is a ritual nobody reads. The instinct to split the difference produces the worst of both: an alert that is noisy enough to be ignored and quiet enough to miss the real event. ## Four condition shapes, and what each one catches | Shape | Fires when | Catches | Misses | |---|---|---|---| | Level | Backlog above a fixed number for a window | A genuine runaway on a steady stream | Anything on a stream whose normal varies by an order of magnitude | | Trend | Backlog has not fallen across a sustained window | A reader that is losing ground or has stalled mid-drain | A drain so slow it still finishes late | | Deadline | Backlog above a small number at a stated hour | The slow drain that misses the downstream commitment | Anything happening in the hours before that hour | | Progress | Stored read position has not advanced while records arrive | A dead or wedged reader at any backlog size, including near zero | A reader that is moving but far too slowly | They are complementary, not alternatives. The trend and the progress rule are the ones that page; the deadline rule is the one that expresses what the business actually cares about. ## Making a trend condition survive contact with reality A rate-of-change condition is easy to write badly. Three guards make the difference: 1. **Give it a floor.** A rule that fires because a backlog rose from eight records to eleven is arithmetically correct and operationally worthless. Require an absolute level to be exceeded before the trend clause is even evaluated. 2. **Compare across a window, not between two samples.** These numbers move in steps, so two adjacent samples can show any slope you like. Ask whether the backlog is lower than it was a window ago, or whether it failed to fall at any point during the window. 3. **Know which half of the cycle you are in.** During the writing phase a growing backlog is the design; during the drain phase it is a fault. If the rule cannot tell the phases apart, restrict it to the drain window, or use the deadline rule for the writing phase instead. ## The trap in a falling backlog A backlog that is going down looks like health and is not the same thing. It is also exactly what you see when the writer has stopped: nothing new arrives, the reader finishes what is left, and the number slides to zero and stays there. A condition set that contains only level and trend rules reads that as the best night it has ever had. This is why the progress rule matters and why it is expressed against two facts rather than one: records are still arriving, and the reader's stored read position is not moving. Either fact on its own is ambiguous. Together they describe a reader that has work and is not doing it - which is the thing you actually wanted to be told about, and the reason the condition set needs more than one rule. ## Setting the deadline number without guessing The deadline condition is the only one in the set whose number is not an engineering choice. It comes from the commitment the stream feeds: a report due at seven means the backlog must be effectively empty by half past six, so the condition is written against half past six and reviewed when the commitment moves. Record the commitment next to the threshold. A threshold whose justification is lost becomes untouchable - nobody dares raise it and nobody can defend it. ## What varies across platforms Where readers own a durable stored position, all four shapes are available and the progress rule is the cheapest of them. Where the broker deletes a record on acknowledgement there is no position to watch, and the equivalent progress signal is the acknowledgement rate against the arrival rate, or the age of the oldest unread record refusing to fall. Some platforms expose the backlog per reader group and per partition, in which case a total that is draining nicely can conceal one part that has not moved since midnight - on those, the trend and progress rules belong on the worst part, not on the sum.