After an outage, what must a reader group's reading rate exceed before its backlog of unread records starts to shrink?

A **backlog of unread records** is the set of records that exist in a stream and have not yet been handled by the reader group responsible for them. It appears whenever the reading side stops, slows or crashes while the writing side carries on, and it is the normal aftermath of a deployment gone wrong, a downstream outage or a night of undersized capacity. The moment the readers are healthy again, the operational question is not *are they running* but *are they running fast enough*, and that turns out to be arithmetic rather than judgment. ## Three numbers describe the whole situation - **Arrival rate (`R`)** — records written into the stream per second. This is the number people forget, because writers are unaffected by how far behind the reading side is. - **Drain rate (`D`)** — records the reader group actually completes per second, measured end to end, including the downstream work each record triggers. Not what the readers could fetch; what they finish. - **Backlog (`B`)** — the unread count at the moment the drain starts. Where a stream is measured in time rather than records, the equivalent is the **unread age** of the oldest unread record, and `B` is roughly that age multiplied by `R`. The backlog falls at `D - R` records per second. That difference is the drain; `D` on its own is not. ``` surplus = D - R time to close = B / (D - R) example: R = 4,000/s D = 5,000/s B = 18,000,000 surplus = 1,000/s time = 18,000,000 / 1,000 = 18,000 s = 5 hours ``` Read with `D` instead of the surplus, the same example gives one hour — a five-fold under-estimate, and the reason so many recovery estimates are announced and then quietly revised. ## Why steady-state capacity never recovers Most reader groups are sized so that they keep up with normal traffic and no more, sometimes with a modest safety factor. That sizing gives `D` approximately equal to `R` and a surplus of approximately zero. Such a group, restarted after a two-hour outage, is not broken and is not falling further behind — it is simply frozen at the depth the outage created. Every record it handles is matched by a new arrival. Nothing on a dashboard looks like an error; the gap is flat rather than rising, and it stays flat for as long as you leave it. Recovery therefore requires deliberately creating capacity that does not exist in normal operation, which is what makes a drain a plan rather than a wait. ## The same arithmetic in time units Where the gap is tracked as the age of the oldest unread record rather than a count, the drain shows up differently. While the group is behind, it reads historical records, advancing through `D / R` seconds of history for every second of wall clock. The unread age therefore changes at `1 - D / R` per second: it falls only when `D > R`, holds when `D = R`, and climbs at up to one second per second when the readers are stopped altogether. | Quantity | What it is | Behaviour while `D > R` | |---|---|---| | unread count | records not yet handled | falls by `D - R` each second | | unread age | age of the oldest unread record | falls by `D / R - 1` each second | | arrival rate | records written each second | unchanged by the drain | Both units give the same answer for when the group is level, which is a useful cross-check on a plan: if they disagree, one of the two inputs is wrong. ## Peak, not average A surplus computed against a daily average is optimistic for the hours that matter. If traffic doubles for three hours each evening, a group with a small surplus at the average rate has a **negative** surplus during the peak and gives back part of what it gained. Compute the plan against the rate that will actually be arriving during the drain window, and if that window spans a peak, say so in the estimate rather than discovering it. ## What varies between platforms The arithmetic is universal, but what limits `D` is not. Where a stream is split into a fixed set of parts and each reader takes some of them, the number of readers that can usefully work at once is capped by the stream's shape, and the surplus is capped with it. Where readers merely compete for records from a shared queue, there is no such cap, and the ceiling appears somewhere else — a broker-side rate ceiling on the client, or, more often than anyone expects, the downstream system each record is written into.