A reader group was down six hours on a stream taking 10,000 records a second and can now drain 12,000 — when is its backlog of unread records gone, and what could still be lost?

This is the arithmetic a drain plan is made of, and it has two answers rather than one: when the group is level, and whether anything is destroyed before it gets there. ## Working the numbers ``` arrival rate R = 10,000 records/s drain rate D = 12,000 records/s outage 6 hours = 21,600 s backlog B = R x outage = 10,000 x 21,600 = 216,000,000 records surplus = D - R = 2,000 records/s time = B / surplus = 216,000,000 / 2,000 = 108,000 s = 30 hours ``` The same plan in time units, as a check. While the group is behind it is reading historical records, so it advances through `D / R` = 1.2 seconds of history for every second of wall clock. The age of the oldest unread record therefore falls at `D / R - 1` = 0.2 per second of wall clock, which is 0.2 hours per hour. Starting at six hours of unread age, that is thirty hours — the two methods agree, and disagreement between them would mean one input was wrong. ## The deadline is expiry, not patience Thirty hours is a long time to be behind, but the question that decides whether this is an incident or an inconvenience is different: **will the oldest unread record still exist when the group reaches it?** Records do not wait indefinitely. A cluster holds a **retained window** — a bound on how far back records survive — and designs differ in how it is expressed: as an age or size bound on a whole stream, as a per-subscriber bound, or as a maximum lifetime for an individual waiting record. In every case there is a boundary the drain is racing, and it is not one that patience wins. The useful property is that while the surplus is positive, the unread age falls monotonically. It starts at six hours and only shrinks, so as long as six hours was already inside the retained window when the drain began, nothing ages out during the recovery. Here the drain is safe — thirty slow hours, but not a lossy thirty hours. ## Where the loss actually comes from | Situation | Unread age behaviour | Outcome | |---|---|---| | Drain rate above arrivals | falls steadily | Group levels; nothing expires during the drain | | Drain rate equal to arrivals | flat | Never levels, and never loses records either | | Drain rate below arrivals | climbs | Reaches the retained window; records expire unhandled | | Readers stopped | climbs at one second per second | Fastest path to the boundary | So loss during a drain means one thing: the surplus went to zero or below for long enough for the unread age to reach the retained window. In this scenario that needs arrivals to exceed 12,000 a second — a daily peak, a retried flood from an upstream system that also just recovered, or a second incident. It is worth checking explicitly whether the thirty-hour window crosses a peak, because an estimate computed on an average arrival rate can hide a period of negative surplus inside it. ## Why this loss is quiet Expiry during a backlog is one of the least visible failures in broker operations, for three reasons: 1. **Nothing errors.** Removing records that have passed the retained window is routine housekeeping; the cluster is doing exactly what it was configured to do. 2. **On designs where expiry removes unread records, the unread count can fall with nothing processed at all.** A falling gap reads as recovery when it is actually data leaving the other end. 3. **The reading side sees no gap.** A reader continues from where it was; it is not told that the records between its position and the new oldest record ever existed. The defence is to plan the drain against the smaller of the two deadlines — the time to level and the time until the unread age reaches the retained window — and to state both when reporting. If the second is the binding one, the honest conclusion is that the current capacity plan does not recover this backlog intact, and the decision that follows is not an operator's alone. ## What this question is really testing That you treat catching up as a plan with numbers in it rather than as waiting. An engineer who answers "thirty hours, and nothing is lost because the gap only shrinks while we are ahead of arrivals, but if the evening peak takes arrivals over twelve thousand this stops being true" has demonstrated the whole of it: the surplus, the cross-check, the second deadline, and the condition under which the safe answer stops being safe.