Consumer Operations & Lag

Three events: a member joins, a member leaves cleanly, or a member stops proving it is present — by missing its liveness signal or by holding work past its progress deadline. Designs where readers merely compete reassign nothing.

Unread count states the gap in records; unread age states it in time. Two million records may be seconds old on a fast stream or half a day old on a slow one, so only the age reading shows staleness.

Moving a reader group's read position forward abandons every record between the old position and the new one, so nothing processes them. The gap drops to zero at once because work was skipped, not done.

A slow member still completes records, just fewer than arrive, so its read position keeps advancing. A stuck member completes nothing: its position is parked. Only the slow case is answered with capacity.

Three levers: more readers, more records per request, and temporarily waiving in-order handling. Extra readers stop helping at the parallelism ceiling the stream's shape fixes, or wherever the shared downstream system saturates first — both walls arrive sooner than most plans assume.

A share may be held by only one member at a time, so the old holder must stop before the new one starts. How much of the reader group stops depends on the design: many stop every member until the division settles, others stop only the shares that move.

Flat means arrival and drain rates match, at whatever height. A steady rise means the drain rate is below the arrival rate, and the slope is the difference. A sawtooth means arrival or reading happens in bursts rather than continuously.

A stored read position and the records it points at live under two independent lifetimes. Many platforms discard the position after the owning reader group has been inactive long enough, while the records stay inside the retained window.

Handling is slower than the progress deadline, so a live member is declared gone; its share is reassigned, the group pauses, the member rejoins and starts the same slow work, and the cycle repeats. The group loses most of its time to the loop, not to the work.

Every effect downstream of the new position fires again — including the ones that leave the system, such as customer email, payments, third-party calls and records republished onward, which no reader can take back.

Four exits: fix the handler, set the record aside, discard it, or wait for a transient cause to clear. Fixing loses nothing but is slowest; discarding is fastest and loses the record silently. Adding capacity is not an exit.

Because only one of the two shapes assigns anything. Where a stream is split into parts and each part has one holder, a new member forces the division to be recomputed. Where readers compete for records from a shared queue, nothing is held, so nothing is reassigned.

A running member holds its own place in memory and writes it back, so it overwrites the move or carries on past it. Stopping the whole group first makes the stored position the only writer of record.

It depends on the unit of recorded progress. Where a share's progress is one advancing read position, the record blocks everything behind it on that share. Where each record is acknowledged on its own, it blocks only itself.

Thirty hours: 216 million unread records divided by a 2,000-a-second surplus. Nothing is lost while that surplus holds and the six-hour unread age sits inside the retained window — loss arrives only if the surplus goes to zero or negative and the oldest unread records age out unhandled.

The group total is a sum, and a sum hides its distribution. One share of the work can be hours behind while the others are current, leaving the total small. The missing reading is the per-share breakdown, and specifically the maximum unread age across shares.

Forward means purging the waiting records — discarding them for every reader at once — because there is no bookmark to move. There is no way back: reprocessing requires a second copy written when the records were first produced.

The start-from rule ran. A reader with no valid position begins where that rule says, typically the newest record, the oldest retained record or a point in time, and beginning at the newest skips everything between. Applying a configured default is not an error.

Pre-authorise the levers that change only cost and latency — adding members to the ceiling, enlarging batches within a stated bound. Anything that changes what the system computes, such as waiving in-order handling, needs a per-stream answer recorded by its owner before the incident, not improvised during one.

Queue depth — the count of records still waiting to be handed out — plus the age of the oldest waiting record. It is the same signal read without a position to subtract from, so it describes the whole queue rather than any one reader.

Make the stored position lifetime at least as long as the retained window plus the longest reader outage you intend to survive unaided. Where the reading side keeps its own position, that store's cleanup and restore policy becomes the lifetime, and someone must own it.

Settle it before the incident: classify each stream by tolerable loss, pre-authorise the cheap exits, require the record to be captured durably before any discard, name an owner for anything set aside, and set a deadline before the retained window decides for you.