A reader group is reassigned every ninety seconds and pauses about twenty each time, with no deploys running. What is happening?

## The loop, step by step The pattern has a fixed shape, and once you recognise it the diagnosis is quick: 1. A member is handed a share and begins working through records. 2. One unit of work takes longer than the **progress deadline** — the timer that bounds how long a member may hold work without showing that it is still moving. 3. The deadline expires. The member is declared gone, even though its process is alive and its **liveness signal** is still arriving on time. 4. Its share is redistributed. Part or all of the reader group pauses while the division settles. 5. The member finishes, discovers it no longer holds the share, and rejoins. 6. Rejoining is itself a membership change, so the group is reassigned again. 7. The member is handed work, is exactly as slow as before, and step 2 repeats. The cadence is the fingerprint. Crashes and deploys are irregular; a timer produces a rhythm. A group reassigning on an interval that nobody scheduled is reporting a timer expiring over and over. ``` t+0s member holds share, starts a slow unit of work t+60s progress deadline expires -> member declared gone t+60s reassignment begins -> group pauses t+80s shares settled -> reading resumes t+85s slow unit finishes; member finds its share gone; rejoins t+85s reassignment begins again -> group pauses again t+90s member handed work, equally slow -> back to t+0 ``` ## Why a healthy member gets declared gone Most designs run **two independent timers**, and conflating them is why this looks impossible at first: | Timer | What it proves | What its expiry means | |---|---|---| | Liveness signal | The process exists and can still talk to the cluster | Crash, host loss, or a broken network path | | Progress deadline | The member is still working through what it holds, not wedged | The member is alive but has held work too long without progressing | A handler that makes one slow call per record, or that occasionally hits a pathological record, satisfies the first timer perfectly while failing the second. The system's belief that the member is gone is not wrong by its own definition — the member really has held a share for longer than the group is willing to wait — but it is wrong about the cause, and the remedy it applies makes things worse rather than better. ## Reading the numbers you have - **Reassignment interval roughly equal to the deadline.** Strong evidence that the deadline is what is firing, not the liveness timer, which is shorter. - **Unread count rising in a staircase.** It climbs through every pause and barely falls between them, because each cycle leaves less productive time than the last. - **The same records attempted repeatedly.** Every restarted unit of work resumes from the last recorded read position, so work done but not recorded before the deadline expired is done again by whoever inherits the share. - **Members alive throughout.** No restarts, no crash traces, no host events — which is what separates this from a genuinely crashing instance. - **Adding members makes it worse, not better.** Each newcomer is another membership change and another pause, and if the slowness is per record the new members are equally slow. ## Breaking the loop There are only two honest levers, and choosing between them is the judgment being tested: 1. **Make the cycle finish inside the deadline.** Reduce how much a member takes on before it next reports progress, or move the slow step out of the path that the deadline measures. This is right when the slowness is incidental — a chatty dependency, an unbatched lookup, an accidental serial loop. 2. **Raise the deadline to exceed the real worst case.** This is right when the work genuinely takes that long and nothing about it is wasteful. The price is honest and should be stated: a member that truly dies while holding a share is now undetected for longer, so its share sits unread for the new, longer timeout. What does **not** work: restarting the members, which hands the same slow work back to a fresh process; or adding capacity, which adds membership churn to a group already spending its time churning. Stabilise the loop first, then scale. ## What this is not Two neighbouring incidents look similar on a chart and are not this one. A member that **keeps** its share and simply makes no progress is a different diagnosis — nothing is being reassigned there, and the rhythm is absent. And a **broker node** leaving or returning moves data between cluster nodes; that is a cluster-membership event, not a reader-group one, and no amount of tuning a reader's deadline will address it. Establish which of the three you are looking at before you change anything, because the fixes are mutually useless.