How does a cluster decide that a follower copy has fallen far enough behind its leader to drop out of the caught-up set?

## Why a boundary is needed at all On a leader-follower design, every follower copy of a stream is always at least a little behind the leader — it has to read a record before it can hold it. So "caught up" cannot mean "identical". It has to mean "close enough", and something has to define close enough. That definition is the **catch-up window**: the allowance a copy has before the leader stops counting it as part of **the caught-up set**. The window is not a health check and not a failure detector. It is a currency threshold, evaluated against **the position a follower has replicated to** compared with the leader's newest record. ## The two units, and why the choice matters | Window unit | What it asks | Where it holds up | Where it misleads | |---|---|---|---| | A number of records | how many records short of the leader is this copy? | steady traffic, uniform record sizes | a burst pushes every healthy follower past it at once, so load looks like sickness | | Elapsed time | how long since this copy was last current with the leader? | bursts, uneven record sizes, mixed streams | a copy that keeps making requests but never closes the gap can look better than it is | The burst case is the one that decides the argument in practice. Suppose the write rate into a stream multiplies for thirty seconds. Every follower is now further behind in records than it was, not because anything is wrong with any of them but because there is simply more to pull. A record-count window cannot tell that apart from three sick disks; an elapsed-time window can, because each follower is still reaching the leader's current position moments after the leader reaches it. This is why platforms that use a time-based allowance describe it as *time since the copy was last caught up*, rather than *time since we last heard from it* — the second is liveness, which is a different subject, and a copy can be perfectly alive and hopelessly behind. ## Dropping out and being re-admitted Both transitions are routine, not exceptional: 1. A follower's disk starts retrying, or the link between its failure domain and the leader's saturates. Its gap widens. 2. The gap crosses the catch-up window. The leader removes it from the caught-up set. Nothing is deleted; the copy keeps every record it already pulled. 3. The copy carries on pulling. To rejoin, it must read faster than the leader is writing — it needs surplus rate, not merely a working disk. 4. Once it is inside the window again, the leader re-admits it and the set widens back. ``` leader writes ──────────────────────────────────────────▶ copy A close behind ............................. in the set copy B close behind ..╲ disk slows ╲............. drops out copy B ╲.. still pulling ..╱..... re-admitted ``` Step 3 is where a lot of intuition fails. A copy that dropped out because it cannot keep up with the incoming rate does not recover simply because time passes; it recovers when it has spare throughput to close a gap *and* keep pace with new arrivals. ## Sizing the window - **Too tight** and copies **flap**: normal variation — a compaction on the volume, a brief link contention, a garbage burst — pushes a healthy follower over the line and back repeatedly. Each transition re-narrows and re-widens the set, and the durability the stream offers oscillates for no useful reason. - **Too loose** and the set lies in the other direction: a copy well out of date is still counted, so writes are recorded as backed by copies that do not hold them yet. - The honest framing in an interview is that the window is a **false-positive / false-negative trade**, exactly like any threshold: tighten it and you eject healthy copies, loosen it and you keep stale ones. Platforms pick a default that tolerates ordinary variation; the point of knowing the setting exists is that a cluster whose hardware or traffic shape is unusual may need a different one. ## Where this does not apply A catch-up window belongs to designs that maintain a standing membership of current copies. Platforms that commit a write when a majority of copies has answered do not need one: a slow copy is simply not among the majority that answered *this* write, and it catches up afterwards with no drop-out event. In designs where durability comes from shared underlying storage rather than per-node copies, there is no follower to be behind in this sense at all. Before tuning any allowance, confirm your platform is the shape that has one.