In a rolling restart of a broker cluster, why is the process answering again a weaker gate than a cluster-side catch-up check?

## Two different facts, minutes apart A rolling restart loop needs a signal that says it is safe to stop the next node. There are three candidates, and only one of them is about the thing at risk. | gate | what it actually asserts | what it misses | |---|---|---| | the process answering | this node started and accepted a connection | nothing at all about how current its copies are | | a fixed sleep | this much clock time has passed | that catch-up time is set by traffic, not by the operator | | every copy back in the caught-up set | the data this node holds is current again | little - this is the condition the risk is actually about | The first two are cheap to implement, which is why scripts reach for them. The third is the **catch-up gate**: the condition a rolling restart waits on between steps. ## Why the node's own answer cannot carry it A returning node answers early by design. It has opened its storage, registered itself, and made itself reachable - all node-local facts it can establish about itself in isolation. But the question the loop is asking is not *is this node alive*; it is *has the redundancy I removed been put back*. That is a statement about units of ownership across the cluster, and the node cannot know it alone: - Some of its copies may be current within seconds and others still far behind, depending on how much traffic each unit took. - A node can be serving units perfectly while the copies it merely holds for other units are still fetching. - On some designs the node begins accepting work before it has finished fetching, precisely so it can be useful early. So a node that is fully up, healthy and serving can still be several minutes away from having restored what the stop removed. ## Why a fixed sleep is the same mistake wearing a clock A sleep is not a weaker gate than the catch-up gate - it is a different kind of thing. It converts a **condition** into a **guess**. The catch-up distance a returning node owes depends on: 1. how long the process was actually down, including any delay before it was restarted; 2. the write rate on the units whose copies it holds, which varies by hour and by stream; 3. how much bandwidth the fetching is allowed and how much is left after live reads and writes; 4. whether the design resumes from where the copy stopped or resynchronises the whole unit. A constant chosen on a quiet afternoon is therefore right on a quiet afternoon and wrong the evening a campaign doubles the traffic. Worse, it fails silently: the loop still finishes, still reports success, and leaves the thinning behind it invisible. ## Evaluating the gate too early There is a second, subtler failure that only appears once the loop does use a cluster-side check. If the check runs immediately after the stop, it can pass **vacuously**: the cluster has not yet noticed the node is gone, so its copies are still counted as current, and the gate reports exactly the healthy picture that existed a second earlier. A correct loop therefore needs both halves - it must wait to see the condition go bad before it waits for it to come good, or it must assert the condition against a view known to be fresh. ## What to gate on, said neutrally Platforms express the condition differently, and a candidate should name the shape rather than a metric: - where one node serves and others follow, the gate is that no unit of ownership is short of its expected current copies; - where writes are committed by a majority, the gate is that the returning node has reached what the majority already holds, and that a majority never dropped below its floor during the step; - where records live on detached storage, there is far less to wait for, and the gate is closer to ownership having settled than to bytes having moved; - on queue-shaped brokers, the equivalent is that each replicated queue has a synchronised second copy again before the node holding it is touched. ## The operational habit The discipline is one sentence: **the loop advances on a cluster-side condition about data, never on a node-side answer about liveness and never on a timer.** Everything else in rolling restart practice - the order of nodes, the pace, the abort rule - is built on top of that gate, and none of it is worth anything if the gate is the wrong kind of fact.