Why is removing a node from a broker cluster an evacuation with a completion condition, rather than simply stopping its process?

## A stopped node is a failed node A broker or streaming cluster reacts to what it observes, and what it observes when you stop a process is exactly what it observes when the machine dies: a member that no longer answers. Every consequence of a failure follows. - Each **unit of ownership** — a partition (or queue) that exactly one node serves at a time — that this node led must be handed to another node immediately, while writers retry into the gap in which nobody serves that unit. - Every **copy** the node held — its complete stored duplicate of a unit — stops being current the moment the process stops, and falls out of **the caught-up set**, the subset of copies current enough to serve a write or be promoted. - Any unit whose remaining current copies are now fewer than the cluster wants will have new ones built somewhere else. That is the same copying an evacuation would have done, except unplanned, unpaced and starting at the second you chose. So "just stop it" does not avoid the data movement. It reorders it: the movement happens *after* the reduction in redundancy rather than before it, which is the wrong way round. ## What an evacuation does instead Evacuating a node means moving every unit of ownership off it and confirming it is empty before the process stops. The node stays up and keeps serving throughout, so nothing is promoted in a hurry and no unit is ever short of a current copy because of you. The order matters more than the mechanism: 1. Declare that this node should hold nothing, and that its units belong elsewhere. 2. Let the receiving nodes fill their copies while the leaving node continues to serve. 3. As each receiving copy becomes current, leadership and ownership move to it — within the same cluster, one unit at a time. 4. When the node holds nothing, stop it. | | Stop the process | Evacuate first | |---|---|---| | Redundancy during the change | drops first, recovers later | never drops | | Promotions | many, all at once, unplanned | one at a time, as each copy becomes current | | Copying performed | the same bytes, afterwards, unpaced | the same bytes, beforehand, paced | | Writers' experience | a gap while owners are promoted | ownership moves under a healthy node | | When you know it is done | you do not | the node holds nothing | ## The completion condition and the duration The two things an evacuation has that a shutdown does not are a **completion condition** and a **duration**. The completion condition is checkable: *this node holds no unit of ownership, and no unit it used to hold is left short of a current copy.* An empty node is not enough on its own — a unit can be off the leaving node while the copy that replaced it is still filling. Both halves have to be true before the process stops. The duration is arithmetic: the records the node holds, divided by the bandwidth the cluster can give to moving them. That is why an evacuation is scheduled rather than issued. A node holding a few hundred gigabytes is a coffee break; a node holding many terabytes is an overnight change, and it is competing for the same disks and links that production is using the whole time. ## Where storage is detached, and where it is not On designs that keep records on shared or remote storage, a leaving node is still evacuated — ownership still has to be handed over, and anything it has not yet pushed to the shared store still has to be settled — but almost no bytes move, so the completion condition is met in seconds rather than hours. The discipline is identical; only the duration collapses. On designs where each node keeps its own copies on local disk, the duration is the whole story. ## Sequencing traps worth naming - **The thin unit.** If some unit of ownership is already down to very few current copies, the node you are about to evacuate may be holding one of them. Evacuating it is still the right move — but the receiving copy has to be built *before* this one goes away, which is precisely the order an evacuation enforces and a shutdown does not. - **Two at once.** Evacuating a batch of nodes together is only safe if no unit loses its remaining current copies and the bandwidth exists to fill the replacements. One after another is slower but bounded. - **The empty-looking node.** A node that appears idle may still be the only current copy of something rarely written. Emptiness of traffic is not emptiness of data. - **The reversal.** Until the node is stopped, an evacuation can be abandoned and the node left in place; after it is stopped, putting it back is a join, with all the copying a join implies.