A nightly script restarts each broker node with a fixed 60-second sleep between steps; afterwards acknowledged records are missing - why?

## How a clean-looking loop loses acknowledged records The script did everything the runbook said: one node at a time, in order, with a pause between. What it did not do is check anything. A sleep asserts that time has passed, and the risk is about whether data is current, so the loop never noticed that it was spending redundancy it was not putting back. Walk one unit of ownership - a partition, or on queue-shaped brokers a queue - through the loop. Say it is held by three nodes, and the cluster has four: 1. **Stop the node that holds no copy of it.** Nothing changes for this unit; three current copies. 2. **Stop the second node.** Its copy leaves the caught-up set. Sixty seconds later the script moves on, but on the night's write rate the copy needs several minutes. Two current copies, one behind. 3. **Stop the third node.** Same story. One current copy, two behind. 4. **Stop the fourth node** - the one holding the only current copy. Zero current copies. At step four the unit has no node that holds everything the writers were told had been stored. ## What happens at zero What the platform does next is one of two things, and both are bad in different ways: - **It refuses writes for that unit** until a node with a current copy is back. The writers see errors or stall; nothing is lost, but the outage is real and it is the operator's doing. - **It restores service from a copy that was behind.** Availability returns immediately, and every record that copy never received disappears - including records the broker had already acknowledged, which is why the loss is invisible to the writers, who were told the writes had landed. Which of those happens is a platform and configuration matter, and a candidate should say so rather than assert one. The important point is that the operator chose neither - the loop chose for them. ## Why nobody noticed - **A sleep cannot fail.** There is no exit code for having waited too little. - **The end state looks healthy.** By morning every node is up, every unit has its copies current again, and the count of missing records is not a metric anyone has. - **The window of exposure is transient.** The shortage existed only while the loop ran, so a dashboard sampled afterwards shows nothing. - **The loss surfaces downstream.** Someone finds a gap in a reconciled total days later, by which point the maintenance is not a suspect. ## What would have prevented it The repair is not a longer sleep. A longer sleep is the same guess with more slack, and it will be wrong the night traffic doubles or a copy has an unrelated shortage to work off. The repair is to change the kind of thing the loop waits on: 1. **Gate on the catch-up condition** - before stopping any node, require that no unit of ownership is short of its expected current copies, taking the restarted node's copies into account. 2. **Check the precondition before the first stop, too.** A cluster that was already short before maintenance began will be short at every step, and the loop must refuse to start rather than compound it. 3. **Abort, do not skip.** If the condition has not come good within a generous bound, the loop stops with the unit named. Continuing is precisely the behaviour that produced the loss. 4. **Make the safety independent of the script.** Where the platform can enforce a floor on current copies, set it, so that an operator command that would take the last current copy down is refused rather than obeyed. ## The part that generalises This failure is not really about restarts. It is about a loop whose gate is the wrong kind of fact, running long enough to exhaust a reserve that nothing was measuring. The same shape appears wherever maintenance walks a set of nodes: the individual step is safe, the sequence is not, and only a condition evaluated between steps can tell the difference. The framing to give in an interview is blunt: **a rolling restart is only a safe operation if something between the steps is checking that the previous step has been paid back.** Otherwise it is a batch restart with extra steps and a longer total outage.