Why does leadership pile onto a few cluster nodes after failures and restarts, and why does it stay there?

## What moves when a node stops On designs where exactly one node serves each **unit of ownership** — a partition, or on queue-shaped brokers a queue — stopping that node forces a choice for every unit it was serving. Another copy that is current enough takes over, the change is recorded, and clients refresh their **owner map** (their cached view of which node serves what) on the next miss. No records move: the copies were already there, which is exactly why the handover is fast enough to be survivable. The selection is constrained by who is available and current at that instant, not by any plan for even distribution. That is the seed of skew. ## The home node Each unit has a plan naming the nodes that should hold a copy of it. The first node in that list is its **home node**: the one leadership is expected to sit on when nothing is wrong. The home nodes are chosen so that, taken across the whole cluster, leadership would be spread evenly. That expectation is only realised while nothing has failed. The plan describes where leadership *belongs*; the live assignment describes where it *is*. Skew is the gap between those two, and the gap only ever opens in one direction on its own. ## Why it does not come back by itself When a stopped node returns, three things happen: its process starts, its copies fetch whatever they missed until their **catch-up distance** reaches zero, and they rejoin the **caught-up set** — the copies current enough to be promoted or to satisfy a write. None of those steps includes taking back the units this node used to serve. There is no penalty for leaving leadership where it is; the units are being served correctly, just not from home. So the return to home must be an explicit act, and designs differ in who performs it: - some run a **restoration continuously or periodically** on their own, correcting drift without anyone asking; - some expose it as an **operator action** that has to be run deliberately; - some never concentrate this way at all, because no single node serves a unit — where consumers compete for work from a shared queue there is nothing to hand back. Saying which of those you mean is the difference between a correct answer and one platform's answer. ## The patterns that concentrate it 1. **A node that fails repeatedly.** Each of its failures sheds leadership onto its peers; each recovery gives none back. A flapping node ends up serving almost nothing while its peers carry everything. 2. **A rolling restart in a fixed order.** Each node in turn sheds what it was serving onto nodes already restarted or not yet touched. The last node in the order is left holding least, and the imbalance survives the restart. 3. **A batch of nodes stopped together**, for a host-level change or a failure domain going away. Everything they served lands on the remainder, and the remainder keeps it. 4. **A node re-added after a long absence.** It rejoins with copies to rebuild and no claim on anything; even after it is fully current it is serving nothing. The common thread is that leadership is *reassigned on loss* and *not reclaimed on return*, so its distribution is a running record of past failures rather than a reflection of the current cluster. ## Why nobody notices Every symptom of drift sits in a per-node view, and every default dashboard is a cluster-wide one. Copies stay even — no records moved — so storage looks correct. Replication looks correct, because every unit still has its copies. What changed is an assignment, and the only number that shows it is the count of units each node currently leads. A cluster can run skewed for months, absorbing it in headroom, and discover it during the first traffic peak or the first loss of the wrong node. | What happened | What moved | What a cluster total shows | |---|---|---| | Node stopped | Leadership for its units | Nothing unusual | | Node returned, caught up | Records it had missed | Brief copy traffic, then nothing | | Nothing hands leadership back | Nothing | Nothing, indefinitely | ## What an interviewer is listening for That you separate the *loss* event, which must reassign leadership immediately or the units stop being served, from the *return* event, which has no reason to reassign anything and therefore does not. Candidates who describe the cluster as self-healing have usually only operated a platform that restores leadership on its own, and have not noticed the machinery doing it for them.