Beyond electing a new leader correctly, what concrete mechanisms stop a still-running, previously-elected leader from continuing to write to shared storage after a network partition has caused a new leader to be chosen, and why is preventing the election alone not enough?

## The control plane and the data plane A subtle but critical gap exists in almost every leader-election protocol taught in isolation: the protocol guarantees that the cluster reaches agreement on who the current leader is, but it says nothing about forcibly stopping a previous leader that hasn't yet learned it's been replaced. Election algorithms like Bully, Ring, Raft's `RequestVote`, or ZooKeeper's ephemeral-znode recipe all operate over the **control plane** - the messages nodes exchange to agree on leadership - but a leader's actual dangerous action is on the **data plane**: writing to a shared database, appending to a replicated log, or granting an exclusive lock. A partition can very easily separate these two planes: the old leader might be cut off from the rest of the cluster (so it can't see the new leader's heartbeats or a quorum) while remaining fully connected to the shared storage system it writes to. From the old leader's own local point of view, nothing looks wrong yet - it just can't reach its peers, which it may interpret as everyone else being down rather than itself being isolated - so it keeps right on writing. ## Why a correct election is not a claim about writes This is precisely why 'we ran a correct election' is not the same claim as 'writes are safe.' Even Raft, which is provably safe about log replication among the nodes that participate in its own quorum, relies on the assumption that only quorum-participating writes matter; if some external system (a downstream cache, an external API the leader calls, a file the leader writes directly) is written to by the leader outside of Raft's own replicated log, Raft's internal safety guarantees don't automatically extend to that side effect - the leader still needs some way to prove, to that external system, that its leadership claim is current at the moment of the write. ## Fencing tokens move the check to the resource Fencing tokens solve this by moving the safety check to the one place that actually matters: the shared resource itself. 1. Every time leadership changes hands - whether via lease renewal, a new Raft term, a new ZooKeeper ephemeral znode, or any other election mechanism - **the new leadership grant carries a monotonically increasing number** (a term number, a znode's transaction ID, a dedicated counter from the coordination service). 2. **The leader is required to attach this token to every write** it sends to shared storage. 3. Critically, **the storage system itself must track the highest token it has ever accepted** and reject any incoming write whose token is lower. This means even if a zombie old leader wakes up and issues a write, that write simply gets rejected the moment a legitimate newer leader has performed even a single write with its higher token - the old leader's write is denied not because anyone told it to stop, but because the resource itself refuses stale credentials. This shifts the safety property from 'every process behaves correctly and stops promptly when told' (a liveness-dependent, cooperative assumption) to 'the resource enforces correctness regardless of what any client believes' (a much stronger, storage-enforced guarantee). ## STONITH, the blunter alternative A blunter alternative used in classic high-availability clustering software (Pacemaker, older SAN-based failover clusters) is **STONITH** - Shoot The Other Node In The Head - where, instead of relying on cooperative token checks at the storage layer, the cluster's fencing mechanism actively and forcibly powers off, reboots, or cuts network/power access to the presumed-dead old leader's physical or virtual machine before allowing a new leader to take over shared resources like a SAN volume or a floating IP. This is more disruptive (it genuinely kills the old node rather than just rejecting its writes) and requires out-of-band control (IPMI, a cloud provider API, a managed power strip) that's independent of the primary network path the cluster uses for its own coordination messages, precisely so that a network partition affecting the coordination path doesn't also block the fencing action. STONITH trades operational complexity and blast radius for a guarantee that doesn't depend on every downstream storage system correctly implementing token checks. ## Which one fits your constraints Which approach a team picks depends heavily on what they control: | The constraint | What fits | |---|---| | **If you own the storage layer** (a custom service, a database you can add a token column and a rejection check to) | fencing tokens are strictly better - no forced reboots, no destructive action, and correctness holds even if the old leader keeps running harmlessly forever, since its writes are simply always rejected. | | **If the shared resource is something you can't modify to add token checks** (legacy hardware, a third-party SAN, a physical device) | STONITH-style forcible fencing may be the only option available, accepting the operational cost of actually killing a node that might otherwise have recovered gracefully. | Production systems that care about this distinction explicitly document it: - **Kubernetes' controller-manager leader election** combined with resource version checks on the Kubernetes API server effectively implements token-based fencing (`resourceVersion` acts as the token). - **Pacemaker/Corosync clusters** for traditional shared-storage HA setups document STONITH as a hard requirement, refusing to promote a new primary at all until fencing of the old one is confirmed, specifically to avoid the exact split-brain-writes-corrupt-data scenario this question is about.