Split-Brain and Fencing

## The setup that makes it possible A relational HA cluster has one primary that accepts writes and one or more standbys that replay its change stream (WAL/redo/binlog). Only one node may accept writes, because the replication stream is a single linear history: every standby assumes it is replaying one authoritative sequence of changes. Nothing in the engine merges two write streams. Failover means choosing a standby, stopping its replay, and making it writable. The dangerous part is the decision, not the promotion. ## Why detection is fundamentally ambiguous Nodes decide a peer is dead by absence of evidence: heartbeats stop, a TCP connection breaks, a health query times out. All of these are indistinguishable between: - the primary process actually crashed or the host died; - the network between observer and primary partitioned, while clients on the primary's side still reach it happily; - the primary is alive but unresponsive: swap storm, saturated IO, a long checkpoint, a VM pause, a stop-the-world GC in the layer in front of it. The second and third cases are the split-brain generators. In case two, the primary keeps committing user transactions while the observers conclude it is gone. In case three, the primary may unfreeze seconds after promotion and resume writing, unaware anything happened. ## What actually breaks Call the old primary A and the promoted standby B. After promotion both accept writes: - Rows diverge. A row updated on A and differently on B has two committed values, both durable. - Surrogate keys collide. Sequences and auto-increment counters continue independently, so both nodes hand out id 5001 to different orders. - Uniqueness is violated cluster-wide even though each node's constraint held locally. - Application side effects double up. If the app on each side charged a card or emitted an event, that already left the database. When the partition heals there is no automatic reconciliation in single-primary replication. Operators pick a survivor and rebuild the loser from a fresh base backup, so the loser's committed transactions are lost. Recovering them means manual forensic extraction from WAL/binlog and hand-merging, which is slow and often impossible without violating business invariants. Meanwhile any standby that was still following A now has a history incompatible with B and must also be rewound or rebuilt. A second, subtler harm: clients. Old connections, cached DNS, and a proxy pointing at A can keep routing writes to the demoted node long after promotion, so split-brain can persist even if the cluster software thinks it resolved. ## Why it is not solved by better monitoring No timeout distinguishes slow from dead; that is a property of asynchronous networks, not a tuning gap. Longer timeouts reduce false promotions but raise downtime, and shorter ones do the opposite. The workable answers change the shape of the problem: 1. Make promotion require agreement from a majority of a fixed member set, so a minority partition can never elect anyone. 2. Make leadership a time-bounded lease that the primary must keep renewing, so a primary that loses contact demotes itself before anyone else is promoted. 3. Fence the old primary: power it off, revoke its storage access, or drop it out of the routing layer before the new primary opens for writes. In practice these are combined: a majority quorum store holds a leader lease, the leader demotes itself on lease loss, and a fencing action covers the case where it cannot. ## What to say in an interview Define it in one sentence, name the partition/freeze cause, state that the damage is divergent committed history with no merge path, and then name quorum plus fencing as the structural fix. Candidates who only say we should monitor better have missed the point.