Split Brain

## The failure this rule exists to prevent A replicated in-memory store puts a copy of the **whole keyspace** on each node. One node is the **primary** and accepts writes; the others are **replicas** and receive them. When the primary stops answering, something promotes a replica so writes can continue. **Split brain** is the state where that promotion happens while the old primary is still alive and still accepting writes. Two nodes each believe they own the same keyspace, and callers that can reach either one are told their writes succeeded. It is worth being precise about the cause, because candidates usually name the wrong one: a crash does not produce split brain, because a crashed node writes nothing. A **network partition** does — the link between two groups of machines fails while every machine keeps running, each group sees the other as dead, and the group holding replicas has every reason to promote one. ## What the majority gate actually says The rule is simple to state: **a side may promote only if it can reach more than half of the deciding members.** Two disjoint sides cannot both hold more than half of one membership, so at most one side of any split is ever permitted to promote. That is the whole of the guarantee. The arithmetic of why a strict majority is the right quorum, rather than some fixed count, is consensus material. In an interview about a store you cite it as a conclusion and move on. What is specific to this tier is **which members are being counted**, because that varies by product and by deployment, and each arrangement makes a different deployment unsafe. | Who the deciding members are | Smallest shape that survives losing one | What the isolated side sees | |---|---|---| | The store nodes themselves | Three nodes in three independent failure domains | Its peers unreachable; it must act on that alone | | Dedicated observer processes | Three observers, placed apart from the store nodes | Observers unreachable; the store node is not itself a voter | | A controller outside the data plane | Depends on the controller's own availability, not on node count | Whatever the controller does to it, which may include cutting it off at the network | A candidate who names all three, and says which one their deployment uses, is demonstrating the thing this question is for. A candidate who assumes the peers vote has silently ruled out the other two arrangements. ## The deployment shapes this rule condemns - **Two nodes.** One primary, one replica, nothing else. Neither side of a split holds more than half of two, so nobody may promote. Teams then either lose writes entirely or disable the gate and manufacture the split brain it existed to prevent. - **An even number of deciding members.** Split evenly, no side promotes; and an even count buys no additional failure tolerance over the odd count below it, so it is cost without benefit. - **Every deciding member inside the two sites you are protecting against each other.** Whichever site holds the larger share is the only site that can ever promote alone. That may be a deliberate choice, but it should be a choice, not a discovery made during an incident. - **Observers co-located with the store nodes they watch.** Losing the machine takes the watcher with it, so the membership shrinks exactly when you need it. The repair for the first three is the same: put one deciding member in a **third failure domain**. It holds no keyspace and serves no traffic; it exists so that a two-site deployment has an odd membership and a tiebreaker that survives either site. ## What the gate does not do This is where most answers overreach. The gate governs **who may be promoted**, and nothing more. 1. It does not stop the old primary. An isolated primary is still running, still reachable by some callers, and still accepting writes. Stopping it is a separate obligation that falls on that node itself, or on whatever can still reach it. 2. It does not preserve the writes already accepted. Under acknowledge-then-propagate the promoted replica is missing whatever had not yet reached it, and on this tier those writes are simply gone rather than replayable from a durable record. Some stores in this class instead withhold acknowledgment until at least one copy holds the write, and some let a caller choose that per write, which changes the size of the gap but not the existence of the gate. 3. It does not keep the minority side available. By design, the smaller side gives up writes. That is the trade being bought. ## The trap worth naming out loud The most common way teams get a real split brain is by reacting to the *availability* failure. The link drops, no side has a majority, nothing promotes, and someone lowers the required threshold or promotes by hand on the smaller side to restore service. Both are the same mistake: they permit two primaries. The honest responses are to add a deciding member in a third place, to accept that one site cannot survive alone, or to promote manually only after confirming the other side is genuinely stopped rather than merely unreachable.