Nodes are split evenly across exactly two failure domains — why is a third domain still worth adding?

## The arithmetic, not the hardware The question of how many failure domains to spread across is answered by counting, not by comparing data-centre brochures. Two things in a cluster are decided by majority: on platforms where a write commits once a majority of a partition's copy set holds it, and in the small membership that holds cluster metadata and admits every structural change. **Majority means strictly more than half.** Everything about the two-domain trap follows from that one definition. Split an even number of members or copies evenly across two domains and each domain holds exactly half. Lose either one, and the survivor holds exactly half — which is not more than half. The layout looks maximally symmetric and is, in majority terms, the worst arrangement available: it converts any single domain failure into a loss of majority. ## Counting it out | Layout | Held in each domain | After losing one domain | |---|---|---| | Two domains, even split (2/2 or 3/3) | half and half | exactly half survives — no majority either way | | Two domains, uneven split (3/2) | majority in the larger | survives losing the small one, fails losing the large one | | Three domains (2/2/1 or 1/1/1) | roughly a third each | about two thirds survive — a majority whichever domain went | The uneven two-domain row is worth dwelling on. It is not a fix; it is a bet on which domain fails. A cluster arranged that way has a coin-flip outcome, and the failure you get is chosen by the world rather than by you. ## Why the third domain can be small A third domain does not have to be a third of the machine. Its job is arithmetic: - it makes the total odd, so an even split becomes impossible; - it ensures that after any single domain loss, the surviving two domains together hold strictly more than half; - it needs to hold a member of the membership, or a copy, that can be counted — not a full share of the traffic. That is why a third rack, a small third building, or a third location that holds one tie-breaking member is a normal shape. The rule you are satisfying is: **no single power or network domain holds a majority of a copy set, or of the membership holding cluster metadata.** ## Both halves of the cluster, not one The same rule applies twice over, and teams routinely apply it once: 1. **Copies of each partition** are spread so no domain holds enough of the copy set to stop it being served. 2. **The membership holding cluster metadata** is spread on exactly the same principle, because a cluster whose data survived a domain loss but whose metadata membership did not has survived the failure and cannot react to it. A layout where the record-serving nodes are neatly spread across three domains and the metadata membership sits in one of them is a three-domain cluster for reads and a one-domain cluster for change. ## Where the model varies Not every platform commits writes by majority, and the arithmetic has to be read against the design in front of you: - **Majority-commit copy sets** are the clean case: half is never enough, and the table above applies directly. - **A leading copy with followers that must be caught up** does not count a majority at all; it counts against a floor of caught-up copies that somebody configured. A domain loss that drops the caught-up set below that floor stops writes just as effectively, so the placement question is the same one asked with a different number. - **Shared storage under the record-serving nodes** removes per-node copies entirely, and the domain question moves to how that storage system is spread — it does not disappear. - **Platforms whose metadata is held by a separate coordination service** still have a membership needing a majority; it simply is not made of the same machines, and it needs spreading on its own. - **A rented cluster** may place across domains for you and give you no visibility into it at all; then the question to ask the vendor is how many domains, not which racks. ## The half-fix to recognise Two domains plus an accepted manual intervention is a legitimate engineering position — someone decides, out of band, which surviving domain is allowed to proceed. It is legitimate because it is explicit, and it is expensive because it needs a human who is awake, correct and fast. What is not legitimate is an even two-domain split that nobody has noticed is one, described in a design document as "highly available across two sites". And the standing constraint does not go away: a third domain further from the other two adds a network hop to any write that must reach it before being acknowledged. That is why the third domain is usually placed as close as it can be while still failing independently.