What should an organisation's standing rule be for how many failure domains a cluster spans, and what does that rule foreclose?

## The rule is one sentence, and the argument is about its terms A usable estate rule reads roughly: *every production cluster spans at least three independent failure domains, and no single domain holds a majority of any partition's copy set or of the membership holding cluster metadata.* Everything contentious is in the words. - **"Independent"** is the load-bearing word. Three racks on one breaker, or three buildings on one fibre path, are one domain wearing three labels, and no arithmetic downstream can detect that. - **"At least three"** is the smallest number for which any single loss leaves a majority. Two is the trap: an even split leaves neither survivor decisive, and an uneven one is a bet on which domain fails. - **"Or of the membership holding cluster metadata"** is the clause teams drop. A cluster whose data survived a domain loss but whose metadata membership did not survived the failure and cannot react to it. - **"Spans"** needs a unit. The rule must name whether a domain is a rack, a building or a region, because that choice decides both what the cluster survives and what it pays. ## Choosing the unit | Domain unit the rule fixes | Survives | Puts into the write path | |---|---|---| | rack | a power feed, a switch, row maintenance | essentially nothing | | building or zone within one locality | a whole building, its power and its cooling | a short extra hop on writes that must cross | | distant localities, one cluster stretched across them | the loss of an entire locality | a long round trip on every write that must cross | The table is why "spread as widely as possible" is not the answer. Distance between domains is bought with a hop in the acknowledged write path, and how much of that is worth buying is a separate decision from this one. Most estates settle on three domains within one locality as the default and treat anything wider as a different design with its own justification. ## What the rule forecloses A standing rule is worth having only if you can say what it costs, and this one costs: 1. **Cheap single-domain clusters.** The development cluster somebody wanted in one rack is now either an exception with a name and an owner, or it is three domains too. 2. **Locations that only offer two.** Some sites have exactly two independent domains. The rule forces a choice: a tie-breaking member placed somewhere else, an explicit and rehearsed manual intervention when a domain is lost, or not running production there. 3. **Free choice of machine.** The machine type, the storage kind or the accelerator you want may not exist in all three domains. The rule wins, or it is not a rule. 4. **Capacity flexibility.** Each domain has to be able to carry its share of the work when another is gone, which is a standing reservation rather than a one-off sizing exercise. 5. **Silence about placement.** A rule nobody measures is a preference. The per-domain layout has to be visible next to the numbers people already watch. ## Enforce it at provisioning The decisive operational point is when the rule is applied. Before data exists, satisfying it is a placement decision. Afterwards it is a migration: copies must be moved, one at a time, on a running cluster. So the rule belongs in the path that creates clusters and admits nodes — the domain label comes from the system that knows where the machine physically is, and a node without one does not join. A rule enforced by an annual review is a rule that documents drift rather than preventing it. ## Where it meets the platform The rule survives contact with different designs, but its terms shift: - Where a write commits on a **majority of copies**, the rule is literal. - Where a copy leads and the others must be **caught up**, the majority language becomes "no domain holds enough of the caught-up set to stop writes" — the same placement test against a configured floor. - Where nodes sit on **shared storage** rather than holding copies, the rule applies to the storage system, and you need its answer rather than yours. - Where the cluster is **rented**, the rule becomes a procurement question: how many domains the service spans, whether spreading is contractual or best effort, and what it reports. ## The part a lead actually owns Not the number — the exceptions. Every estate accumulates clusters that cannot satisfy the rule, and the value of having written it down is that each of those becomes a named exception with an owner, a stated failure mode and a review date, rather than a surprise discovered during a rack outage on a Sunday.