You must run a highly available relational database across two data centres with automatic failover. How do you place voting members so that losing one site does not either strand the cluster read-only or allow two primaries, and what tradeoff would you accept?

## Why two sites is the hard case Quorum works because two disjoint groups cannot both be majorities. With three independent failure domains that maps cleanly onto geography: one member each, and any single site loss leaves two of three. With two domains, every allocation is lopsided: - 2 members in A, 1 in B: losing B is fine; losing A leaves one vote out of three, so no promotion. Site A is a single point of failure for automated write availability. - 2 and 2: any site loss leaves 2 of 4, not a majority, so a site loss stops writes entirely; also a partition gives a 2-2 tie. - 1 and 1: neither side can ever act alone. This is arithmetic, not a product limitation, and no cluster software escapes it. The frequent bad response is to disable quorum checks or lower the required votes during an incident, which converts a partition into two live primaries. ## Option 1: buy a third failure domain The cheapest genuine fix is a vote-only member somewhere independent: a third small site, a colo, or a small cloud instance in a different provider or region. It stores no data, so cost and bandwidth are negligible; what matters is that its connectivity does not share fate with the inter-site link. Verify that: a witness reachable only through the same circuit that connects A and B is not a third domain. Latency matters modestly. The witness participates in consensus writes, so its round trip is added to leadership operations, not to user transactions, provided it never holds data. Tens of milliseconds is usually acceptable; if the store's election timeouts are tighter than the link's jitter, you will get leadership churn. Check the failure matrix explicitly: for each of the three domains, does a majority survive its loss, and can the surviving majority reach a data-carrying, sufficiently current candidate? A quorum that survives but has only the witness and a badly lagging replica is a promotion you may not want to happen automatically. ## Option 2: accept asymmetry deliberately If a third domain is genuinely impossible, be explicit rather than clever. Site A carries the quorum and runs automated failover between its own nodes, covering the common failures: a host dies, a disk fails, a process crashes. Site B is a replica for disaster recovery, promoted only by a human-run runbook that includes verifying A is truly gone, fencing it (power or network, and at minimum revoking its route), and accepting the RPO implied by the replication mode. The honest statement to stakeholders is that host failure is automatic and site failure is a decision with a stated time to restore and a stated data-loss bound. That is a much better place to be than automation that silently promotes across sites on a link flap. ## Separating the two questions people conflate Leadership quorum and durability are different knobs. Placing votes decides who may write. Synchronous commit decides whether a transaction is acknowledged before a remote copy has it. Across a WAN, synchronous commit adds the round trip to every write, so most systems run cross-site asynchronously and publish an RPO of seconds. If the business demands zero data loss cross-site, that cost lands on transaction latency, and you should say so numerically rather than promising both. ## Failback and dual-site operations Also plan for what happens after: the failed site returns with a stale, possibly diverged primary. It must not rejoin as a writer, and it will need a rewind or rebuild. If your inter-site bandwidth cannot rebuild a multi-terabyte replica in an acceptable window, that constraint should shape the design before the incident, for example by keeping a local backup at each site to seed from. ## Interview framing State the arithmetic first so it is clear the constraint is structural, then present exactly two credible designs (third domain, or explicit asymmetry), then name what you refuse to do (equal votes per site, co-located witness, quorum overrides in automation). Finish by separating leadership placement from durability mode and attaching numbers: RTO, RPO, and rebuild time.