Why does automated database failover require a majority quorum, and what is a witness (or arbiter) node used for in a cluster that would otherwise have an even number of members?

## The arithmetic Quorum-based failover fixes membership up front: the cluster is exactly these N members. Any state change that matters (electing a leader, granting a lease) requires votes from a strict majority, floor(N/2)+1. The key property is trivial but decisive: two disjoint sets cannot both be majorities of the same N. Therefore, however the network partitions, at most one partition can elect a primary. Split-brain becomes impossible by counting rather than by hoping detection was accurate. The price is availability. A partition that leaves no side with a majority leaves the cluster with no writable primary. Quorum systems deliberately choose unavailability over divergence. ## Fault tolerance by member count - N=2: majority is 2. Either node's loss stops writes; a partition gives each side one vote. Tolerates zero failures for automated failover. - N=3: majority is 2. Tolerates one failure. - N=4: majority is 3. Still tolerates only one failure, while adding a node that can fail. Even counts buy nothing. - N=5: majority is 3. Tolerates two failures. This is why clusters are sized odd. Going from three to four members increases failure surface without increasing tolerance. ## What a witness is A witness (arbiter, quorum device, tiebreaker) is a member that votes but carries no data replica and can never be promoted. Its purpose is purely to make N odd and to sit in a third failure domain. Because it stores no data it needs almost no disk or CPU, so a small instance or container is enough. Typical shapes: - Two data nodes plus one witness: majority is 2, so the surviving data node plus the witness can promote while a lone isolated node cannot. - A three-node consensus store (etcd/ZooKeeper/Consul) holding the leadership state, with database nodes as clients. Here the quorum lives in the store rather than in the database processes. Be precise about which layer holds the quorum. In the Patroni pattern the quorum is etcd's; Patroni agents merely read and write the leader key. Losing etcd quorum means no agent can hold or acquire leadership, which is the intended safe state. ## Placement is the part people get wrong Quorum only helps if members fail independently. Common mistakes: - Witness on the same hypervisor, rack, or power feed as a data node, so one hardware failure removes two votes. - Witness in one of two datacentres, making that datacentre's loss fatal to the whole cluster while the other site's loss is survivable. That is an asymmetric HA design people rarely intend. - Witness on a network path that shares fate with the replication link, so a link failure removes it exactly when it is needed. The useful rule: place the tiebreaker in a third, independent failure domain, and check whether the majority still exists after each single-domain loss you claim to survive. ## What quorum does not give you Quorum decides who may become primary. It does not by itself stop the old primary from writing. A partitioned old primary may be entirely unaware it lost the election, and clients pinned to it keep writing. That is why quorum is paired with self-demotion on lease expiry and with fencing, and why the routing layer must be updated as part of promotion rather than after it. Quorum also says nothing about data currency. A member can win an election while lagging; choosing a leader and choosing an up-to-date leader are separate concerns, which is why managers additionally compare replay positions and may refuse to promote a candidate beyond a configured lag threshold. ## Interview framing State the arithmetic, give the two-node failure case, explain the witness as a vote-only third failure domain, note odd sizing, and close with the caveat that quorum prevents a second election but not a second writer, which is what fencing is for.