Why is the membership that holds a cluster's metadata role kept small and odd-sized rather than spanning every node?

## What the number is actually buying The metadata role is the single authority for cluster state, and it stays a single authority by requiring that **a majority of its membership agrees before any change is admitted**. Requiring a majority is what stops two halves of a split cluster both believing they are in charge: two disjoint sets cannot both contain more than half of the same membership. The size you choose for that membership therefore decides two things at once — how many of its members can be lost while change is still possible, and how much agreement every single change has to buy. ## The arithmetic, in full | membership size | majority needed | losses tolerated | |---|---|---| | 3 | 2 | 1 | | 4 | 3 | 1 | | 5 | 3 | 2 | | 6 | 4 | 2 | | 7 | 4 | 3 | Read the table down the even rows and the argument for odd sizes writes itself: - going from three to four raises the majority from two to three, and still tolerates only one loss; - going from five to six raises the majority from three to four, and still tolerates only two; - the even member is pure cost: one more machine to run, patch and monitor, one more participant in every agreement, and not one extra failure survived. ## Why small, and not just odd Small is a separate argument from odd, and it has three legs: 1. **Agreement is paid per change.** Every structural change — a stream created, a partition moved, an ownership change recorded — needs a majority to accept it. More members means more messages and more waiting on the slowest of the members needed to form that majority. 2. **Tolerance grows slowly and cost grows steadily.** Going from three to five buys exactly one more tolerated loss. Going from five to eleven buys three more, for more than twice the agreement cost, which is almost never a trade anyone wants. 3. **The membership is not where scale comes from.** Serving more traffic means more record-serving nodes. The coordination membership does not get bigger because the cluster does; a cluster of a hundred record-serving nodes is commonly coordinated by three or five members. This surprises people who assume every node must vote. There is one real reason to prefer five over three, and it is worth saying out loud: with three, a single member down leaves you one failure from being unable to change anything, which makes routine maintenance on a coordination member a tense operation. Five gives you room to take one out deliberately and still tolerate one unplanned loss. ## The confusion to avoid The majority discussed here is about **cluster state**, and it is a different number from **how many copies of a record must acknowledge a write before the write counts**. They are chosen by different people for different reasons, they protect different things, and on many platforms they are configured in completely separate places. A candidate who answers this question by talking about how many replicas must confirm a write has answered a different question — a good one, but not this one. A second confusion: the coordination membership does not store records. Its state is the shape of the cluster — members, streams, partitions, ownership, settings — which is small in bytes and demanding in latency and durability of change. That is why sizing it looks nothing like sizing a record-serving node. ## How platforms differ Where the membership lives varies, and the sizing logic survives the variation: - where the role is an **internal membership** of the cluster's own members, you choose the count yourself and live with the arithmetic above; - where it is a **separate coordination service**, the same odd-and-small rule applies to that service's own members, and it is sized independently of how many brokers there are; - where the cluster is **rented and the role is hidden**, the provider made the choice; the arithmetic still governs what you experience, but the only thing you can observe is whether structural change is being admitted. ## What good answers sound like A strong answer states the rule (a majority must agree), does the arithmetic out loud for at least one even size, separates the coordination majority from write acknowledgement, and says that the membership does not scale with cluster size. A weak answer says "odd numbers avoid ties" and stops — which is not wrong as a slogan, but it misses that the fourth member also costs agreement on every change, and it gives no way to reason about three against five.