If the membership behind a cluster's metadata role loses the majority it needs while every record-serving node stays healthy, what continues and what stops?

## The rule behind the symptom The metadata role only admits a change to cluster state when a majority of its membership agrees. Lose that majority — two of three members gone, three of five gone, a network split that leaves no side with more than half — and the role cannot admit anything. It has not lost the data; it has lost the right to change it, which is exactly what it was there to protect. ## What stops, concretely Everything that is a change to the shape of the cluster: - creating or deleting a stream or queue; - changing a stream's settings or the cluster-wide configuration; - raising a parallelism ceiling; - admitting a new node or retiring an existing one; - moving a partition from one node to another; - recording that a different copy now owns a partition. Administrative calls of this kind typically fail or hang. On a rented cluster this shows up as management operations returning errors while the data path looks fine. ## What often continues Here the platforms genuinely differ, and a good answer says so rather than asserting one behaviour: - on many designs, **traffic to partitions whose ownership is already settled continues**. Clients already know which node owns what, and produce and fetch go to that node directly without consulting the metadata role per record. - on other designs the cluster **fails closed**: if cluster state cannot be consulted or confirmed, writes are refused rather than accepted into a shape nobody can vouch for. - on a **rented cluster**, the vendor's own behaviour decides, and you may see nothing but failing management calls until something else breaks. The honest summary is that the data path is only loosely coupled to the metadata role, so it may survive — not that it always does. ## Why 'still serving' is a dangerous kind of healthy The cluster is frozen. It will keep doing exactly what it is already doing, and it can react to nothing: 1. **The next failure is unanswerable.** If a record-serving node dies while the majority is lost, nothing can record that a different copy now owns its partitions. Those partitions stop being served — not because a copy is missing, but because no one can admit the change that would put it to work. 2. **Deployments and repairs are blocked.** A rolling change, a node replacement or a capacity addition all need cluster state to change, so all of them stall halfway. 3. **Monitoring lies by omission.** Throughput, error rate and consumer progress can all look normal. The failing signal is the one nobody graphs: whether the metadata role currently has its majority. That combination — green dashboards, frozen shape, zero tolerance for the next fault — is why this is a page-someone incident and not a ticket. ## Comparison | | before the majority is lost | while it is lost | |---|---|---| | writes to a settled owner | served | often served; some platforms refuse | | creating or deleting a stream | admitted | refused or hanging | | moving a partition | admitted | refused | | a record-serving node dying | its work is reassigned | its work cannot be reassigned | | tolerance for the next fault | normal | effectively none | ## What an operator actually does The first job is to know. Whether the metadata role has its majority is a first-class signal wherever you can observe it — the count of live coordination members against the majority it needs — and where the role is hidden by a provider, the observable proxy is whether structural changes are being admitted at all. The second job is restraint. While the majority is missing, changes that need cluster state will not go through, and repeatedly retrying them adds noise to an already degraded system. Restoring the missing members — or restoring the network path between them — is what returns the cluster to a state where it can change again. The third is prevention, and it is a shape question, not an incident question: it is why the membership is odd-sized, why a three-member set makes maintenance tense, and why running every coordination member somewhere that can fail together is a bad idea. Those choices are made long before the incident. ## What separates a strong answer A strong answer distinguishes the two jobs cleanly, says that the traffic path may survive while naming that some platforms fail closed, and — the part interviewers listen for — explains that the real damage is the loss of the ability to respond to the next failure rather than the immediate loss of service. A weak answer says "the cluster goes down", which is both too strong for many platforms and, worse, hides the actual risk.