In a volatile tier that fails over without a human, which parties can be entitled to declare the primary dead, and why is one observer never enough?

## Why the question has to be asked at all "The tier fails over automatically" names an outcome, not a component. Something concrete concluded that the primary was gone and performed the reconfiguration, and which something it was changes the smallest safe deployment, what happens on the isolated side of a network problem, and how a caller ends up addressing the new node. A stem that quietly assumes one of the three marks the other two wrong, which is why this is worth asking explicitly. ## The three deciding parties **1. The surviving peer nodes.** The data nodes themselves carry the monitoring role: they probe each other, and the survivors reach agreement that one of them is gone before one is promoted. Nothing extra to deploy, and the deciding party dies exactly when the data does. The sizing consequence is the one people get wrong: you need enough independent voices that agreement is still possible after losing one, so two nodes is never a safe count — neither survivor can tell being alone from being cut off from a peer that is fine. In practice this means at least three participating nodes, and deployments commonly use odd counts so that a split of the participants has an unambiguous larger side. **2. A set of dedicated observer processes.** Separate processes whose only job is to watch, agree and reconfigure. They are deployed beside the tier but not on it — an observer sharing a host with the primary dies with the primary and its vote goes missing at the worst possible moment, which also argues for spreading them across whatever your failure domains are, racks or availability zones. Because they already hold the current view, they are typically also the thing a caller asks "who is the primary now?", which collapses part of the rediscovery problem. The cost is a second thing to deploy, patch and monitor, and a deployment where the observers are healthy but the data nodes are not is its own operational state. **3. A controller outside the data plane.** In a managed offering the decision is made by the provider's control plane, which is not part of the request path at all. Usually the caller keeps one stable address that the controller re-points, so failover is nearly invisible from the caller side. The cost is that the silence threshold, the candidate-selection rule and the resulting windows are the provider's: you measure them, you do not set them. | Arrangement | Smallest safe deployment | How the caller gets the new address | What it costs you | |---|---|---|---| | Surviving peers | At least three participating nodes, spread across failure domains | The caller asks a node, or holds a map it refreshes | The data nodes carry the monitoring role and the decision dies with them | | Dedicated observers | Several observers, off the data hosts and spread across failure domains | The caller asks the observers for the current primary | A second component to deploy, patch and monitor | | External controller | Whatever the provider runs; you deploy nothing | One stable address the controller re-points | Windows and rules you cannot inspect or tune | ## Why one voice is never enough Two independent reasons, and candidates usually have only the second. 1. **A single observer cannot tell the two failures apart.** From one vantage point, "the primary is dead" and "my link to the primary is cut" produce exactly the same evidence: silence. A single severed link between an otherwise-healthy observer and an otherwise-healthy primary would, if that observer could act alone, trigger a promotion nobody needed — and the healthy primary keeps serving whatever callers can still reach it. Requiring several independent voices to agree before acting is what makes the evidence mean something, because a cut affecting one observer's path does not silence the others. 2. **A single observer is a single point of failure.** If the one thing entitled to decide is itself down when the primary fails — or was on the same host, or behind the same failed switch — nothing decides, and you have an automatic failover mechanism that produces a manual failover. This is also why observer count and data-node count are separate sizing exercises. Adding a fourth replica does not improve the decision; adding a third observer, on an independent host, does. ## What to say in an interview Name the three arrangements as a set, say which one the system you are describing uses, and give its smallest safe deployment and its address-discovery story. Then give the single-voice argument in the epistemic form — one vantage point cannot separate a dead node from a dead link — rather than only as "you need more than one for redundancy". The first answer shows you know why agreement is required; the second shows only that you know it is.