When Couchbase auto-fails-over a data node, what happens to its vBuckets and what must you do next?

## Failover versus rebalance These are different operations for different situations and conflating them is the classic Couchbase interview error. **Rebalance** is for planned change on a healthy cluster: it moves data before changing ownership, and loses nothing. **Failover** is for a node that is already gone or is being removed abruptly. It changes the cluster map immediately, without moving data first, by promoting replicas that already exist elsewhere. It is fast by design and it can lose the tail of writes that had not yet replicated. ## The three flavours **Hard failover** takes the node out immediately. Use it when the node is dead or unreachable. Data that lived only in that node's memory is lost. **Graceful failover** is for a node that is still healthy and responsive — you are taking it out for maintenance. It drains the active vBuckets to their replicas first, so no data is lost, then promotes and removes the node. It takes longer than hard failover because it waits for that sync. **Automatic failover** is the cluster performing a hard failover on its own when a node stops responding for longer than the configured auto-failover timeout (default 120 seconds; the minimum configurable value is much lower). It is what keeps a cluster available at 3 a.m. without a human. ## What happens at the moment of failover The orchestrator rewrites the cluster map: for every vBucket whose active copy lived on the failed node, a replica copy on a surviving node becomes active. Clients receive the new configuration by push, and any request still aimed at the old owner gets "not my vBucket" with the current map attached and retries. From the application's perspective there is a short window of errors and elevated latency, then normal service. What is lost is exactly what the failed node had that nobody else had: mutations acknowledged from its memory but not yet replicated. This is where durability levels earn their cost — a write acknowledged under `majority` was on a replica before the client saw success, so it survives the promotion. This is also why running with `num_replicas = 0` and expecting failover to be safe is nonsense: there is nothing to promote, and the vBuckets are simply gone. ## After failover: the cluster is not healthy yet This is the part candidates skip. Post-failover the promoted vBuckets have one fewer copy than the bucket's configured replica count. In a one-replica bucket they now have **no** replica at all — a second node failure loses data outright, and durable writes to those vBuckets become impossible. Nothing recreates replicas automatically. You must run a **rebalance**, which streams fresh replica copies onto the remaining nodes (or onto a replacement node you add in the same rebalance). If the failed node comes back and its data files are intact, you can add it back with a **recovery type**: *delta recovery* reuses the node's on-disk data and only catches it up on what it missed, which is far cheaper; *full recovery* discards its files and re-streams everything. Delta recovery is only viable when the node's storage genuinely survived and the divergence is small. ## Why auto-failover has guardrails Automatic failover is deliberately conservative because a failover is unrecoverable in the direction of data: - **A timeout** (default 120 seconds) before it fires, so a GC pause, a brief network partition, or a node restart does not trigger it. - **A cap on automatic events** before an operator must intervene. This prevents a cascade where a flapping network fails over node after node until the cluster cannot serve at all. The budget is restored by a successful rebalance. - **Safety refusals.** The cluster declines to auto-fail-over when doing so would clearly cause unavailability or guaranteed data loss — most notably in very small data-service deployments, where auto-failover requires at least three data nodes so a partition cannot leave two halves each failing the other over. ## Server groups When server-group (rack/zone) awareness is configured, replicas are placed in a different group from their actives, so an entire group can be lost and every vBucket still has a copy. Recovering from a whole-group failure is still a failover of all its nodes followed by a rebalance, and you need the spare capacity in the surviving groups for that rebalance to succeed — capacity planning, not a runtime feature. ## Interview traps Saying "failover restores full redundancy" is wrong. Saying "use failover to remove a healthy node" is wrong — that is graceful failover at best, a rebalance-out at correct. Claiming auto-failover fires instantly ignores the timeout that exists to prevent flapping. And attributing zero data loss to hard failover ignores the unreplicated memory tail unless durability was requested.