What happens to vBuckets and connected clients during a Couchbase rebalance?

## What rebalance is for A Couchbase bucket is split into a fixed set of vBuckets, and the cluster map records which node holds the active copy of each and which nodes hold its replicas. **Rebalance** is the operation that changes that map and physically moves the data to match it. You run it after adding nodes, after marking nodes for removal, after a failover (to restore the missing replicas), and after changing a bucket's replica count. Crucially, rebalance is not a repair tool for a *broken* node. A node that is down or unreachable is handled by failover; rebalance is for planned topology change and for restoring redundancy afterwards. ## How one vBucket moves The unit of work is a single vBucket, and the move is designed so that the data is never unavailable: 1. The destination node opens a **DCP** (Database Change Protocol) stream from the node that currently owns the vBucket. 2. It **backfills** — receives the existing items in that vBucket — and then continues receiving live mutations as they happen. 3. When the destination is caught up, ownership is handed over and the cluster map is updated to point at the new node. 4. The old copy is discarded once it is no longer needed. Until step 3, the original node is still the active copy and still serves reads and writes. That is why a rebalance is an online operation: at any instant every vBucket has exactly one node designated active, and clients are told which one it is. The cluster performs several such moves concurrently rather than one at a time, and the concurrency and throughput are tunable so a rebalance can be made slower and gentler on a loaded cluster. Replica vBuckets are rebuilt by the same mechanism, which is why a rebalance after a failover costs real I/O even though no client-visible ownership needs to change: the surviving actives must stream full copies to whichever nodes are now supposed to hold their replicas. ## What connected clients see SDKs cache the cluster map, so a topology change has to reach them. Two mechanisms cover it. The cluster pushes updated configurations to connected clients, and as a backstop, any node that receives an operation for a vBucket it no longer owns replies **"not my vBucket"** with the current configuration attached. The SDK installs the new map and retries the operation, inside the caller's timeout. The practical consequences: a modest rise in operation latency and in retries during the rebalance; higher network and disk utilisation on both source and destination nodes; and, if the cluster was already near saturation, a real risk that added rebalance traffic pushes latency past application timeouts. That is the main argument for rebalancing during a quieter window and for throttling it rather than running it flat out. ## Swap rebalance If you add one node and remove another in the same rebalance, Couchbase performs a **swap rebalance**: the vBuckets from the outgoing node are streamed directly to the incoming node, and the rest of the cluster is left alone. This moves far less data than removing a node in one rebalance (which redistributes its vBuckets over all survivors) and then adding a node in a second (which pulls a share back from everyone). Replacing hardware node-for-node is the canonical case, and doing it as two separate rebalances is a common and expensive mistake. ## Stopping, failing, and resuming A rebalance can be stopped by the operator, and a rebalance that fails — a node becomes unreachable mid-move, for example — leaves the cluster in its previous consistent state with the map only partially advanced; you fix the underlying problem and rebalance again. Because vBucket moves are individually complete before ownership flips, an interrupted rebalance never leaves a vBucket half-owned. It does leave the cluster unbalanced, which is a monitoring concern, not a correctness one. ## Rack and group awareness When server groups are configured, rebalance also honours the constraint that a vBucket's active and replica copies live in different groups, so a whole rack or availability zone can be lost without losing a vBucket entirely. That constraint influences which moves are legal and can make a rebalance move more data than a naïve balance would. ## Interview traps Weak answers claim the bucket is read-only or unavailable during a rebalance, or that ownership flips first and the data is copied afterwards. Both invert the design. Another common miss: assuming that after a failover the cluster is fully protected again — it is not; replicas are only recreated by the subsequent rebalance.