Membership & Data Movement

Writes to that partition fail rather than block: the client's cached owner map still names the dead node, so sends are rejected until a replacement copy is promoted. Records survive only if the writer's retries outlast that gap.

Leadership is a second distribution, separate from where copies sit. Where exactly one node serves each partition (or queue), the work follows leadership rather than storage, so perfectly even copies can still leave one node leading — and serving — most units.

A reassignment plan only declares which node should hold which partition (or queue). Accepting it starts ordinary copy traffic that streams the unit's retained records to the new node, and ownership settles only once that copy has caught up.

A restart takes that node's copies out of service, and while it is down they fall behind. Stopping the next node before they have caught up removes a second current copy and can leave a partition with none.

Stopping the process is indistinguishable from that node failing: its copies stop being current and the units it led must be promoted in a hurry. An evacuation moves ownership off first and finishes only when the node holds nothing.

A number raised at every change of leadership makes a superseded leader's writes rejectable: whoever receives a write compares its stamp with the newest it knows and refuses anything older, without first having to establish whether the sender is alive.

The ceiling caps how much bandwidth a migration's copy traffic may consume, because that traffic competes with production. Set it too high and live reads and writes slow down; set it too low and the cluster sits half-migrated indefinitely, holding copies on both node sets.

A process answering is a node-side fact about itself; what the next step needs is a cluster-side fact about data - that every copy the restarted node holds is back in the caught-up set. The two become true minutes apart.

It costs acknowledged records. A copy outside the caught-up set is missing writes the cluster already confirmed to producers, and promoting it discards them silently — no error reaches anyone. The alternative is keeping every record and staying unwritable for an unknown time.

The cluster cannot tell them apart. A crashed node and a node cut off by the network produce the same observation from outside — silence — so the old leader has to be treated as possibly alive and still accepting writes.

Tolerance must exceed the node's ordinary pauses — garbage collection, a slow disk flush, a congested link — or the cluster promotes replacements for nodes that are alive. Shortening it shortens outages but buys needless promotions, each with its own gap and client churn.

Every stop hands a unit's leadership to a surviving copy that is caught up, and a node that comes back rejoins as an ordinary copy. Leadership accumulates on whichever nodes happened to be alive at the wrong moments and stays until something hands it back.

Because relief has to be copied onto the new nodes, and that copying reads records off the very nodes that are already at their ceiling. The cluster does extra work for hours and gains nothing until the new copies are current.

Because authority is checked where the write must become durable, not where the client sends it. The stale route is refused under an out-of-date generation number, the error travels back, and the client's cached ownership view is corrected rather than the write silently landing on a node nobody reads from.

A leadership-only rebalance changes which node serves each unit and moves no records, so it costs no bandwidth. It costs a brief interruption per unit as service hands over and clients refresh, which means the real decision is how many units to hand over at once, and when.

A bandwidth ceiling caps migration bytes on the wire, not the work that produces them. The source node must read cold historical records off disk, which competes with live reads and can evict the warm data they were being served from; the destination pays a sustained write.

The intermediate mapping is real cluster state, not an operator's intention. Walking away leaves it in force, holding copies on both node sets indefinitely and blocking later plans; cancelling is a deliberate operation that reverts the mapping, discards partial copies and moves finished units back.

The sleep was shorter than the catch-up each returning node owed, so current copies were never restored between steps. The count fell step by step until the loop stopped the node holding the last current copy of a unit.

Sometimes. Detaching storage turns every membership change from a data movement of hours into an ownership handover of seconds, which is worth a great deal to an elastic estate — but it makes one shared store a dependency of every node, and it rarely removes the recent records that still live locally.

Neither extreme survives contact. Continuous restoration prevents skew accumulating but hands traffic back unattended, including to a node that just returned unwell. Operator-run restoration is predictable but relies on someone noticing an invisible drift. The workable policy is automatic above a threshold, with health preconditions and an alert.

Machine-checkable evidence, gathered before the stop and enforced by the tool: no unit of ownership is short of its current copies, the previous step has been paid back, and the loop aborts rather than proceeds when that cannot be shown.

Decide by what a misleading answer costs. Self-demotion after a silence period shortens the time a client can believe it is writing successfully, at the price of stepping aside during blips; relying only on refusal keeps serving through blips but leaves work provisional for longer.

The accountable owner of each stream decides, in advance and per stream, not the operator during the incident. The policy should turn on what the stream carries, whether it can be republished from upstream, and what an outage costs relative to silent loss.

Where records live on shared or remote storage rather than a node's local disk, changing which node owns a unit becomes a metadata handover with no bytes in flight: no bandwidth ceiling, no half-migrated state. Placement decisions, cold starts and a new shared dependency remain.