A reassignment has run for hours with many units still in the intermediate mapping — how does cancelling it differ from walking away?
answer
- in-flight state is real, not intent
- both node sets hold copies meanwhile
- nothing expires on its own
- cancelling discards and moves back
- decide on remaining catch-up, not elapsed time
basics
~20 sThe intermediate mapping is real cluster state, not an operator's intention. Walking away leaves it in force, holding copies on both node sets indefinitely and blocking later plans; cancelling is a deliberate operation that reverts the mapping, discards partial copies and moves finished units back.
solid answer
~50 sWhile a plan is executing, the union of the old and the new node sets holds copies of each unit of ownership, and that union is recorded cluster state. Walking away does not undo it: the copying continues at whatever rate the ceiling allows, disk stays committed at both ends, extra-copy alarms keep firing, and a later plan touching the same units is generally refused or queued behind this one. Cancelling is a real operation — it reverts the declared mapping to the original node set and stops the transfer — but it is not free either: partially copied data on the destination is discarded, and any unit that already completed has to move *back*, which is a second migration. The usual right answer is neither: finish the units that are nearly done, and cancel only the ones that have barely started.
go deeper
The takeaway is that an unfinished migration is a real state the cluster is in, not a pending intention, and that someone has to finish it or cancel it deliberately.
Explain what the intermediate mapping is: both the old and the new node sets hold copies, disk is committed at both ends, and copy counts read as abnormal until the plan resolves one way or the other.
Show the judgement. Price walking away, cancelling and finishing against remaining catch-up and its trend, and reach for the split option — finish what is nearly done, cancel what has barely started.
Turn it into policy: a migration needs an owner, a projected completion time and an abandon criterion before it starts, and the estate needs to know in advance what its platform can and cannot cancel.
## The intermediate mapping is real state When a reassignment plan is executing, each unit of ownership — a partition, or on queue-shaped brokers a queue — is not simply "on its way" from one node set to another. The cluster records an intermediate mapping in which the union of the original holders and the intended holders all hold copies. That union is durable state. It survives restarts, it is what monitoring reports, and it is what later operations check before they run. That has three immediate consequences: - **Disk is committed at both ends.** Until the plan resolves, the estate is storing more copies of those units than either the old or the new mapping calls for. - **Monitoring reads as abnormal.** Copy counts exceed the target, which is indistinguishable at a glance from a genuine over-replication problem. - **Later work queues.** A second plan touching the same units is generally refused or deferred while one is in flight, so the half-finished move blocks unrelated maintenance. ## Walking away Nothing expires. The plan stays in force, the destination keeps fetching, and the state above persists for as long as that takes — which, under a low bandwidth ceiling or against a write rate the copy cannot out-run, may be forever. "Leave it and see" is a decision to keep paying all three costs with no end date, and it is the option operators choose by accident more often than deliberately. ## Cancelling Cancelling is a deliberate operation and it is genuinely different: it reverts the declared mapping to the original node set and stops the migration traffic. It is also not a free undo. 1. **Partial work is lost.** Records already streamed onto a destination that never joined the caught-up set are discarded; a later retry generally begins the transfer again rather than resuming, though designs differ in how much they can reuse. 2. **Completed units move back.** Any unit that already finished, joined the caught-up set and had ownership handed over is now in the new place. Reverting the mapping means a *second* migration in the opposite direction, with the same byte cost as the first. 3. **The cancellation itself takes time.** It is a change of declared state followed by whatever cleanup the platform does, not an instant rollback. ## The three options, priced | Option | What it costs | When it is right | |---|---|---| | Walk away | Disk on both node sets, blocked maintenance, no end date | Never deliberately | | Cancel | Partial transfers discarded; finished units move back | Most units have barely started, or the plan itself was wrong | | Finish | More hours of copy traffic, possibly at a raised ceiling | Remaining catch-up is small relative to what has moved | ## How to decide The deciding number is not elapsed time; it is **remaining catch-up** across the units still in flight, and its trend. - If remaining catch-up is falling and small, finishing is cheaper than reverting, because reverting throws away completed work and adds a return trip. - If it is flat or growing, the move cannot converge at the allowed rate; either raise the ceiling or reduce concurrency, and if neither is available, cancel rather than sit in the intermediate state. - Where the tooling allows it, split the difference: let the nearly-complete units finish and cancel only the ones that have barely started. That is usually the cheapest outcome and the one a senior candidate is expected to reach for. - Whatever you choose, choose it. The defect this question is really testing is treating an in-flight plan as something that can be ignored. ## Where designs differ How much of a partial transfer can be reused on a retry varies: some platforms resume from what is already on the destination's disk, others start again. Whether a plan can be cancelled per unit or only as a whole varies. Whether a second plan is refused outright or merged with the one in flight varies. And on designs with **detached storage**, where records live on shared or remote storage, there is no intermediate mapping of this kind at all — a change of owner either happened or it did not, because no bytes were in flight to be half-copied. Say which of these you are assuming, and say that you would establish it before starting a migration you might want to abandon.
- Why is cancelling a nearly-complete move usually the most expensive option?Because the work is already paid for. Units that finished have moved and joined the caught-up set in their new place; reverting the declared mapping sends them back, which is a second transfer of the same bytes. Meanwhile the units still in flight lose whatever they had copied. Finishing costs only the remaining catch-up, which by definition is the smaller number.
- What would you watch to know an in-flight plan is not going to converge?Remaining catch-up per unit over time. Falling steadily means slow but finishing, and you can project a completion time from the slope. Flat or growing means the allowed copy rate is at or below the unit's incoming write rate, so waiting changes nothing — the only outcomes are more bandwidth, fewer concurrent transfers, less load, or a deliberate cancellation.
- What should you establish before submitting a large plan you might want to abandon?Whether the platform can cancel per unit or only the whole plan, whether a retry resumes from partial data on the destination or starts over, and whether a second plan on the same units is refused while one is in flight. Those three answers decide whether a bad plan costs you an hour or a week, and they are far cheaper to learn before the migration than during it.
saying these in an interview costs you the question
- Treats an in-flight plan as intent that can simply be ignored.
- Assumes an unfinished plan expires and rolls back by itself.
- Calls cancelling a free undo with no work lost.
- Forgets that completed units have to move back on a revert.
- Decides on elapsed time rather than remaining catch-up.
- Ignores that the intermediate state blocks later maintenance.