A job holding months of per-key entries must change its grouping key. What are the options, and what does each cost?

## Why a re-key is the hard case Changing the grouping key a stateful step's entries are filed under is the worst of the incompatible edits, and it is worth being precise about why. It is not that the values cannot be decoded — they can. It is that they are **filed in the wrong place**. Every entry sits under a key the new code will never ask for, and because a key determines which bucket it belongs to and therefore which worker owns it, the entries cannot simply be relabelled where they lie: moving them is a redistribution by key, the step that sends every record to the worker that owns its key. That is a job in itself, which is exactly why the routes below are the shape they are. Before choosing, establish two facts, because they eliminate routes: 1. **Do the entries have a bounded lifetime?** If an expiry rule drops an entry a fixed interval after its last write or last access, the transition window has an end. If entries live forever, it does not. 2. **Can the source still supply the period the entries cover?** If not, rebuilding is off the table and the entries are, in the strict sense, irreplaceable. ## The three routes | Route | What happens | Costs | Ruled out when | |---|---|---|---| | **Offline rewrite** | the stored snapshot is read as an ordinary dataset outside the running job, entries are re-keyed, and a new snapshot is written for the new build to start from | job is stopped for the length of the rewrite; the transformation must be written and tested; the snapshot's own format is a compatibility surface | the runtime exposes no way to read or write a snapshot as data | | **Transition window** | the new build writes under the new key and still reads old-key entries, until no old entry can still matter | two code paths in one job, both maintained; more retained bytes while both live; a date to remove the old path, or it is permanent | entries have no bounded lifetime, or the old key cannot be computed from the record any more | | **Rebuild from source** | the step starts with an empty retained set and is refilled from replayed input | the source must still hold the period; a catch-up run; wrong or absent numbers until it is caught up | the source no longer holds it, or the entries were derived from something that cannot be replayed at all | ## Choosing between them The honest decision procedure is short: - If the runtime supports reading and writing a snapshot as data and the stop is acceptable, **rewrite offline**. It is the only route that preserves every entry exactly and finishes at a known moment. - If a stop is not acceptable and entries expire, run a **transition window** and write the removal date into the work item that opens it. The failure mode of this route is not technical, it is organisational: the second code path is never deleted, and two years later nobody knows whether the old branch still fires. - If neither is available, **rebuild**, and treat the period during which the numbers are rebuilding as a consumer-facing fact: say which outputs are incomplete and until when. The mechanics of pushing a long period of historical input through a job is a separate subject with its own owner; what belongs here is that it is the fallback, and that its availability is decided by how long the source retains data — a limit set by somebody else, often without knowing your job depends on it. ## What to do with the entries you cannot rebuild Some retained sets cannot be reconstructed at any price: a running value derived from records the source has since dropped, an entry set fed by an input that no longer exists, an aggregate over a period nobody archived. For those, the offline rewrite is not the cheap option but the **only** one, and if the runtime does not offer it, the honest answer is that the re-key cannot be done without losing history — at which point the design conversation is whether the retained set should have been inside the job at all. ## The judgement this is really testing An interviewer asking this is checking whether you know that the plan is written **before the first deployment**, not at the moment of the re-key. Concretely: an expiry rule, so a transition window can ever end; a stored format with room to move; a note of which entries can be rebuilt from the source and for how long that stays true; and step identifiers pinned, so the re-key is the only problem you are solving rather than one of three. Every one of those is cheap on day one and expensive or impossible on day six hundred.