When every input read is remote, what work inside a running job is still tied to one particular machine?

## Two survivors Once the input is remote blob storage, no machine is nearer to the input than any other and the scheduler's input preference is vacuous. But a running job produces data of its own, and that data does have a home. **1. Bytes the machine produced for itself.** Many runtimes hold intermediate results — output a later step will consume — on the local disk of the worker process that produced them. Reading those back on the same machine is a local read; reading them from anywhere else means a fetch over the network. So the work that consumes them has a genuinely nearer candidate, and that candidate is one specific machine. **2. Work bound to a key.** Where a job accumulates something per key, the accumulated value has to live with one worker, the one that owns that key. A record for that key arriving anywhere else either has to be routed to the owner or has to have the state fetched or rebuilt before it can be processed. That routing is a nearness constraint in exactly the same sense as the original one, and the mechanics of key-bound state are owned by the long-lived-state subject. Notice what both have in common and how they differ from input placement: | | input bytes (remote store) | intermediate bytes | key-bound state | |---|---|---|---| | is any machine nearer? | no | yes, the producer | yes, the key's owner | | created by | storage, before the run | the run itself | the run itself | | survives machine loss? | yes, untouched | no, must be recomputed or refetched | only via a saved recovery point | ## Affinity is the opposite of interchangeability This is the part worth saying out loud in an interview. The reason remote storage was worth its bandwidth cost is that compute machines became interchangeable and therefore disposable. Every scrap of nearness a run creates for itself claws some of that back: - A machine holding intermediate bytes cannot be reclaimed for free. Whatever depended on those bytes has to be recomputed from its inputs, or the bytes fetched elsewhere first. - A machine holding key-bound state cannot be removed without moving that state, and it cannot be replaced by a fresh machine without restoring it. - Adding machines mid-run does not immediately help the work that is pinned, because the pinned work cannot follow. How far this goes — what is recomputed, what is restored, how machines are added or dropped mid-run — belongs to the failure-handling and machine-supply subjects. What belongs here is the cause: the run manufactured its own locality, and locality is what makes a machine special. ## Where runtimes genuinely differ This is the claim most likely to be wrong if you generalise from the engine you know: - **Batch-lineage runtimes** typically materialise intermediate results to the producing machine's own disk, so the first survivor is strongly present. **Record-at-a-time continuous runtimes** often push records across the network as they are produced with no materialisation at all, so they have little or none of it — their only affinity is state. - The **oldest model in this class**, which writes every intermediate result to disk between the two halves of a job, has the first survivor in its most extreme form, and it is exactly why that model's recovery story is simple. - Where continuous work is run as a **rapid succession of small finite jobs**, intermediate data is short-lived and the per-key accumulation is kept somewhere the next small job can pick it up, which makes the affinity look different again. - Some deployments store intermediate results away from the compute machines on purpose — the same trade as the input, made again — which removes the first survivor deliberately so that machines stay disposable. ## How this shows up in practice 1. **A machine is reclaimed late in a long run and the run takes a large step backwards.** The bytes it held were not input; they were produced, and nothing else has them. 2. **A job is restarted at a different width and the first records are slow.** Key ownership moved, so accumulated state had to be re-established before the work could proceed. 3. **A cluster scaled out mid-run and throughput did not rise proportionally.** The pinned work could not move to the new machines. ## The interview form Asked "does data locality still matter?", the answer that separates a senior candidate is: not for reading the input, which is now equidistant from everywhere by design, but yes for what the run makes — and the price of that surviving nearness is that some machines stopped being disposable. Then say which of the two survivors your runtime actually has, because they are not the same in every engine.