A job is being taken down for a deploy: what decides whether you can stop it and submit the new version, or must save first?

## Two stops that look identical from outside When a job goes away, only two things can be true of the work it was in the middle of: it wrote down enough to resume, or it did not and something will have to be done again. A crash gives you no opportunity to write anything down. A deliberate stop — for a new build, a changed setting, a machine rotation — hands you that opportunity, and this whole subject is about spending it well. The first decision is whether you need to spend it at all. ## The test: what does this job carry between records? Ask what the job could not obtain again if every one of its processes disappeared this second. - **It carries nothing between records.** It reads a record, maps it, filters it, enriches it from an external lookup, writes it, and retains no trace. Whatever sits in a worker's memory at any instant is derived from the record currently in hand. Nothing here is irreplaceable, because reading the input again reproduces it. - **It has accumulated something.** Running totals per key, one side of a match waiting for its partner, the last value seen per customer, the set of identifiers already seen. That content exists *only* because this job has been running — sometimes for weeks — and no future record can reconstruct it. The second kind is a real storage system that happens to live inside a job, and stopping it carelessly destroys a database nobody remembered was there. ## Case one: stop and resubmit — but not for free Even a job carrying nothing has one durable artifact worth protecting: the recorded read position. If it does not survive the stop, the new version starts wherever its defaults say. Starting from the earliest retained record reprocesses everything and writes the output again; starting from the newest leaves a hole exactly the size of the outage. So the cheap case is not *nothing* — it is *one* thing, and it is usually already durable because the running job was maintaining it anyway. A **finite** job — one over a fixed, bounded input — is a further special case. Stopped halfway, it is normally not resumed at all; it is re-run, and the cost is the compute already spent rather than any knowledge. How much of that spend survives depends on the engine: in **the two-phase disk-to-disk model**, where each phase materialises its output to durable files before the next phase reads them, completed work may still be on disk and reusable, while an engine that streams results between steps in memory generally has to redo them. ## Case two: write a restart point before exiting For the accumulating job the sequence is: stop reading new input, let the work already inside the job finish, write a **restart point taken on purpose** — a recovery point the operator asks for before stopping, where a *recovery point* is a durable copy of everything the job would otherwise lose, held on **durable shared storage**, meaning storage outside any one machine and reachable from whichever machine resumes, because a copy on the dead machine's disk is no recovery point — and only then exit. The new version is submitted pointing at that artifact instead of at the beginning of the input. ## Where engines differ, and they do | | Carries nothing between records | Has accumulated something | |---|---|---| | What must survive the stop | The recorded read position | That position **and** the accumulated contents | | Deploy procedure | Stop, submit new version, resume | Drain, write a restart point, submit against it | | Cost of getting it wrong | Reprocessing or a gap in output | Silent loss of weeks of arithmetic | | Outage window dominated by | Acquiring machines, resubmission | Writing, then reading back, what was held | And across the engines themselves: 1. A runtime that serves an endless input as **repeated small finite jobs** — cutting it into short bounded pieces and running a complete little job over each — already reaches moments when nothing is in flight, so the quiet point a clean stop needs exists for free. 2. A runtime doing **record-at-a-time processing**, where each record traverses the whole job as it arrives and workers hold results between records, has no natural quiet moment and must manufacture one. 3. Whether an operator-requested restart point is a distinct kind of artifact or merely a retained ordinary one varies by engine, as does whether the engine promises a later build can read it. One thing does not vary: the platform restarting your process, or rescheduling it onto another machine, restores no work at all. It gives you a running process with an empty memory, which is the crash case wearing a deploy's clothes.