A six-hour fit step in a nightly forecasting graph dies near hour five — what must its checkpoints record to resume?

## Why weights alone are not resumable state The fit step reads ninety days of per-store, per-item features and takes about six hours. It dies near hour five. Restarting from zero costs a nightly refresh; resuming costs an hour — but only if what was written is genuinely enough to continue from. A checkpoint that holds only model parameters produces a resumed run that is *not* the interrupted one: it restarts the input from the beginning, re-draws the same random choices, and double-counts whatever it had already consumed. To match the uninterrupted run up to what the checkpoint records, it needs: - **Learner state** — model parameters, plus whatever the optimisation carries between updates (momentum, accumulated gradients, the trees built so far and the current residuals). - **Input position** — the shard, epoch and row cursor already consumed, so the resume continues rather than repeats. - **Random-number state** — the generator's position, not merely the seed, so sampling, shuffling and initialisation continue the same stream. - **Accumulated metrics** — running training loss or error, so the resumed run's curve is continuous and early-stopping logic is not reset. - **Binding identity** — the run, the step, the declared input the fit is reading and the hyperparameters in force. A checkpoint is only valid for the thing it was taken from. - **A completeness marker** — the record is either fully published or not present. ## Writing a checkpoint so it survives the crash that caused it The checkpoint is written by the process most likely to die. 1. Write to a **new location** each time; never modify the previous one in place. 2. Flush and confirm the bytes are durable before anything points at them. 3. Publish by an **atomic pointer swap** to the new location, so a reader sees the previous complete checkpoint or the new complete one. 4. **Retain at least the previous complete checkpoint.** Keeping exactly one means a crash mid-publish leaves nothing usable. 5. **Validate on resume** — read the record, confirm its input identity and hyperparameters match the step about to run, and fall back to the previous checkpoint if they do not. A fit that overwrites one file in place breaks exactly this: the crash that truncates it also removes the only thing that would have saved the five hours. ## Where the checkpoint lives relative to the graph The fit is one node. When it fails, the graph retries **that node**, and the retry must find the checkpoint the previous attempt wrote — so the checkpoint location is keyed by **run and step**, not by attempt. Key it by attempt and every retry starts from zero while cheerfully writing checkpoints nobody will read. That also sets the contract with the rest of the graph: - the step is not complete until its **declared output** — the fitted model artifact — is published atomically, exactly like any other step; - checkpoints are internal scratch, not the output, and are discarded once the step completes; - resuming does not weaken the output guarantee, because the published artifact is still produced by one atomic publish at the end. ## How often to checkpoint Checkpointing is a trade between write overhead and redone work. If a checkpoint costs *C* to write and failures arrive on average every *M*, the spacing that minimises total waste is near **sqrt(2 x C x M)**. With C = 60 seconds and M = 6 hours (21,600 seconds): - spacing ≈ sqrt(2 x 60 x 21,600) = sqrt(2,592,000) ≈ 1,610 seconds ≈ **27 minutes**; - write overhead ≈ 60 / 1,610 ≈ **3.7 percent** of runtime; - expected redone work per failure ≈ half the interval ≈ **13 minutes**. So on a six-hour fit, roughly thirteen checkpoints cost about fourteen minutes in total and cap the loss from any single crash at about a quarter of an hour. Checkpoint every minute and the overhead alone exceeds an hour; checkpoint hourly and the average crash costs thirty minutes. For a step that finishes well inside one interval, checkpointing is not worth writing at all — rerun it. ## What resume must not change | Aspect | Must match the interrupted run | Why | |---|---|---| | Declared input identity | Yes | Resuming onto different data produces a model fitted to two datasets | | Hyperparameters | Yes | A mid-run parameter change makes the curve uninterpretable | | Random-number state | Yes | A reset stream re-draws choices already made | | Input cursor | Yes | A reset cursor re-consumes rows and skews the fit | | Worker placement | No | Any worker may pick the step up, provided the checkpoint is readable | | Wall-clock time | No | The record is bound to the run, not to when it resumed | The rule behind the table: a resume is legitimate only when the step it re-enters is the same step, fitting the same declared input, under the same parameters. Anything else is a new run wearing an old run's weights.