A retraining graph rerun after a pricing fix reused a cached aggregate step, so the fix never reached the model — why?

## How a step cache decides to skip In the nightly demand-forecast graph, most branches do not change between runs — the store dimension, the holiday calendar, the price list. Recomputing them nightly is waste, so the graph asks, for each step: > *Do I already hold an output produced from exactly these declared inputs, by exactly this step definition?* If yes, the step is skipped and its previous output stands in. The key behind that question covers two things, and only two: - the identity of every **declared input**; - the step's own **definition** — its code version and its parameters. The cache is therefore only as truthful as the declarations. It is not watching the file system, and it does not know what the step's body actually touched. ## The hidden edge The pricing fix went into a reference table that the aggregate step reads to convert units into price-adjusted demand. The step's declaration listed the sales events and the store dimension; the pricing table was read directly in the body and never declared. So: 1. The fix landed in the pricing table. 2. The graph was rerun for the affected dates. 3. The aggregate step's declared inputs were unchanged and its code was unchanged, so the key matched the previous run. 4. The step was skipped and its stale output reused. 5. The fit step's own inputs were therefore unchanged too, so it was skipped as well — the skip propagated down the branch. 6. The published model was byte-identical to yesterday's, and every step was green. That last property is what makes this class hard: **there is no failure**. The run is fast and successful, which looks like the caching working perfectly. ## Confirming it 1. **Compare the output's provenance to the fix.** The reused partition was produced by an earlier run than the one that carried the fix. A step whose output predates the change it was meant to absorb is the whole diagnosis. 2. **Rerun the branch with caching disabled** for the same dates and diff the outputs. If the no-cache output differs, the cache key is missing an input. 3. **List what the step body reads** against what it declares. Reference tables, configuration blobs and lookup files are the usual omissions, because they feel like environment rather than data. ## The two ways a key goes wrong | Symptom | Cause | Consequence | |---|---|---| | Steps skip when they should rerun | The key omits something the step reads — an undeclared reference table, an unpinned code or parameter version | Silent staleness: the fix never lands, the run is green | | Steps never skip | The key includes something that changes every run — a wall-clock read, a per-run temporary path, a mutable pointer resolved inside the step | No reuse at all: full-cost nights and a backfill that cannot be throttled | Both are declaration defects, and they pull in opposite directions, which is why teams tend to overcorrect from one straight into the other. ## Making declarations honest - **Every read is an input.** Reference tables, price lists, holiday calendars, category mappings. If the step's output would change when that data changes, it belongs in the declaration. - **The step definition is part of the key.** A change to the aggregation logic or to a parameter alters the output for identical data; if it is outside the key, a genuine code fix is skipped exactly like the pricing fix was. - **Resolve mutable pointers once, at the top of the run.** A step that resolves a `latest` pointer inside its own body has an input that is not stable across attempts; resolve it once and pass the resolved identity down, so every step in the run sees the same thing. - **Keep ambient reads out of the body.** Wall-clock reads, environment lookups and per-attempt temporary paths make a step either uncacheable or wrongly cached; pass them in as declared parameters instead. - **Make caching opt-out per step.** A step that genuinely cannot declare everything it reads should be marked never-skip rather than quietly wrong. - **Rebuild without cache periodically.** A scheduled full recomputation compared against the incremental one is the audit that catches a hidden edge before a stakeholder does. Platforms differ in how much of the surrounding environment they fold into the key — some include the step's runtime image, some only the declared data — so the safe habit is to pin the step's version explicitly rather than assume the platform noticed it. ## The operational lesson Step caching is a claim about **completeness of declarations**, not about data freshness. The moment a step reads something it did not declare, the graph's skip logic becomes a bug generator, and it generates bugs that look like successful runs. When a fix demonstrably fails to reach the model while everything is green, the undeclared read is the first place to look.