In Luigi, why can a crashed task leave an output that makes the next run skip it?

## The failure mode A task runs at 02:00, opens `/data/agg/orders_2026-08-20.csv`, writes 400,000 of 900,000 rows, and the worker is OOM-killed. At 02:30 the pipeline is re-invoked. Luigi rebuilds the graph, calls `complete()` on this task, the default implementation asks the Target whether it exists, the filesystem says yes — and the task is skipped. Every downstream task then runs happily on a truncated file. Nothing errors; the numbers are just wrong. This is the single most common production complaint against target-based completion, and it is a favourite senior interview scenario precisely because the framework is behaving exactly as designed. ## Why Luigi cannot tell Luigi keeps no run history to consult, so it has no memory that the earlier attempt died. Its only question is whether the promised artifact is present. Existence is a *proxy* for correctness, and a partially written file breaks the proxy: it is present but not correct. The framework cannot fix this on your behalf, because only the task knows what a complete result looks like. What it can do — and does — is give you the tools to make presence mean completeness. ## Make the appearance of the Target the commit point The rule is that data must become visible at its final location only when it is whole. Everything else follows from it. **Use the Target's own writer.** `luigi.LocalTarget.open('w')` does not open the final path. It writes to a temporary file and moves it into place when the stream is closed cleanly; if the process dies first, the final path never appears and the task is correctly still incomplete. This is why idiomatic Luigi code reads `with self.output().open('w') as f:` rather than `open(self.output().path, 'w')` — the latter throws away the protection you were given for free. **Use temporary_path() for external processes.** Plenty of tasks do not write bytes themselves: they shell out to a tool, hand a path to a library, or run a job that produces a file. `FileSystemTarget.temporary_path()` is a context manager that yields a scratch path and moves it to the Target's real path only if the block exits without an exception: ```python def run(self): with self.output().temporary_path() as tmp: subprocess.check_call(["render-report", "--out", tmp]) ``` **Stage-and-move for directories and object stores.** When the output is a directory of part files, or a prefix in an object store where there is no cheap atomic rename, write under a temporary prefix and move or copy into place at the end. Where even that is expensive, invert the check: write the data first, write a small `_SUCCESS` marker last, and point the Target — or an overridden `complete()` — at the marker. The marker is small enough that its write is effectively atomic, and it appears only after the data is whole. **Databases have their own version.** Write to a staging table and swap, or wrap the load in a transaction and let the commit be the visible event. A Target that checks for rows in a table being loaded row-by-row has the same partial-visibility bug as the file case. ## Strengthen the completion check where the shape is unusual When atomic publication genuinely is not available, override `complete()` to check something that partial output cannot fake: the presence of a marker, a recorded row count matching a manifest, a checksum. Keep it cheap and side-effect free, because Luigi calls it on every node on every scheduling pass. ## Operational habits around it Even with atomic writes, keep two habits. First, make outputs easy to delete at a known granularity — one directory per parameter value beats a single append-mostly file — because deleting the Target is Luigi's only rerun lever. Second, resist the temptation to "fix" incidents by hand-deleting files as a routine: if you find yourself doing it weekly, the pipeline is publishing non-atomically somewhere and the deletion is a symptom, not a remedy. ## Interview framing The tell of a weak answer is blaming the scheduler, or proposing that Luigi should validate contents before marking a task done. The strong answer names the mechanism — completion equals existence, so publication must be atomic — then names the concrete tools in order: the Target's own `open()`, `temporary_path()` for external writers, staging plus move or a `_SUCCESS` marker for directories and object stores, and a stricter `complete()` as the last resort. It also acknowledges the honest limitation: atomic publication protects against a crash mid-write, not against a task that runs to completion and produces confidently wrong numbers.