A nightly address import writes validated rows in one batch at the end, and a malformed row at position 9,998 signalled a failure — why were the earlier good rows lost?

## Two endings, one flush A sequence ends once, in one of two ways: **normally**, or with a **failure**. The accumulate-then-write shape attaches the entire side effect of the job to the first of those. Read it as a conditional and the defect is obvious: *if the sequence ends normally, write everything*. There is no branch for the other ending, so when a failure arrives at position 9,998 the accumulated rows are simply discarded along with the subscription that held them. Notice what this is **not**. It is not a rollback: nothing undid the work, because the work had not happened yet. The rows only ever existed in the accumulator. That distinction matters, because the two failure modes need opposite fixes — a rollback problem needs transaction boundaries, while this needs the write to stop depending on an ending it cannot guarantee. ## Why it survives testing - **Fixtures are clean.** A test file of twenty well-formed rows never raises, so the normal ending always fires and the flush always runs. - **The defect is conditional on a failure landing mid-run**, which is precisely the case the happy-path test omits. - **It scales against you.** If any single record has a small chance of being rejected, the chance that a file of ten thousand contains at least one approaches certainty. The shape is most fragile exactly where it is most used — big nightly loads. - **The symptom is confusing.** The destination is untouched, so operators report "the import did nothing" rather than "the import stopped at 9,998", and the investigation starts in the wrong place. ## What is durable when the failure lands | write shape | durable after a failure at 9,998 | memory held | write cost | |---|---|---|---| | accumulate all, write on normal ending | nothing | the whole file | one large write | | bounded chunks written as they fill | every completed chunk; the open chunk is lost | one chunk | one write per chunk | | write each row as it passes | every row before the failure | one row | one write per row | None of these is universally right. Accumulating gives the largest single write and the cleanest all-or-nothing story, and it is defensible when the destination can accept the whole set atomically and you genuinely want all-or-nothing. It is indefensible when it was chosen by accident, which is the usual case. The chunked middle is where most bulk imports belong: work already done survives, memory is bounded, and the write cost stays reasonable. Be precise about what it costs — the rows in the **open** chunk when the failure arrives are still lost, so the guarantee is "every completed chunk", not "everything before the failure". ## The rerun question Any shape that makes partial work durable creates a second obligation: a rerun must be safe over records already written. That usually means writing by a stable identity so that re-processing a record already present changes nothing, rather than appending blindly and producing duplicates on the second attempt. Engineers who have run these jobs mention this unprompted, because the first production incident after switching to chunked writes is almost always duplicates. And the deeper fix stands above all of these: if a malformed postcode is an expected per-record outcome, it should never have ended the sequence at all. Carry the rejection as a value, and position 9,998 costs you one record on a rejects report instead of a whole night's run. Chunked writing limits the blast radius of a *genuine* stop condition; making expected rejections non-terminal removes most of the stop conditions in the first place. ## What an interviewer wants to hear 1. The mechanism: the flush was conditional on a normal ending, and the failure was a different ending. 2. The correction of vocabulary: nothing was rolled back, because nothing had been written. 3. The structural fix with its cost stated honestly, including the open chunk. 4. The rerun consequence, unprompted. An answer that only says "add error handling" has not identified the defect: the defect is that the entire output of the job was made conditional on an ending no long-running pipeline can promise.