A read in pieces still exhausted memory: every piece was appended to a list and joined at the end. Where was the peak?
answer
- where is the peak, really
- the pieces are still referenced
- the join allocates alongside them
- reduce or write, then release
basics
~10 sAt the combination step. Every piece is still referenced by the list while the joined result is allocated alongside it, so the peak is at least the finished table plus all the pieces.
solid answer
~50 sAt the combination, not during the reads. A list holds a reference to every piece, so nothing was released; the join then allocates the finished table while all of them are still alive. Peak is therefore at least the table plus the pieces, which is worse than the one-shot read the loop was supposed to replace. How much worse depends on the design: an eager one that copies pays the full doubling, while a design that is immutable by default and can share untouched buffers between the pieces and the result pays less — but the pieces are resident either way. The loop bounds memory only when each turn reduces its piece to something much smaller, or hands it to a writer that stays open across turns, and then releases it before the next turn is asked for.
code
pseudocode · 14 linespieces = []
for piece in reader.next_piece(rows_per_turn = P):
pieces.append(clean(piece))
result = combine_rows(pieces) # peak: result + every piece, all still alive
running = new_running_result()
out = open_writer(destination)
for piece in reader.next_piece(rows_per_turn = P):
cleaned = clean(piece)
running.absorb(cleaned)
out.append_rows(cleaned)
drop(cleaned) # last reference gone before the next turn
out.close()go deeper
Know that reading in pieces only helps if you finish with each piece and let it go before asking for the next one. Keeping them all defeats the point entirely.
Explain what is resident at the moment the pieces are combined: every piece, still referenced by the collection, plus the result being allocated beside them.
Diagnose it from where the failure lands, after the last read rather than during it, and rewrite the loop so each turn reduces or writes its batch and then releases it.
Recognise it as a standing trap rather than one bug. A loop that looks bounded and is not will be written again by the next person, so the shape belongs in whatever the team reviews against.
## Where the peak actually is The loop looks bounded. Each turn asks for a batch of records, so at no point does the reader hold the whole file, and that part is true. What the reader holds is not what the process holds. The list is the problem: appending a batch to it keeps a live reference to that batch, so nothing the loop has read has ever been released. After the last turn the list contains the whole file, in pieces, at full width. Then comes the combination. Building one table out of forty pieces means allocating somewhere to put the result, and while that allocation happens the list is still holding all forty. **Peak is at least the finished table plus every piece.** That is strictly worse than the one-shot read this loop was written to avoid, which would at least have built the table once. ## Why the combination is the worst possible moment Three things are true at that instant and only at that instant: - every piece is alive, because the list still names them; - the result is being built, so a second full-size allocation is in flight; - nothing can be freed in between, because the combination needs all of its inputs until it finishes. This is why the failure lands where it does. The reads all succeed, the loop reports its last turn, and the process dies at the step after the loop — which is also why the reader usually gets the blame. It was the list, not the read. ## Designs do not all pay the same How bad the combination is depends on the model underneath: - On an **eager design that copies**, the result is a fresh allocation of everything, and you pay the doubling in full. - On a design that is **immutable by default and can share untouched buffers**, the result may reference the pieces' existing storage rather than copying it, so the doubling is partial or absent. In both cases the pieces themselves are resident, so keeping all of them is a bad plan on every design — but a claim like *the join doubles memory* is a claim about one family of designs, and it is worth saying which one you mean. ## The shape that actually bounds 1. Ask for a batch. 2. Reduce it to something much smaller than itself, or hand it to a writer that stays open across the turns. 3. Release it — let the last reference to it go — before asking for the next batch. 4. Carry across the turns only the small state from step 2. With that shape the peak is one batch plus that small state, which is the bound people believe they are buying when they write the loop in the first place. The rule of thumb is blunt: **if the loop keeps anything whose size grows with the number of turns, it is not a bounded pass**, it is a slow one-shot read with extra steps. ## When a list of pieces is perfectly fine The pattern is not wrong in itself. Collecting the turns is the right answer when what you collect is much smaller than what you read — each turn contributing a compact per-turn result — or when the columns and rows you refused at the read leave a result that genuinely fits. What fails is the unexamined version, where full-width, full-length batches are appended verbatim and the combination is the first moment anybody thinks about size. ## Telling which loop you wrote A quick audit, in order: - Look for a collection that is appended to inside the loop and consumed after it. That is the tell. - Ask what the largest thing alive at the last statement of the pass is. If the answer includes the input, the loop is decorative. - Watch occupancy across the run rather than at the end. A bounded pass is flat and a collecting pass is a staircase — and a staircase that never plateaus is the one that will fail on a slightly larger file next month. - Check that nothing outside the loop is quietly retaining the batches as well, such as a log line or a cache that keeps the last few turns for debugging.
- When is collecting the pieces in a list the right answer?When each turn contributes something much smaller than the batch it came from, or when the columns and rows refused at the read leave a result that genuinely fits. The pattern is fine; the unexamined version, appending full-width batches verbatim, is what fails.
- Why does this failure get blamed on the reader?Because the loop runs cleanly to its last turn and the process dies at the step afterwards, long after the final read. The read is the thing that was changed, so it attracts the blame, but the memory was spent by the list that was kept.
- What would you watch to confirm the diagnosis before changing any code?Occupancy across the whole run rather than at the end. A bounded pass is flat turn after turn; a collecting pass climbs in steps and then spikes once at the combination. The shape of that curve identifies which loop was written.
Bailing out a flooded basement works if each bucket goes out of the window. Lining the full buckets up along the wall and pouring them into a bathtub at the end means you need room for all the water twice over: once in the buckets, and again in the tub beside them.
saying these in an interview costs you the question
- Says a read in pieces caps memory at one piece, whatever the loop does.
- Blames the reader when the failure lands at the combination step.
- Thinks appending to a list is free because each piece was small.
- Assumes the combination releases the pieces before allocating the result.
- Believes every design pays the same doubling when the pieces are joined.