A file larger than memory is read in pieces on one machine: what is one piece, and where may the cuts fall?

## The read becomes a loop A read call that returns the entire file as one object is the default everywhere, and it is exactly the thing that fails when the file is larger than the room you have. The alternative is to turn that single call into a loop. You tell the reader how many records you want per turn, and each turn it hands back a small **labelled table** — a rectangle whose columns each carry one type and whose rows may carry an identity of their own — built from the next stretch of the file. You do something with that batch, let it go, and ask for the next one. The file is walked once, front to back, and at no moment is all of it resident. Everything worth knowing about the mechanism sits in three questions: where a turn is allowed to cut, what has to be carried from one turn to the next, and what the loop body does with each batch so that the loop bounds anything at all. ## A byte offset is not a cut The tempting move is to divide by size: take the file's length in bytes, split it into equal ranges, hand each range to the reader. For a text shape this does not work, because records there are variable length. A position two fifths of the way through the bytes lands wherever it lands, and that is almost always inside a record. The range before it ends with half a record; the range after it begins with the other half. Both halves are damaged, and what becomes of them depends entirely on the reader's disposition for a line that does not fit the expected shape: it may stop, it may drop the line, or it may hand back a row with its fields shifted along by one. A legal cut is a record boundary. What marks the end of a record is decided by the format's own rules, and the reader is the component that knows them — which is the real reason you ask for a count of records per turn rather than a count of bytes. The reader finds the boundaries; you never compute one. Some readers also let you bound a turn by input bytes, and where they do they still finish the record they are inside, so the bound is a target rather than a cut point. ## What has to ride across the turns A loop has state, and a read loop carries more of it than people expect: - **The names of the columns.** In a text shape they usually live in the first line. When the reader owns the loop it reads that line once and applies the names to every turn. When you slice the file by hand, only your first block contains it, and every later block starts with an ordinary record that a reader will happily promote to column names. - **The types.** A reader that has to guess will guess again for every turn unless you hand it one declaration to use for all of them. Two turns over the same file can then disagree about the same column. - **Whatever your loop body is accumulating.** This is the only state that legitimately grows across the pass, and it is the state to keep small. ## The two shapes do not pose the same problem | | re-parsed delimited text | a self-describing binary columnar file | |---|---|---| | where a cut is legal | at a record boundary the reader locates by scanning the bytes | at batches the file's own declaration already locates, so nothing is hunted for | | what the first turn carries | the line naming the columns, when the file has one | nothing extra: names and types are written into the file | | does a turn re-decide types | yes, unless one declaration is handed to every turn | no, the declaration is read out of the file | A pass in pieces over a text shape is a loop with real bookkeeping. The same pass over a self-describing shape is mostly just a loop. ## What the loop owes each piece 1. Do something with the batch that is smaller than the batch — reduce it to a partial result, or hand it to a writer that stays open across turns. 2. Let it go before asking for the next one. The bound you are buying is one batch at a time, and it only exists if the previous batch is no longer referenced. 3. Carry only small state across turns. A loop that keeps every batch and combines them at the end has bounded nothing, and it fails later and more confusingly than the one-shot read it replaced. ## When there is no loop to write Say what the read call returns before you reach for a loop at all. Where it returns the data itself, a file larger than memory needs an explicit pass in pieces and the rules above are yours to obey. Where it returns a description of the read that is executed only when you ask for an answer — a deferred design, in which a plan is built first and run later — the columns and rows you never asked for are dropped inside the read and the iteration belongs to the engine. The costs do not vanish; they stop being your loop's to get wrong.