skip to content

A step whose finished result occupies 6 GB is killed on a 16 GB machine. What actually had to fit?

level: middleimportance: must knowfreq 58%

answer

  1. it dies at the worst instant
  2. input and output are alive together
  3. the design decides how much is copied
  4. eager chains add a buffer per operator
  5. sample during the step, not after

basics

~20 s

The peak had to fit, not the result. During the step the input is still referenced while the output and any intermediate are being allocated, so the worst instant can hold several copies of the data at once.

solid answer

~50 s

A run dies at its **peak footprint** - the worst instant inside a step - not at the steady size the data settles to afterwards. While a step runs, the input is usually still referenced by the caller and therefore still resident, the output is being allocated, and any intermediate the step needed is alive too. How much that adds depends on the design: if a step materialises fresh buffers for every column, the peak sits near the sum of input and output; if untouched column buffers are shared and only the rewritten columns are newly allocated, the peak grows by those columns alone; and where the expression is recorded and run as one fused traversal, the intermediate may never be built as an object at all. So measure the resident size *during* the step, and size the machine for that number.

go deeper

for a junior

Recall that the number that matters is the largest the process ever got, not the size of the answer. During a step the old data and the new data can both be in memory.

for a middle

Explain what is co-resident: input still referenced, output being allocated, intermediates alive. Then say that how much the step adds depends on whether untouched columns are copied or shared.

for a senior

Demonstrate measurement. Sample resident size while the step runs, attribute the jump to an operation, and name the lever - fewer columns touched, the input reference released, a shorter eager chain.

for a principal

The angle is headroom as a standing property: how close the peak runs to the machine, who notices when a new column moves it, and whether the pipeline is written so that the peak is attributable at all.

## Two numbers, and only one of them kills the run The **steady footprint** is what the data occupies while it sits between steps. The **peak footprint** is the worst instant inside a step, when more than one thing is resident at the same time. Every out-of-memory death is a statement about the peak, which is why a job that reports a comfortable 6 GB result can die on a machine with 16 GB: nobody was ever asked to fit 6 GB. The object being worked on here is a **labelled table** - a rectangle of columns where each column carries one type and the rows carry identity of their own - and the useful question about a step is not *how big is the answer* but *what is alive at once while the answer is built*. ## What is co-resident at the worst instant During a step that derives a new result from an existing table, the worst instant can hold all of: - **the input**, still referenced by whatever called the step, and therefore not reclaimable; - **the output**, partly or fully allocated; - **any intermediate** the step needed to get from one to the other; - **space the allocator cannot reuse yet**, because the input's bytes are still live while the output's are being requested. The last one is the trap people miss. Even where the input becomes garbage the instant the step finishes, that is one instant too late: the allocation that failed happened while both were live. ## The peak is a property of the design, not only of the step The same logical step costs very different peaks under different execution models, and this is where a remembered rule such as *a new object doubles memory* goes wrong: | Execution model | What the step allocates | Peak, roughly | |---|---|---| | Eager, every column materialised fresh | new buffers for all columns | input plus output | | Immutable columns, untouched buffers shared | new buffers only for rewritten columns | input plus the rewritten columns | | Recorded plan run as one fused traversal | one output buffer, no per-operator temporary | input plus output, no chain term | The same split governs chains. Under eager evaluation each operator returns a complete column before the next is applied, so a chain of four element-wise operations allocates four full-length buffers and the peak scales with the length of the chain. Where the steps are recorded and executed as a single traversal, the identical author-visible expression costs one output buffer and no per-operator temporary. Two candidates can give opposite answers about the same line of code and both be right about the tool they have used. ## Reading the peak instead of guessing it 1. Watch the process's resident size **across** the step, sampled while it runs, rather than reading a number after it has settled. 2. Attribute the peak to a step. A footprint that is flat and then jumps tells you which operation to look at; a single end-of-run number tells you nothing. 3. Re-check after any change to the column mix, because a step that touches one wide text column has a different peak from the same step on a narrow numeric one. Levers that actually move the peak, in rough order of effect: - **touch fewer columns per step**, since under a copying model the peak grows with what the step materialises, not with what the answer needs; - **stop referencing the input before the output is built**, where the step does not need it to finish - a reference held for convenience keeps the whole table resident through the allocation; - **avoid long eager chains** over full-length columns, because each link in the chain is another full-length buffer; - **work over pieces of the input**, which lowers the peak only when the computation can be answered from pieces at all, and only when whatever is carried between them does not itself grow with the data. ## Why it fit last time is not evidence A peak is a knife edge, and the things that push it over are unremarkable: a few percent more rows, one new column, a source whose text column arrived in a costlier representation this month, or a chain that grew by one operator during a review. A run that finished with nothing to spare and a run that had headroom look identical in the output. The number worth recording after a successful run is not that it finished but how close the peak came to the machine.

  • Why can dropping a reference to the input before the step runs lower the peak so much?
    Because the peak counts what is live at the same instant. While the caller still holds the input, its buffers cannot be reused for the output, so the allocator must find fresh room for both. Releasing the reference first lets the space be reclaimed and reused - provided the step genuinely does not need the input to produce its result.
  • Does a step that only adds one narrow column always have a small peak?
    Only where untouched column buffers are shared with the result. Under a model that materialises every column fresh for each new object, adding one narrow column still allocates the whole table again, so the peak is near double even though the logical change is tiny. Establish which model you are on before predicting the cost.

saying these in an interview costs you the question

  • Sizes the machine for the finished result rather than the worst instant
  • Assumes any step producing a new object doubles memory, whatever it touched
  • Thinks the input is freed as soon as the step begins
  • Reads one resident number after the run and calls it the footprint
  • Believes a long chain of element-wise operations is free under every design