A unit of work that overflowed to local disk reports 9 GB spilled in memory but 1.1 GB on disk — why both?

## Two numbers, one event When an operator cannot hold its working set and writes part of it out to disk attached to the worker, a run's reported numbers frequently show that single event twice, under two names and two very different sizes. Neither is wrong and neither is a duplicate count. - **The in-memory figure** is how much of the operator's borrowed space those records were occupying at the moment they were evicted from it. - **The on-disk figure** is how many bytes were actually written to local scratch disk. The records are the same records. What differs is the form they are in. ## Why the forms differ so much A record in an operator's working memory is held in one of two ways: - **As host-runtime objects** — ordinary objects of whatever language the worker runs, each carrying the runtime's own per-object bookkeeping, references between them, and padding. A small logical record can occupy several times its logical size this way. - **As engine-managed binary records** — a packed byte layout the engine allocates and interprets itself, where a field is read at a known offset instead of by following a pointer. This is much closer to the logical size, though still not equal to it. What is written to disk is neither: it is a compact encoded form, often with a compression step on top, chosen because disk traffic is the thing being minimised. Column-shaped or dictionary-friendly data can shrink dramatically; opaque binary payloads barely shrink at all. So the ratio is a property of **the data and the engine's representation**, not a bug. It is often several times, and much smaller on engines that already hold their records as packed bytes. | Figure | What it measures | Chiefly tells you | |---|---|---| | In-memory size | Space the records held in the operator's borrowed memory | How much more operator working memory would have avoided the spill | | Bytes on disk | Bytes written to local scratch disk | The disk traffic and space cost you actually paid | | Their ratio | The encoding and compression multiple for this data | Whether the records are representation-heavy or already compact | ## Which one answers which question This is where candidates lose the point. The instinct is to take the smaller number, because it looks concrete, and to conclude that a little more memory would have avoided the whole thing. That reasoning is backwards: to avoid spilling, the records would have had to stay **in the form they had in memory**, which is the larger number. Sizing memory from the on-disk figure undersizes it by exactly the encoding multiple, and produces a job that still spills after the change. Conversely, when the question is cost — the time this step spent writing and reading, and whether the local device has space — the on-disk figure is the honest one. Charging the in-memory size against disk would overstate it just as badly. A third reading comes from the pair together. An unusually large ratio says the records are expensive to hold relative to what they really contain, which points at representation rather than at volume; an unusually small ratio says the data is already dense and there is no encoding win left to find. ## What varies between engines - Some engines report both figures per unit of work; some report only one; some aggregate to the step and leave you unable to see whether one unit of work produced nearly all of it. - Engines that hold packed bytes show a smaller gap than engines that hold ordinary language objects, so a ratio that is remarkable on one platform is unremarkable on another. - Where the numbers are aggregated across a step, a large total can mean either every unit of work spilled a little or one spilled enormously — two situations with different remedies, which is why the per-unit view matters more than the total. - Some engines count the final merging pass's reads in the same counter, and some do not, so comparing the ratio across platforms is not meaningful. ## How to use it in practice 1. Read the **per unit of work** figures, not just the step total, and look at the spread across units. 2. Take the in-memory figure as the memory question and the disk figure as the cost question. 3. Before asking for more memory everywhere, check whether more, smaller pieces would bring the in-memory figure under the space each unit of work can borrow — the same total memory, divided differently. 4. Treat a large ratio as a hint about how records are held, which is its own subject, rather than as an anomaly in the spill itself. The short version for an interview: the two numbers are the same records in two forms; size memory from the memory one and cost from the disk one.