In the Spark UI, one task reports Spill (Memory) of 48 GB on an executor with an 8 GB heap — what is that measuring?

## Two numbers, two units The stage detail page in the Spark UI shows, per task and aggregated per stage: - **Spill (Memory)** — the size the spilled records occupied *in memory*, deserialized, at the instant of each spill, summed over all spills by that task. - **Spill (Disk)** — the bytes actually written to the executor's local disk, after serialization and compression. They measure the same records through different lenses, which is why the memory number is routinely several times the disk number. That ratio is your serialization plus compression factor and is not, by itself, a problem. ## Why the memory number can exceed the heap This is the part that trips people up. Neither number is a peak. Each is a running total. A task whose sort buffer grows to 2 GB, spills, grows again, spills again, and repeats twenty-four times has never held more than 2 GB at once — but reports 48 GB of memory spill. Seeing a spill figure larger than `spark.executor.memory` tells you the task spilled *many times*, which is itself the useful signal: many small spill files mean an expensive merge phase afterwards. ## Who spills, and when Execution memory is requested incrementally by spillable operators. When a task asks the executor's execution memory pool for more space and cannot get it, the operator flushes its current contents to local disk and starts fresh. The main spillers are: - `ExternalSorter` — sort-based shuffle writes and sorts. - `ExternalAppendOnlyMap` — map-side combining and aggregations. - The Tungsten sort/aggregate operators, which spill their binary pages the same way. - Sort-merge join, which sorts both sides before merging. All of them are designed to spill. Correctness is never at risk; the cost is extra IO and a merge pass over the spill files. ## Why a task cannot use the whole pool The executor's execution memory is shared dynamically among its concurrently running tasks. A task is guaranteed at least `1/(2N)` of the pool and may acquire at most `1/N`, where N is the number of active tasks — that is, `spark.executor.cores` when the executor is saturated. So an 8-core executor with a 6 GB unified pool gives one task at most roughly 750 MB of execution memory, no matter how much heap the machine has. Doubling executor cores to get more parallelism halves each task's memory budget, and spill is usually the first symptom. The second squeeze is caching. Cached blocks occupy the storage side of the same unified pool. Once storage holds its protected floor (`spark.memory.storageFraction`), execution cannot evict any further, so a heavily cached job leaves execution a smaller pool to divide. ## Reading the evidence Open the stage's task table and look at the distribution, not the sum: - **Every task spills a similar amount** → the stage's per-partition data volume simply exceeds the per-task memory budget. Raise the partition count so each partition is smaller, or reduce concurrency per executor. - **One or two tasks spill enormously and run long while the rest finish fast** → skew. The fix is in the data distribution, not the memory settings. - **Spill appears only after a `cache()` was added upstream** → storage is crowding execution. Cache less, use a disk-backed or serialized level, or lower `spark.memory.storageFraction` so execution may evict more. ## What to change, in order 1. **More partitions.** For SQL shuffles this is `spark.sql.shuffle.partitions`; for RDD code it is the `numPartitions` argument or an explicit repartition. Smaller partitions mean smaller per-task working sets, and this is the lever that works most often. 2. **Fewer cores per executor.** Fewer concurrent tasks means a larger `1/N` slice each. Throughput per executor may drop slightly while total runtime improves. 3. **Less cached data**, or a cheaper storage level, to free the pool. 4. **A bigger heap**, last — it is the expensive lever and does nothing about skew. ## What spill is not Spill is not an out-of-memory error, and "zero spill" is not a tuning goal. A job that spills a modest amount and finishes on time needs no attention. Spill is also not the same as *shuffle write*: shuffle write bytes are the normal output of a map stage, written to disk by design for every shuffle. Confusing the two leads people to "fix" healthy shuffles. Finally, spilling in Spark is unrelated to a garbage-collection problem — high GC time is a separate metric in the same task table, and the two have different remedies.