Why does a YARN NodeManager kill a container for exceeding physical memory limits?

## What the container was promised When an ApplicationMaster requests a container it names a memory size in MB. The ResourceManager rounds that request **up** to a multiple of `yarn.scheduler.minimum-allocation-mb` (default 1024) and refuses anything above `yarn.scheduler.maximum-allocation-mb` (default 8192, and routinely raised on real clusters). The NodeManager then admits the container against what it advertises in `yarn.nodemanager.resource.memory-mb`. That advertised number is a fiction the whole cluster depends on: YARN packs containers onto a node assuming none will exceed its grant, so a container that does would push the machine into swapping or into a kernel OOM kill that could take down unrelated containers or the NodeManager itself. ## How enforcement works The NodeManager runs a containers-monitor thread that periodically walks each container's **process tree** and sums its resident set size. Two checks apply: - **Physical memory**, `yarn.nodemanager.pmem-check-enabled`, true by default. Exceeding the granted MB kills the container. - **Virtual memory**, `yarn.nodemanager.vmem-check-enabled`, true by default, with the allowance computed as granted memory times `yarn.nodemanager.vmem-pmem-ratio` (default **2.1**). The diagnostic looks like: `Container [pid=...] is running beyond physical memory limits. Current usage: 8.4 GB of 8 GB physical memory used; 12.1 GB of 16.8 GB virtual memory used. Killing container.` Read *which* limit the sentence names — physical and virtual overruns have different causes and different fixes. Where the LinuxContainerExecutor is configured with cgroups, CPU is enforced by the kernel as a genuine limit; memory enforcement in the common configuration remains this monitor-and-kill behaviour, which is why you see a killed container rather than a throttled one. ## Why the JVM heap is not the number that matters This is the heart of the question. A JVM launched with `-Xmx6g` inside a 6 GB container will be killed, because the process consumes far more than its heap: - **Metaspace and code cache** — hundreds of MB in a large application. - **Thread stacks** — roughly 1 MB per thread; a container running many concurrent tasks accumulates real memory here. - **Direct and off-heap buffers** — network buffers, memory-mapped shuffle files, off-heap execution memory. - **Native libraries** — compression codecs and native math libraries allocate outside the heap. - **Child processes** — PySpark launches Python workers whose memory belongs to the same process tree and counts against the same container. Spark models this explicitly: a YARN container for an executor is sized as `spark.executor.memory` (the heap) plus `spark.executor.memoryOverhead`, which defaults to 10% of executor memory with a **384 MiB** floor and is regularly too small for shuffle-heavy or Python workloads. PySpark additionally has `spark.executor.pyspark.memory`. The corresponding driver settings apply when the driver runs inside the ApplicationMaster container in cluster deploy mode. ## Diagnosing a real occurrence First, read the message and note the ratio between usage and limit — a container 5% over needs more overhead, one at 2x needs a design change. Second, decide whether the overrun is *steady* or *spiky*: steady overrun means the container is simply undersized; spikes at a particular stage point at a specific operation, usually a shuffle fetch, a large broadcast, a wide aggregation, or collecting too much data into the coordinator. Third, look at concurrency inside the container — Spark's `spark.executor.cores` multiplies the number of tasks sharing one container's memory, so halving cores per executor is often a more effective fix than adding memory. Fourth, check for skew: if exactly one container dies repeatedly while hundreds succeed, the memory setting is not the problem, the data distribution is. ## Fixes, in the order to try them 1. Raise the framework's overhead allowance so the container request matches real consumption. 2. Reduce per-container parallelism so fewer tasks share the same memory. 3. Address the workload — reduce partition sizes, avoid pulling large results into the coordinator, fix skew. 4. Only then raise total container memory, keeping in mind that fewer, larger containers pack worse onto nodes and are refused outright once they exceed `yarn.scheduler.maximum-allocation-mb`. ## The one thing not to do Disabling `yarn.nodemanager.pmem-check-enabled` is the tempting one-line fix and is almost always wrong: you have not made the container use less memory, you have removed the safety net that keeps one greedy container from destabilising the whole node. The defensible exception is the **virtual** memory check. Virtual memory overruns often reflect address-space reservations rather than real usage — glibc per-thread arenas are the classic cause — so many clusters legitimately turn `yarn.nodemanager.vmem-check-enabled` off while leaving the physical check on. Know the difference between those two decisions.