Resource Pressure & Eviction

## Two shortages, not one Every node carries two memory constraints that are enforced separately, and confusing them is why this symptom looks impossible. The first is the **ceiling** declared for a container in its workload spec. The runtime enforces it through the kernel's resource-accounting mechanism, against that one container's own usage. Cross it and a process inside that container is killed — even if the machine has gigabytes free. The second is the **machine's physical memory**, shared by every container on it plus the node agent, the runtime itself and whatever else the operating system is holding. Exhausting that is not a breach of anyone's declared figure. It is an arithmetic outcome: the workloads placed there were allowed to use more, in total, than the node had. | | Ceiling breach | Host exhaustion | |---|---|---| | Scope of the accounting | one container | the whole machine | | What triggers it | that container's usage crossing its declared figure | total free memory on the node approaching zero | | Who is stopped | a process inside that container | a process chosen at host scope, in any container | | Relation to fault | the victim is the cause | the victim need not be the cause | | Node's free memory at the time | may be mostly free | by definition nearly gone | ## How the victim is chosen at host scope The kernel's out-of-memory killer is not looking for a culprit. It is looking for **relief**, and it scores candidates mainly by how much memory stopping each one would return. Platforms then bias that score using what the workload declared — a workload sitting inside its reservation is made a less attractive victim, a workload that declared nothing a more attractive one — and platforms differ in how strongly they apply that bias. The consequences are counter-intuitive in exactly the way interviews probe: - The **largest resident set wins the lottery**, not the fastest grower. A report renderer holding a big working set entirely within its ceiling is cheap, effective relief. - The workload that actually consumed the node may be **small at the instant of the decision** — it allocated in a burst, the node hit zero, and the scoring saw a modest process. - **Nothing about your compliance is consulted directly.** The ceiling you honoured describes the boundary you were told not to cross; it says nothing about what happens when someone else exhausts the shared pool. - The kill is **abrupt**. There is no notice, no grace, and no negotiation with the process. ## The orderly path the platform would rather take The kernel's kill is the last resort, and a healthy platform tries to act before it. The node agent watches free memory and, at a line drawn **above** the point at which the kernel would act, starts reclaiming and evicting by rank: 1. Free memory falls past the agent's line. The agent reclaims what it can and, if that is not enough, stops workloads in a deliberate order, recording a reason. 2. If free memory keeps falling faster than reclaim can work, the line is overrun. 3. The kernel acts at host scope, by footprint, with no ranking and no record beyond the fact of the kill. So the *presence* of a host-scope kill is itself evidence: the orderly path did not get there in time. ## Reading the evidence at 02:00 - **Several unrelated workloads stopping within the same few seconds** points at the node, not at any one of them. - **One container dying repeatedly at a reproducible usage figure, on a node with memory to spare**, points at its own ceiling. - **A sharp slowdown with no stop at all** is not this at all — that is a CPU ceiling, a different resource with the opposite consequence. - **An abrupt stop with no recorded reason** points at the kernel; **a recorded rank and reason** points at the platform's own eviction. ## Why memory behaves differently from CPU CPU is compressible. Demand above a CPU ceiling is throttled: the process runs slower and stays alive, and when demand falls it recovers on its own. Memory a running process already holds cannot be taken back by asking. There is no throttle for it, so the only lever left is to stop something — which is why every path through a memory shortage ends in a kill or an eviction, and why the interesting question is only ever *which one*. ## What this actually changes for your workload - A declared ceiling protects the **node from your workload**, not your workload from the node. - The lever that affects your odds under host-scope pressure is the **reservation** you declared, because that is what the platform's ranking and victim-score bias are drawn against. - Sizing the working set close to what was declared, rather than leaving a declared figure that no longer reflects reality, is the practical form of that. - A kill with no recorded reason is worth escalating as a **node** problem, not a workload bug — treating it as a leak in your own service sends the investigation to the wrong place.