Pods on a Kubernetes node are being killed with the message "The node was low on resource: ephemeral-storage". What counts toward a container's ephemeral storage usage, and how would you stop this from recurring?

## What "ephemeral storage" means Every container gets a **writable layer** on top of its read-only image layers. Anything the process writes that is not on a mounted volume lands there and disappears when the container is deleted. Kubernetes groups three things under the resource name `ephemeral-storage`: 1. the container **writable layer**, 2. **`emptyDir` volumes** (disk-backed by default), 3. the pod's **container logs** written by the kubelet on the node filesystem. What does *not* count: PersistentVolumes, `configMap`/`secret`/`downwardAPI` volumes, `hostPath` mounts (they consume the host's disk but are not attributed to the pod), and read-only image layers, which are accounted to the image filesystem rather than to the pod. ## Two filesystems, two signals The kubelet tracks up to two filesystems: - **`nodefs`** — where the kubelet root lives: `emptyDir` volumes and pod logs. - **`imagefs`** — where the container runtime stores images and writable layers, when the runtime is configured with a separate filesystem. If there is no separate one, everything is `nodefs`. Each has an `available` and an `inodesFree` signal. Breaching either raises `DiskPressure=True`, which taints the node `node.kubernetes.io/disk-pressure:NoSchedule` so no new pods land there. Before evicting anything, the kubelet tries **garbage collection**: it removes dead containers and then unused images, driven by `imageGCHighThresholdPercent` (start collecting) and `imageGCLowThresholdPercent` (stop). Only when that fails to bring the signal back does it evict pods. Ranking works like memory: pods whose ephemeral-storage usage exceeds their `requests.ephemeral-storage` go first, then by pod priority, then by size of the overage. A pod with no ephemeral-storage request is over request by definition. ## The other kill: per-pod limits Separately from node pressure, if a pod declares `resources.limits.ephemeral-storage` and its combined usage across writable layers, `emptyDir`s, and logs exceeds it, the kubelet evicts **that pod alone**, even on a perfectly healthy node. Detection is periodic (the kubelet polls volume and filesystem stats, typically every ~10–15 seconds), so a burst can briefly exceed the limit before the eviction lands — this is not a hard cgroup ceiling like memory. An `emptyDir` with `sizeLimit` behaves similarly: exceed it and the pod is evicted. ## Diagnosing it 1. `kubectl describe pod` — the eviction message names `ephemeral-storage` and often the offending container and its usage versus request. 2. `kubectl describe node` — confirm `DiskPressure` and read the events (`EvictionThresholdMet`, `ImageGCFailed`). 3. On the node: `df -h` and `df -i` (inodes can run out with disk free), then find the consumer — `/var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir` for scratch space, `/var/log/pods` for logs, and the runtime's image directory. 4. `kubectl exec` plus `du -xh --max-depth=1 /` inside a suspect container to find files written into the writable layer. ## Fixing it durably **Declare it.** `requests.ephemeral-storage` makes the scheduler reserve disk the way it reserves memory, so the node stops being overcommitted on disk; `limits.ephemeral-storage` converts "one pod fills the node and everyone is evicted" into "the offending pod is evicted". That containment is the single biggest win. **Cap scratch space.** `emptyDir.sizeLimit: 1Gi` bounds a temp directory. Note `medium: Memory` emptyDirs are tmpfs and count against the pod's *memory*, not disk — a common surprise that turns a disk problem into an OOM. **Fix logging.** An app that logs to stdout at high volume fills `nodefs` through the kubelet's log files; set `containerLogMaxSize` and `containerLogMaxFiles` in kubelet config. An app that writes its own log file inside the container fills the writable layer invisibly — redirect it to stdout or to a sized volume. Ship logs off-node so retention is not the node's problem. **Manage images.** Nodes that pull many large images churn `imagefs`; tune GC thresholds, provision a bigger disk, or give the runtime a dedicated filesystem so image churn cannot evict workloads through `nodefs`. **Use real storage.** If a workload genuinely needs tens of gigabytes of working data, that is a PersistentVolumeClaim (or a generic ephemeral volume, which gives per-pod dynamically provisioned storage that is deleted with the pod), not the node's root disk. ## Interview framing Define the three things that count, separate the node-level signal from the per-pod limit, describe the diagnosis path (`describe pod` → `describe node` → `df`/`df -i` on the node), and land on containment: requests/limits, emptyDir sizeLimit, log rotation, image GC, and PVCs for real data.