skip to content

Where does the kubelet keep a Kubernetes container's log files on a node, and how does its containerLogMaxSize rotation change what kubectl logs returns?

level: middleimportance: should knowfreq 46%

answer

  1. runtime writes, kubelet rotates
  2. pods dir keyed by pod UID
  3. containers dir holds symlinks
  4. 10Mi and 5 files by default
  5. kubectl reads the current file only

basics

~10 s

The runtime writes each container's output to /var/log/pods/<namespace><pod><uid>/<container>/<restart>.log, symlinked from /var/log/containers. The kubelet rotates it at containerLogMaxSize (10Mi), keeps containerLogMaxFiles (5), and kubectl logs reads only the current file.

solid answer

~40 s

The container runtime writes stdout and stderr, in the CRI log format, to `/var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container>/<restart-count>.log`, and the kubelet keeps a symlink per container in `/var/log/containers/` for node log collectors. The kubelet rotates these files itself: every `containerLogMonitorInterval` (10s) it checks the size, and past `containerLogMaxSize` (default `10Mi`) it renames the file with a timestamp, gzips older rotated files, deletes beyond `containerLogMaxFiles` (default `5`) and asks the runtime to reopen. The trap is that `kubectl logs` reads only the **current** file, so a chatty container shows just its last few megabytes even though more history sits compressed on the node. Raising the limits costs node disk; durable history belongs in a log pipeline.

code

yaml · 6 lines
yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxSize: 50Mi
containerLogMaxFiles: 4
containerLogMonitorInterval: 5s
containerLogMaxWorkers: 2

go deeper

for a junior

Recall the two directories, /var/log/pods for the real files and /var/log/containers for symlinks, and that kubectl logs shows only recent output.

for a middle

Explain the rotation steps, the 10Mi and 5-file defaults, the 10-second check interval, and why kubectl logs cannot reach rotated files.

for a senior

Size the defaults against a noisy workload, predict how much history kubectl logs will show, and weigh bigger limits against node disk pressure.

for a principal

Treat node log rotation as disk protection only, and make an off-node pipeline the platform's answer for history rather than per-node tuning.

## Who writes the files Three parties touch a container's log on a Kubernetes node: - the **container** writes to **stdout** and **stderr** and knows nothing about files; - the **container runtime** (containerd or CRI-O, behind the **CRI**) captures both streams and appends them to a file at a path the kubelet chose when it created the container; - the **kubelet** owns the directory layout, the rotation and the cleanup, and serves the file back when `kubectl logs` asks. Each line is written in the **CRI log format**: a timestamp, the stream name (`stdout` or `stderr`), a tag (`F` for a full line, `P` for a partial line that continues in the next record) and the content. `kubectl logs` strips that prefix; `kubectl logs --timestamps` puts the timestamp back. ## The directory layout | Path | What it is | |---|---| | `/var/log/pods/<namespace>_<pod-name>_<pod-uid>/` | one directory per pod, keyed by UID so a recreated pod with the same name never collides | | `.../<container-name>/<restart-count>.log` | the current file for one container instance: `0.log`, then `1.log` after a restart | | `.../<container-name>/<restart-count>.log.<timestamp>` | the newest rotated file, left uncompressed | | `.../<container-name>/<restart-count>.log.<timestamp>.gz` | older rotated files, gzip-compressed | | `/var/log/containers/<pod-name>_<namespace>_<container-name>-<container-id>.log` | a **symlink** to the current file, a flat layout kept for node-level log collectors | The pod log root is configurable through the kubelet's `podLogsDir` setting, so check the kubelet configuration before assuming the default path. ## How rotation works Rotation is done by the kubelet, not by the runtime and not by the node's logrotate. Its settings live in the **KubeletConfiguration**: - `containerLogMaxSize` — size that triggers rotation, default `10Mi`; - `containerLogMaxFiles` — maximum files per container, current one included, default `5`; - `containerLogMonitorInterval` — how often sizes are checked, default `10s`; - `containerLogMaxWorkers` — how many containers are rotated in parallel, default `1`. On each check, for every **running** container whose current file exceeds the limit, the kubelet: 1. removes leftovers of an earlier rotation that failed half-way; 2. deletes the oldest rotated files so that at most `containerLogMaxFiles - 2` remain; 3. gzips any rotated file that is still uncompressed; 4. renames the current file to `<restart-count>.log.<timestamp>`; 5. calls the runtime's `ReopenContainerLog` so the runtime starts a fresh current file. After that, a container holds at most five files by default: the current one, one uncompressed rotated file and up to three compressed ones. Because the size is checked on an interval, a very chatty container can run past `10Mi` by whatever it writes between checks. ## What `kubectl logs` actually shows The kubelet serves `kubectl logs` from the container's **current** log file only. Rotated files stay on the node but are not stitched in. Two consequences: - right after a rotation, `kubectl logs` can return only a few lines, even for a container that has run for days; - `--since`, `--tail` and `--limit-bytes` filter within the current file; none of them reaches older files. `kubectl logs -f` keeps streaming across a rotation: the kubelet's log reader watches the directory and switches to the recreated file. It still never goes back to read the rotated one. ## A worked example A video-transcoding worker on a single-node development cluster on a laptop logs encoder progress at about **2.3 MiB per minute**. With the defaults: - a `10Mi` file fills in about 10 / 2.3 ≈ **4.3 minutes**, so `kubectl logs` never shows more than roughly the last four minutes of output; - five files hold about 5 × 4.3 ≈ **22 minutes** of output on disk, most of it in `.gz` form; - a failure the developer noticed ten minutes late is already out of `kubectl logs`, but still on the node in a rotated file, readable with `zcat` from a shell on the node. ## Tuning, and what it costs Raising `containerLogMaxSize` or `containerLogMaxFiles` keeps more on the node, but every container on the node gets the new budget, and log files count against the node's disk. On a busy node that pushes toward disk pressure, which is its own failure mode. Two better levers: - make the application less noisy — progress output at debug level, not on every frame; - ship logs off the node with a collector that tails `/var/log/containers/`, so rotation only has to protect the disk. On a kind or kubeadm-built cluster the settings go in the kubelet's configuration file, and the kubelet must be restarted to pick them up.

  • Why does `/var/log/containers/` exist if the real files are under `/var/log/pods/`?
    It is a flat directory of symlinks, one per container, whose file name encodes the pod name, namespace, container name and container ID. Node-level log collectors tail that one directory and parse the metadata from the name, without walking the nested per-pod tree. The kubelet creates the symlink when it starts a container and removes it when the container is removed.
  • A container writes 40 MiB in the ten seconds between two kubelet size checks. What happens?
    The current file grows to far more than `containerLogMaxSize` before the next check notices it, because the limit is enforced on the `containerLogMonitorInterval`, not on every write. At that check the kubelet rotates the oversized file as usual. Lowering the interval or raising `containerLogMaxWorkers` helps when many containers are noisy; the real fix is less output.
  • Do Docker's `max-size` and `max-file` log options control these files?
    No. Those are options of Docker's own log driver, a separate layer. On a Kubernetes node the runtime behind the CRI writes the files at the kubelet's chosen path, and rotation is driven by the kubelet's `containerLogMaxSize` and `containerLogMaxFiles`. Set rotation in the KubeletConfiguration.

saying these in an interview costs you the question

  • kubectl logs stitches together every rotated file for the container.
  • The node's logrotate service rotates Kubernetes container logs.
  • The files under /var/log/containers are the real logs; /var/log/pods holds copies.
  • containerLogMaxSize is enforced on every write, so a file never exceeds it.
  • Docker's max-size log option sets rotation for pods on a CRI node.
  • Raising containerLogMaxFiles is a free way to keep history for incidents.