A Docker host runs out of disk and the culprit is a multi-gigabyte JSON log file under /var/lib/docker/containers. Explain why that happened and how you would configure the daemon so it cannot happen again.

seniorshould knowfreq 50%

answer

json-file default = no rotation
max-size + max-file (+ compress)
daemon.json, then recreate containers
local driver rotates by default
never rm a live log file, truncate it

basics

~20 s

Docker's default json-file driver writes container output to an unbounded file on the host with no rotation. Set max-size and max-file on the driver — per container or as a daemon-wide default in /etc/docker/daemon.json — then restart the daemon and recreate containers.

solid answer

~50 s

The json-file driver stores every captured stdout/stderr line as a JSON record in /var/lib/docker/containers/<id>/<id>-json.log. By default there is no rotation, so a chatty or crash-looping container grows that file until the filesystem fills — which typically takes the daemon and every container on the host down with it. JSON framing also adds overhead, so the on-disk size exceeds the raw log volume.

The fix is rotation options on the driver:

json

{ "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3", "compress": "true" } }

in /etc/docker/daemon.json, then systemctl restart docker. It applies to newly created containers only, so existing ones must be recreated; per-container --log-opt overrides it. Bounding it caps each container at max-size × max-file.

Do not truncate the live file by deleting it — the daemon holds the descriptor. Use docker rm or truncate in place, and consider the local driver, which rotates by default.

The `json-file` driver stores every captured stdout/stderr line as a JSON record in `/var/lib/docker/containers/<id>/<id>-json.log`. By default there is **no rotation**, so a chatty or crash-looping container grows that file until the filesystem fills — which typically takes the daemon and every container on the host down with it. JSON framing also adds overhead, so the on-disk size exceeds the raw log volume. The fix is rotation options on the driver: ```json { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3", "compress": "true" } } ``` in `/etc/docker/daemon.json`, then `systemctl restart docker`. It applies to newly created containers only, so existing ones must be recreated; per-container `--log-opt` overrides it. Bounding it caps each container at max-size × max-file. Do not truncate the live file by deleting it — the daemon holds the descriptor. Use `docker rm` or truncate in place, and consider the `local` driver, which rotates by default.

code

bash · 9 lines

bash

cat /etc/docker/daemon.json
# {
#   "log-driver": "json-file",
#   "log-opts": { "max-size": "10m", "max-file": "3", "compress": "true" }
# }

sudo systemctl restart docker
docker inspect -f '{{json .HostConfig.LogConfig}}' web
du -sh /var/lib/docker/containers/* | sort -h | tail

go deeper

for a junior

Know that json-file has no rotation by default and that max-size and max-file switch it on.

for a middle

Add where the setting lives (daemon.json or per container) and that it only applies to newly created containers.

for a senior

Handle the live incident correctly, size worst-case disk as max-size x max-file x containers, and mention the local driver.

for a principal

Set a fleet policy: bounded local rotation as a safety net plus off-host shipping, and treat log volume itself as a cost to manage.

Why the default fills disks

When a container runs, the daemon captures the main process's stdout and stderr and writes them through the logging driver. The default driver, json-file, appends one JSON object per line to /var/lib/docker/containers/<container-id>/<container-id>-json.log. Out of the box that file has no size limit and no rotation. A container that logs a stack trace per request, or crash-loops printing startup errors, will happily write tens of gigabytes.

Because /var/lib/docker usually shares a filesystem with the image store and container writable layers, filling it is not a contained failure: image pulls fail, containers cannot write, and the daemon itself may become unhealthy. The JSON envelope (timestamp, stream name, escaping) also inflates on-disk size well beyond the raw log bytes.

Configuring rotation

The driver takes options:

max-size — rotate when the current file reaches this size (e.g. 10m).
max-file — how many rotated files to keep; older ones are deleted.
compress — gzip rotated files.

Set them daemon-wide in /etc/docker/daemon.json under log-opts and restart the daemon, or per container with --log-opt max-size=10m --log-opt max-file=3 (Compose: logging.options). Worst-case disk per container is max-size × max-file, so multiply by the expected container count when sizing the host.

Crucially, log configuration is applied at container creation. Changing daemon.json does not retrofit running containers; they must be recreated. Verify with docker inspect -f '{{json .HostConfig.LogConfig}}' <c>.

The `local` driver

Docker's local driver stores a compact binary format, rotates by default (100 MB × 5 files at the time of writing) and is cheaper to write and read. docker logs works with it. It is a better default than json-file when nothing downstream parses the raw json-file format directly.

Incident handling

When the disk is already full, do not rm the live log file: the daemon holds an open descriptor, so space is not returned until the container stops, and you get a confusing 'disk still full' state. Safer options are truncating in place (: > /var/lib/docker/containers/<id>/<id>-json.log) or removing and recreating the container. docker system df -v and du -sh /var/lib/docker/containers/* identify the offender quickly. Then fix both sides: enable rotation, and reduce the application's log volume, because rotation caps disk but does nothing about the noise or the cost of shipping it.

Beyond one host

Rotation is a safety net, not a log strategy. Rotated logs are deleted, so anything not shipped elsewhere is gone, and rotation deletes exactly when volume spikes — usually during the incident you want to investigate. On a fleet, pair bounded local rotation with a collector that ships logs off-host, so the local file is only a buffer. Note also that log volume interacts with the driver's delivery mode: with a blocking remote driver, a slow log backend can stall the application's writes, which is why bounded local storage plus an out-of-band collector is the common production shape.

## Why the default fills disks When a container runs, the daemon captures the main process's stdout and stderr and writes them through the logging driver. The default driver, `json-file`, appends one JSON object per line to `/var/lib/docker/containers/<container-id>/<container-id>-json.log`. Out of the box that file has **no size limit and no rotation**. A container that logs a stack trace per request, or crash-loops printing startup errors, will happily write tens of gigabytes. Because `/var/lib/docker` usually shares a filesystem with the image store and container writable layers, filling it is not a contained failure: image pulls fail, containers cannot write, and the daemon itself may become unhealthy. The JSON envelope (timestamp, stream name, escaping) also inflates on-disk size well beyond the raw log bytes. ## Configuring rotation The driver takes options: - `max-size` — rotate when the current file reaches this size (e.g. `10m`). - `max-file` — how many rotated files to keep; older ones are deleted. - `compress` — gzip rotated files. Set them daemon-wide in `/etc/docker/daemon.json` under `log-opts` and restart the daemon, or per container with `--log-opt max-size=10m --log-opt max-file=3` (Compose: `logging.options`). Worst-case disk per container is `max-size × max-file`, so multiply by the expected container count when sizing the host. Crucially, **log configuration is applied at container creation**. Changing daemon.json does not retrofit running containers; they must be recreated. Verify with `docker inspect -f '{{json .HostConfig.LogConfig}}' <c>`. ## The `local` driver Docker's `local` driver stores a compact binary format, rotates by default (100 MB × 5 files at the time of writing) and is cheaper to write and read. `docker logs` works with it. It is a better default than json-file when nothing downstream parses the raw json-file format directly. ## Incident handling When the disk is already full, do not `rm` the live log file: the daemon holds an open descriptor, so space is not returned until the container stops, and you get a confusing 'disk still full' state. Safer options are truncating in place (`: > /var/lib/docker/containers/<id>/<id>-json.log`) or removing and recreating the container. `docker system df -v` and `du -sh /var/lib/docker/containers/*` identify the offender quickly. Then fix both sides: enable rotation, and reduce the application's log volume, because rotation caps disk but does nothing about the noise or the cost of shipping it. ## Beyond one host Rotation is a safety net, not a log strategy. Rotated logs are deleted, so anything not shipped elsewhere is gone, and rotation deletes exactly when volume spikes — usually during the incident you want to investigate. On a fleet, pair bounded local rotation with a collector that ships logs off-host, so the local file is only a buffer. Note also that log volume interacts with the driver's delivery mode: with a blocking remote driver, a slow log backend can stall the application's writes, which is why bounded local storage plus an out-of-band collector is the common production shape.

You added log-opts to daemon.json and restarted Docker, but one container's log file keeps growing. Why?
Logging configuration is fixed when a container is created, so the daemon default only applies to containers created afterwards. The existing container keeps its original unbounded config, which you can confirm with docker inspect on its HostConfig.LogConfig. Recreate it to pick up the new default.
Is rotation enough, or do you still need a log collector?
Rotation only bounds disk; it deletes older logs, and it deletes fastest exactly when volume spikes during an incident. Ship logs off-host with a collector so the local file is just a buffer, and keep local rotation as the safety net that protects the node if the collector is down.

saying these in an interview costs you the question

Assuming Docker rotates container logs by default
Deleting the live log file to reclaim space while the container runs
Expecting a daemon.json change to affect already-created containers
Setting max-size without multiplying by max-file and container count when sizing disk
Treating rotation as a substitute for shipping logs off the host

A Docker host runs out of disk and the culprit is a multi-gigabyte JSON log file under /var/lib/docker/containers. Explain why that happened and how you would configure the daemon so it cannot happen again.

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~2 min

Why the default fills disks

Configuring rotation

The local driver

Incident handling

Beyond one host

▾common follow-ups2 questions

▾red flags5 flags

§ context

▾related questions6 questions

▾related topics3 topics

Why the default fills disks

Configuring rotation

The local driver

Incident handling

Beyond one host

answer

go deeper

The `local` driver

context

The `local` driver