skip to content

Docker's default storage driver on Linux is overlay2, which builds a container's root filesystem with an overlay mount. Explain what the lowerdir, upperdir, workdir and merged directories are and how they combine.

level: middleimportance: must knowfreq 48%

answer

  1. lowerdir = read-only image layers, leftmost wins
  2. upperdir = container writable layer
  3. workdir = kernel scratch, same fs as upper
  4. merged = what the container sees as /
  5. files shadow, directories merge

basics

~20 s

lowerdir is the stack of read-only image layers, upperdir is the container's writable layer, merged is the unified view the container uses as /, and workdir is a private scratch directory the kernel needs for atomic operations. Upper entries shadow lower ones.

solid answer

~50 s

overlay2 uses the kernel's overlayfs, mounted with four directories: - **lowerdir** — one or more read-only directories, the image layers, listed highest-priority first and colon-separated. - **upperdir** — the container's single writable layer; all changes land here. - **workdir** — an empty directory on the same filesystem as upperdir, used internally by the kernel to stage operations such as copy-up so they appear atomic. Never touched by you. - **merged** — the mount point presenting the union; this is what the container sees as `/`. Lookup rules: a path found in upperdir wins; otherwise the lowerdirs are searched in order. Directories are *merged* (entries from all levels are visible together), while files are *shadowed* (only the topmost wins). Writes go to upperdir. Modifying a file from a lower layer copies it up first. Deleting one records a whiteout in upperdir.

code

bash · 5 lines
bash
docker inspect --format '{{json .GraphDriver.Data}}' mycontainer

# equivalent mount, conceptually:
# mount -t overlay overlay \
#   -o lowerdir=/l2:/l1,upperdir=/upper,workdir=/work /merged

go deeper

for a junior

Name the four directories and say that the container sees merged while writing only into upperdir.

for a middle

Add the lookup order and the file-shadows/directory-merges distinction, and show GraphDriver.Data on a real container.

for a senior

Connect the layout to shared read-only layers across containers, copy-up and whiteout behaviour, and backing-filesystem requirements such as XFS ftype=1.

for a principal

Discuss the consequences for host density and image design — shared lower layers keep footprint proportional to writes, long layer chains stress mount options — and where mounts should bypass the driver entirely.

## What overlayfs is overlayfs is a Linux kernel union filesystem: it takes several existing directories and presents them as one tree without copying their contents. Docker's `overlay2` storage driver is a thin manager on top of it, deciding which directories play which role for each image layer and container. ## The four directories An overlay mount takes these options: - **lowerdir** — a colon-separated list of read-only directories, ordered leftmost = highest priority. Docker maps the image's layers here, topmost image layer first. - **upperdir** — one writable directory. Every change a container makes lands here. - **workdir** — an empty directory on the *same* filesystem as upperdir, reserved for the kernel. It stages intermediate states so operations like copy-up appear atomic to the container. It must exist and must not be shared. - The mount point itself, which Docker calls **merged**, is the unified view the container uses as `/`. On a typical host you find these under `/var/lib/docker/overlay2/<id>/` with subdirectories `diff` (this layer's own content), `link` and `lower` (bookkeeping for the layer stack), plus `merged` and `work` for the container's own directory. ## Lookup semantics When the container opens a path, overlayfs resolves it top-down: 1. If the path exists in upperdir, that wins. 2. Otherwise it walks the lowerdirs in order and takes the first match. 3. If a whiteout marker for the name exists in upperdir, the name is reported as absent even though lower layers contain it. The crucial distinction is **files versus directories**. A file is *shadowed*: only the topmost instance is visible. A directory is *merged*: its listing is the union of the entries from every level where it exists, unless it is marked opaque. So `/etc` can have some files from the base image, some from a later layer and some written by the container, all visible at once, while `/etc/hosts` shows exactly one version — the highest. ## Writes - **New file** — created directly in upperdir. - **Modify an existing file from a lower layer** — the kernel performs a **copy-up**: it copies the whole file into upperdir, then applies the write there. Lower layers are never modified. - **Delete** — a whiteout entry is created in upperdir; the lower copy is untouched, only hidden. - **Rename** — can be expensive or restricted across layers, since the source may live in a lower layer and must be copied up. Because the image layers are read-only and shared, many containers from the same image share exactly one on-disk copy of those layers. Only their upperdirs differ, so each container costs roughly what it writes. ## Why the writable layer is per-container Each container gets its own upperdir and its own merged mount. That is the entire mechanism behind "containers are isolated but share the image": identical lowerdir list, different upperdir. It is also why the container's changes vanish when the container is removed — only its upperdir is deleted. ## Practical notes - The overlay mount options for a running container are visible in `/proc/mounts` on the host, or via `docker inspect` under `GraphDriver.Data` (`LowerDir`, `UpperDir`, `WorkDir`, `MergedDir`). - The backing filesystem matters: overlay2 requires `d_type` support, which on XFS means the filesystem was formatted with `ftype=1`. Without it, behaviour is broken in subtle ways. - Mount options are passed as a string with a length limit, which historically bounded how many lowerdirs a single mount could carry — one of the reasons very long layer chains are discouraged. - Bind mounts and volumes are mounted *over* the merged view. Anything under such a mount point bypasses overlayfs entirely, which is why volumes avoid copy-up costs. ## Answering well Name the four directories with one clause each, state the two lookup rules (upper wins for files, directories merge), and then connect it to the two behaviours interviewers usually chase next: copy-up on first write and whiteouts on delete.

  • Why must workdir be on the same filesystem as upperdir?
    The kernel uses workdir to stage operations — notably copy-up — and then moves the result into upperdir. That move must be an atomic rename, which is only possible within one filesystem. If workdir were elsewhere the kernel would have to copy across filesystems, losing atomicity, so the mount is rejected.
  • Two containers run from the same image. How do their overlay mounts differ?
    Their lowerdir lists are identical, pointing at the same read-only image layer directories on disk, so those layers are stored once. What differs is that each container has its own upperdir, workdir and merged mount. All isolation of writes comes from that separate upperdir.

Sheets of glass stacked over a printed page: you look down through them all at once. Anything drawn on the top sheet hides what is printed beneath at that spot, but blank areas let the lower sheets show through.

saying these in an interview costs you the question

  • Saying each container gets a private copy of the image layers
  • Claiming a directory present in several layers shows only the topmost version's entries
  • Thinking workdir is where the container's files are stored
  • Believing writes can modify a lower layer directly under some conditions

context

resolving links…