skip to content

A service was changed to run as UID 10001 inside the container, and the Docker daemon has user-namespace remapping enabled. Writes to its bind-mounted data directory now fail with permission denied. How do you reason about the ownership and fix it?

level: seniorimportance: should knowfreq 48%

answer

  1. host UID = subuid base + in-container UID
  2. bind mount = host ownership, no translation
  3. named volume seeds ownership from image
  4. ls -n and id, never trust names
  5. SELinux :z/:Z is a separate cause

basics

~20 s

Work out the host-side UID: remap offset plus the in-container UID (165536 + 10001 = 175537), then chown the host directory to it. Named volumes avoid this because Docker seeds their ownership from the image.

solid answer

~50 s

Permission denied on a bind mount is always a host-side permission check. Two shifts stack here: the process is UID 10001 rather than 0, and remapping adds the subordinate offset, so the kernel sees `offset + 10001` — with the usual 165536 base that is 175537. The host directory is probably owned by root, so the write is refused. Options, roughly in order of preference: 1. **Use a named volume.** On first use Docker copies the image's content *and ownership* into the empty volume, so a directory the image already chowned to 10001 just works. 2. **Chown the host path** to the computed ID: `chown -R 175537:175537 /srv/data`. Deterministic, but couples the host to the mapping. 3. **Group-writable data plus a supplementary group** shared by the container user. 4. **Root entrypoint that chowns then drops privileges** with `gosu`/`su-exec` — works, but the container starts as root. Avoid `chmod 777`; verify with `ls -n` on the host, which prints raw numeric IDs.

code

bash · 4 lines
bash
BASE=$(awk -F: '/^dockremap:/ {print $2}' /etc/subuid)
sudo chown -R $((BASE+10001)):$((BASE+10001)) /srv/data
ls -ln /srv/data
# drwxr-xr-x 2 175537 175537 ... .

go deeper

for a junior

Know that the container's UID must own the host directory, and that ls -n plus id are how you check.

for a middle

Do the arithmetic — subordinate base plus in-container UID — and explain why named volumes usually sidestep it.

for a senior

Compare the options and their costs (host chown couples to the mapping, entrypoint chown reintroduces root), and rule out ro mounts and SELinux labels before concluding it is ownership.

for a principal

Set a fleet convention: fixed runtime UID range, named volumes or data paths provisioned with correct ownership at host build time, so no service ships an entrypoint that needs root to start.

## Why the failure happens Bind mounts expose a host directory straight into the container's mount namespace. There is no ownership translation layer: the kernel checks the writing process's *host-visible* credentials against the *host* inode ownership. Two independent shifts change those credentials: - **The image's USER.** Running as 10001 instead of 0 means the process is a normal unprivileged user with no `CAP_DAC_OVERRIDE` to bypass permission bits. - **User-namespace remapping.** Every in-container ID is offset by the subordinate base from `/etc/subuid`. If the base is 165536, container UID 10001 is host UID 175537. A directory created by an admin as `root:root` with mode 0755 is therefore unwritable. Inside the container the same directory looks like it is owned by an unknown high UID, or by `nobody` when the host owner falls outside the mapped range — that `nobody` display is the classic tell that a mapping is in play. ## The fixes, and what each costs **Named volumes.** When an empty named volume is first mounted over a path that exists in the image, Docker copies that path's contents *and* its ownership and permissions into the volume. So if the Dockerfile did `RUN mkdir -p /app/data && chown 10001:10001 /app/data`, the volume inherits 10001 (remapped consistently) and writes work with no host-side action. This is the cleanest answer and worth stating first in an interview. **Chown the host path.** `chown -R $((165536+10001)):$((165536+10001)) /srv/data`. Correct and explicit, but the number now encodes the daemon's subordinate base; move the volume to a host with a different mapping and it breaks. Script it from `/etc/subuid` rather than hard-coding. **Shared group.** Set the data's group to a known GID with mode 2775 (setgid so new files inherit the group), and give the container that GID via `--user 10001:<gid>` or `--group-add`. Useful when several different services touch the same tree. **Chown at start-up, then drop.** The entrypoint runs as root, fixes ownership of the mounted path, then execs the app under `gosu 10001`. Pragmatic for images shipped to unknown environments, but the container starts privileged, which is exactly what you were trying to avoid; and under remapping the "root" doing the chown can only assign IDs inside its mapped range. **ID-mapped mounts.** Recent Linux kernels can shift ownership at the mount itself, so the on-disk UIDs stay as they are and the container sees them translated. Docker exposes this for bind mounts on engines that support it; check your engine and kernel before promising it, and treat it as the forward-looking option rather than the default answer. ## Diagnosing quickly Use numeric output on both sides — names lie, because the two sides resolve them from different `/etc/passwd` files: - Host: `ls -ln /srv/data` - Container: `docker exec svc ls -ln /app/data` and `docker exec svc id` - Mapping: `cat /proc/$(docker inspect -f '{{.State.Pid}}' svc)/uid_map` If the host UID equals the container UID plus the offset, permissions are the whole story. If they match and writes still fail, look elsewhere: a read-only mount (`ro`), a read-only root filesystem, or an SELinux label mismatch — on SELinux hosts a bind mount needs the `:z`/`:Z` suffix regardless of UIDs, and the denial shows in the audit log rather than as an ownership problem. ## The rule to remember Named volumes carry ownership from the image; bind mounts carry ownership from the host. Choose bind mounts when the host filesystem is the source of truth, and then own the arithmetic — or avoid the arithmetic entirely by using a named volume.

  • Inside the container the mounted directory shows as owned by nobody. What does that tell you?
    The host owner's UID falls outside the container's mapped range, so the kernel has no in-namespace ID to show and reports the overflow ID, which resolves to nobody. It is a strong signal that user-namespace remapping or a rootless daemon is active and the host ownership was never shifted into the mapped block.
  • Why do named volumes usually avoid this problem while bind mounts do not?
    Docker initialises an empty named volume by copying the image's content at that path, including ownership and mode, so an image that already chowned the directory to its runtime UID gets a correctly owned volume. A bind mount is an existing host directory that Docker never touches, so whatever the host filesystem says is what the container gets.

saying these in an interview costs you the question

  • Reaching for chmod 777 on the host directory
  • Forgetting to add the remap offset and chowning to 10001 on the host
  • Assuming Docker chowns bind mounts automatically like it does named volumes
  • Reading usernames instead of numeric IDs when the two sides have different /etc/passwd
  • Blaming permissions when the real cause is a read-only mount or an SELinux label

context

resolving links…