Why does docker exec -it <id> sh fail on a scratch-based image, and what can you do instead?
answer
- The image ships only the application
- exec starts a process, not a session
- The binary is resolved inside the container
- Bring tools from host or sidecar
- Never restart the container to add one
basics
~20 sdocker exec starts a process from the container's own filesystem, and a scratch or distroless image ships no /bin/sh to start. Bring tools from outside instead: nsenter from the engine host, a toolbox container sharing its namespaces, or a copied static binary.
solid answer
~50 s`docker exec` is not a remote login. It asks the runtime to start a **new process inside the target container's namespaces**, including its mount namespace, so the program you name must exist in that container's filesystem. An image built `FROM scratch`, or a distroless image carrying only a runtime, has no `/bin/sh`, so the runtime returns `exec: "sh": executable file not found in $PATH`. Nothing is broken; there is simply nothing to launch. The answer is to bring a shell to the container rather than expect one in it: `nsenter` into the container's namespaces from the engine host using its host PID, so host binaries stay reachable; start a throwaway toolbox container with `--pid=container:<target>` and read the target's files through `/proc/1/root`; or copy a statically linked `busybox` in and exec that. Which route is open depends on whether you have root on the engine host.
code
bash · 5 lines$ docker exec -it billing-cron sh
OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown
$ docker exec -it billing-cron /bin/sh
OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknowngo deeper
Be ready to say where the binary comes from: docker exec looks it up inside the container's filesystem, so an image with no shell has nothing to run. Recognise the executable file not found in $PATH message on sight.
Explain the mechanics: exec enters the container's namespaces and calls execve in its mount namespace, which is why host tools are invisible. Then name at least two ways to bring tooling in and what each one requires.
Show the incident discipline: do not restart or rebuild a container you are diagnosing, because that destroys the live state. Pick a route based on what access you actually have — host root versus only the Docker API.
Own the policy angle: minimal runtime images are a deliberate security posture, so the organisation needs a sanctioned debugging path — debug-tagged image variants, a curated toolbox image, and rules about who may enter namespaces on an engine host.
### What `docker exec` actually does Calling `docker exec` "opening a shell in the container" hides the failure mode. The command asks the engine to start a **new process inside an already-running container**: the runtime enters that container's namespaces and cgroup, applies its user, working directory and security profile, and then calls `execve()` on the program you named. That program is resolved in the **container's own root filesystem** — the read-only image layers plus the writable container layer, assembled in the container's mount namespace. Your laptop's `/bin/bash`, the host's `/usr/bin`, and the daemon's own binaries are all in a different mount namespace and are simply not visible. So for a subscription-billing cron packaged as a single static Go binary in a `FROM scratch` image, `docker exec -it billing-cron sh` fails with something like: ``` OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown ``` The same happens on a distroless image that ships a language runtime and CA certificates and nothing else. The `-it` flags make no difference: they allocate a TTY and keep stdin open for a process that is never created. Naming an absolute path (`/bin/sh`) fails identically — the file does not exist. ### Why images are built this way A runtime image with no shell, no package manager and no coreutils has a much smaller attack surface: an attacker who achieves command execution in it has no interpreter to pivot with, and there is nothing to `curl | sh`. It is also much smaller to pull. Interviewers ask this question because the slim image is the modern default and the on-call engineer still has to debug it at 03:00. ### The wrong instinct The reflex answer — "rebuild the image with a shell in it" — is wrong twice. First, it is slow: on a project whose builder holds a 92 GB build-cache directory, a rebuild-and-redeploy is minutes you do not have during an incident. Second, and fatally, **it destroys the state you were investigating**. Restarting the container throws away the process's memory, its open file descriptors, the half-written file in `/tmp` that reproduces the bug, and the exact scheduling accident you were chasing. Debugging a live container means not touching the container. ### The three real routes **1. Enter it from the host with `nsenter`.** Every container is just processes on the engine host. `docker inspect -f '{{.State.Pid}}' billing-cron` gives the host PID of the container's PID 1; `nsenter -t <pid> -n -p` (as root on the host) puts you into the container's network and PID view while keeping the **host's** mount namespace, so the host's `ss`, `strace` and `ls` all still work. This needs root on the machine running `dockerd`, which you may not have — and on Docker Desktop the daemon lives inside a Linux VM, so there is no such PID on your own machine. **2. Join it from a toolbox container.** `docker run --rm -it --pid=container:billing-cron --network=container:billing-cron busybox sh` gives you a shell that comes from the toolbox image but sees the target's processes and network interfaces. The mount namespace is not shared, so the target's files are reached through `/proc/1/root`. This route needs only the Docker API, not host root. **3. Copy the tools in.** Push a statically linked `busybox` into the running container and exec that binary directly (`docker exec -it billing-cron /busybox sh`). It works when nothing else is available, but it mutates the container's writable layer and fails on a read-only root filesystem. ### A fourth answer worth giving Say out loud that the durable fix is a **debug variant built from the same source**: a build stage that adds shell and tooling, published under a separate tag, run alongside or in place of the slim one only when needed. That keeps production hardened while giving on-call something to reach for. It does not help you *right now* with a live container, which is why the interviewer wants the first three as well. ### What a good answer sounds like Name the mechanism (exec resolves the binary in the container's filesystem), name the failure string, say explicitly that the container is healthy and the tooling is what is missing, then give at least two routes and the constraint that picks between them: do you have root on the engine host, or only the Docker API?
- The image has no /bin/sh but does ship /bin/busybox. What do you run?`docker exec -it <container> /bin/busybox sh`. BusyBox is a multi-call binary: invoked with an applet name as its first argument it behaves as that applet, so it gives you a shell without a `/bin/sh` entry existing. The same trick works for `busybox ls`, `busybox ps` and the rest of the applets compiled into that build.
- Does the same command fail differently on a container that has already exited?Yes, and the different error is diagnostic. `docker exec` on a stopped container fails with a message saying the container is not running, before the binary is ever looked up. Every live-entry route needs a running process: `nsenter` needs a host PID that only exists while the container runs, and joining namespaces from a toolbox container needs those namespaces to still be alive.
- Why not just add a shell to the production image and end the problem?It works, and it hands an attacker an interpreter and, with a package manager, a way to fetch more. The usual compromise is to keep the runtime image minimal and publish a debug-tagged variant from the same build, so the tooling exists as an artefact you can run deliberately rather than as a permanent part of the production attack surface.
Asking docker exec for a shell is like asking a courier to fetch a hammer from inside a sealed room: if there is no hammer in the room, the courier comes back empty-handed. You either open a hatch from outside (nsenter), send in someone who brought their own toolbag (a toolbox container), or post the hammer through the slot (a static binary).
saying these in an interview costs you the question
- Says docker exec runs the command on the host
- Claims the failure means the container is stopped or broken
- Thinks -it creates a shell even with none in the image
- Reaches for docker attach expecting it to open a shell
- Proposes rebuilding the image, discarding the live state
- Believes distroless or scratch images are simply misbuilt