skip to content

How do you get a static busybox running inside a Docker container that has no shell, and what breaks?

level: seniorimportance: nice to knowfreq 22%

answer

  1. Make the container contain the tools
  2. One self-contained binary, many applets
  3. No loader exists in an empty image
  4. The error names the wrong missing file
  5. It stays in the writable layer

basics

~20 s

Place a statically linked busybox binary into the container's writable layer and exec it directly, for example docker exec -u 0 -it app /busybox sh. It fails if the binary is dynamically linked, built for another architecture, or the destination is read-only or mounted noexec.

solid answer

~40 s

Put a roughly 1.9 MB **statically linked** `busybox` into the container's filesystem and run it by absolute path: `docker exec -u 0 -it billing-cron /busybox sh`. BusyBox is a multi-call binary, so `busybox sh` gives you a shell and `busybox ls`, `busybox ps` and friends give you the rest. Three things break it. A **dynamically linked** binary in a `FROM scratch` image fails with `no such file or directory` even though the file is plainly there — the missing file is the ELF interpreter, `/lib64/ld-linux-x86-64.so.2`. An **architecture mismatch** gives `exec format error`. And a container started with `--read-only`, or whose only writable path is a `noexec` mount, gives you nowhere to put an executable. Remember the binary lands in the container's writable layer and stays there, so remove it when you are done.

code

bash · 10 lines
bash
# verify BEFORE copying: static, and the right architecture
file ./busybox     # ... x86-64 ... statically linked
ldd  ./busybox     # not a dynamic executable

docker cp ./busybox billing-cron:/busybox
docker exec -u 0 billing-cron /busybox chmod +x /busybox
docker exec -u 0 -it billing-cron /busybox sh

# clean up when finished
docker exec -u 0 billing-cron /busybox rm -f /busybox

go deeper

for a junior

Know that a single self-contained binary such as busybox can supply a shell and dozens of utilities, and that it must be run by its absolute path because the image has no PATH directories.

for a middle

Explain dynamic versus static linking and the ELF interpreter, so you can account for a 'no such file or directory' error on a file that clearly exists. Know that architecture must match too.

for a senior

Show that you weigh the cost: this mutates a hardened production container, so you check the writable and noexec constraints first, clean up afterwards, and prefer the read-only namespace routes when they are available.

for a principal

Treat needing this at all as a platform gap. If the only way on-call can debug is by injecting binaries into live containers, the answer is a supported debug image variant and a sanctioned access path, not a better trick.

### When this is the route Entering namespaces from the host needs root on the engine host. Joining them from a toolbox container needs the ability to start containers on that engine. If you have neither — a locked-down environment where all you have is `exec` rights on one container through a restricted API, or a platform that will not let you start a privileged sidecar — the last option is to make the container contain tools. You do that by putting a self-contained binary into it and executing that binary directly. ### Why specifically a *static* binary A normal Linux executable is dynamically linked: the ELF header names an **interpreter** (typically `/lib64/ld-linux-x86-64.so.2` on glibc x86-64, or `/lib/ld-musl-x86_64.so.1` on musl), and the kernel loads that interpreter first, which then resolves `libc.so.6` and the rest. A `FROM scratch` image contains none of those files. The result is the single most confusing error in this whole area: ``` $ docker exec -it billing-cron /busybox sh OCI runtime exec failed: exec failed: unable to start container process: exec: "/busybox": stat /busybox: no such file or directory: unknown ``` ...or, in a shell context, `bash: ./busybox: No such file or directory` for a file you can plainly see with `ls`. The kernel's `execve()` returns `ENOENT` for the **missing interpreter**, not the missing binary, and nothing in the message tells you which file it means. A statically linked build has no interpreter and no shared-library dependencies, so it runs in an empty filesystem. Confirm before you copy: ``` $ file busybox busybox: ELF 64-bit LSB executable, x86-64, ..., statically linked, stripped $ ldd busybox not a dynamic executable ``` Sources of a static build include distribution packages named along the lines of `busybox-static`, and BusyBox image variants that are built statically — check with `file`, do not assume from the tag. ### Architecture, which is the second trap The binary must match the container's platform, not your laptop's. Copying an `x86-64` busybox into an `arm64` container yields `exec format error`. On a machine that builds multi-platform images this bites often, because the image you are debugging may well have been pulled for a different architecture than the one you extracted the binary on. Check the container's platform before you extract the binary rather than after the error. ### Where the binary can live The copy lands in the container's **writable layer**, which is normally writable — but not always: * `docker run --read-only` makes the whole root filesystem read-only. The copy fails, or the file appears in a writable mount and nowhere else. * The usual escape is a writable mount that is already present, such as a volume or a tmpfs. But a tmpfs mount can be established with `noexec`, and then the binary is there and still refuses to run — a permission-denied that has nothing to do with file permissions. * If the container runs as a non-root user, `docker exec` uses that user by default. You will typically want `-u 0` both to place the file and to run it, and the file needs the execute bit set. ### Running it BusyBox is a **multi-call** binary: it inspects `argv[0]`, or its first argument, and behaves as that applet. So a single file gives you `sh`, `ls`, `ps`, `netstat`, `cat`, `top` and dozens more: ``` docker exec -u 0 -it billing-cron /busybox sh / # /busybox ps / # /busybox ls -l /proc/1/fd ``` Inside that shell your PATH is useless — there is no `/bin` — so call applets through the binary's absolute path, or run `/busybox --install -s /tmp/bin` to create applet symlinks in a writable directory and add it to PATH. ### The cost, which is the part interviewers care about This is the only route of the three that **changes the container you are debugging**. The binary persists in the writable layer for the life of the container, so it is present in anything derived from that container's filesystem afterwards, and you have just given a hardened, deliberately shell-less workload a shell that will survive until it is removed or replaced. Delete it when you finish, note it in the incident record, and treat the whole manoeuvre as a last resort behind the two read-only routes. If your platform makes this the *only* option, the real fix is organisational: publish a debug-tagged variant of the image from the same build so on-call has a supported way in.

  • The binary is visibly present but exec reports 'no such file or directory'. What is actually missing?
    The ELF interpreter. A dynamically linked executable names a loader such as `/lib64/ld-linux-x86-64.so.2` in its program headers, and the kernel returns `ENOENT` when that loader is absent — which it always is in a scratch image. The message names your binary, but the missing file is the loader. Check with `file` or `readelf -l`, and use a statically linked build instead.
  • The container was started with --read-only. Can you still use this technique?
    Not into the root filesystem — writes are rejected. You need an already-present writable path such as a volume or tmpfs mount, and even then a tmpfs can be mounted `noexec`, in which case the binary lands successfully and still cannot be executed. When there is no writable, executable location, this route is closed and you fall back to entering the container's namespaces from outside.
  • Why is this considered the last resort of the shell-less debugging techniques?
    It is the only one that mutates the container under investigation. The binary sits in the writable layer until someone removes it, so a workload that was deliberately built without an interpreter now has one, and anything derived from that container's filesystem carries it. The namespace routes read the container from outside and leave no trace, which is why they come first.

saying these in an interview costs you the question

  • Copies a dynamically linked binary into a scratch image
  • Reads 'no such file or directory' as the binary missing
  • Ignores the container's CPU architecture when choosing a binary
  • Assumes docker exec always runs commands as root
  • Forgets the binary persists in the writable layer
  • Believes a read-only root filesystem can still take the copy

context

resolving links…