The Docker daemon on a Linux host is unresponsive, yet the application containers on that host are still serving traffic. Explain why that is possible and how you would inspect and manage those containers while the daemon is down.

## Why containers outlive the daemon The process tree on a Docker host is ``` systemd ─┬─ dockerd ├─ containerd └─ containerd-shim-runc-v2 ── your-app (container PID 1) ── children ``` The shims are deliberately not children of containerd, and `dockerd` is not an ancestor of anything in the container at all. `runc` did its work — namespaces, cgroups, seccomp, `pivot_root`, `execve` — and exited. So a daemon that is hung on a lock, blocked on a wedged storage driver, or outright killed does not signal or reap any container. The workload keeps serving; only the *control plane* is unavailable. This is the answer to the pager: **traffic is fine, management is down.** Resist the reflex to restart Docker immediately, because that reflex is what turns a control-plane incident into a customer-facing one on a host where `live-restore` is not enabled. ## Layer down: talk to containerd If containerd is healthy — check `systemctl status containerd` and `ctr version` — you still have near-full control: ``` ctr -n moby containers ls ctr -n moby tasks ls ctr -n moby tasks exec --exec-id dbg <container-id> sh ctr -n moby tasks kill --signal SIGTERM <container-id> ``` The `-n moby` namespace flag is essential: Docker's objects live in containerd's `moby` namespace, and without it the listing looks empty and people wrongly conclude the containers are gone. `nerdctl -n moby ps` gives a friendlier, Docker-like view of the same data if it is installed. On Kubernetes nodes the equivalents are `crictl ps` / `crictl logs` / `crictl exec` against the `k8s.io` namespace. ## Layer down again: plain Linux Even if containerd is also unhealthy, the container is just processes in namespaces and cgroups: - **Find them.** `ps -ef --forest` and look under each `containerd-shim-runc-v2`. The shim's command line carries the container ID. - **Map a PID to a container.** `cat /proc/<pid>/cgroup` shows the cgroup path containing the container ID; `ls -l /proc/<pid>/ns/` shows its namespace inodes, which lets you tell which processes share a namespace. - **Enter it.** `nsenter -t <pid> -m -u -i -n -p sh` gives you a shell in the container's namespaces without any container tooling — invaluable when `exec` is unavailable. Narrower forms are often better: `nsenter -t <pid> -n ss -ltnp` inspects listening sockets in the container's network namespace using the *host's* binaries, which also works for distroless images that have no shell. - **Resources.** `/sys/fs/cgroup/<path>/memory.current`, `memory.max`, `cpu.stat`, `pids.current` show live usage and limits (cgroup v2 names). - **Filesystem.** `/proc/<pid>/root/` is the container's rootfs as seen from the host; you can read config files or copy something out without any Docker command. - **Logs.** With the default json-file driver, `/var/lib/docker/containers/<id>/<id>-json.log` keeps growing because the shim owns the write path. If a remote logging driver is configured, logs continue flowing there too. - **Signals.** `kill -TERM <container-pid-1>` stops a container the blunt way; the shim will still record the exit. ## Diagnosing the daemon before restarting Collect evidence first, because a restart destroys it: - `journalctl -u docker -n 500 --no-pager` and `journalctl -u containerd`. - Send `SIGUSR1` to `dockerd` — it dumps all goroutine stacks to its log (or `/var/run/docker/goroutine-stacks-*.log`), which usually shows the deadlock or the blocked syscall. - Check the usual culprits: disk full on `/var/lib/docker` (very common; `df -h` and `df -i` for inodes), a hung NFS or FUSE mount blocking the storage driver, an exhausted file-descriptor limit, or a runaway number of containers/networks. - `docker system df` if the API responds at all. ## Recovering With `"live-restore": true` in `/etc/docker/daemon.json`, `systemctl restart docker` re-attaches to the existing shims and containers never stop. Without it, the restart stops every container — so on a production host with live-restore off, plan the restart like a maintenance window (drain traffic first) rather than firing it blind. After the daemon returns, reconcile: containers that exited while it was down are reported from the shims' recorded exit status, and any orphaned shims (from an unclean containerd crash) show up as tasks with no matching container and may need `ctr -n moby tasks kill`/`delete` cleanup. ## The takeaway for the interviewer The answer they want is that you understand the stack well enough to *not* need Docker to operate a Docker host — and disciplined enough to capture the daemon's state before restarting it.