Why does each running container get its own long-lived shim process sitting between the container manager and the container's init process, and what does that buy you when the container manager or Docker daemon is restarted?

## The problem the shim solves When `runc` starts a container it creates the namespaces, applies cgroups and security settings, then `execve`s the entrypoint — and exits. Something has to remain behind, because a running process needs a parent that will: - collect its **exit status** (only the parent gets it from `wait()`; if a process is re-parented to host PID 1 the status is reaped by init and lost to the container system), - hold the **stdout/stderr pipes or the TTY master**, otherwise the writing end closes and the process gets `SIGPIPE` or its logs vanish, - accept **exec** requests into the container later, - keep the container's **cgroup and namespaces** referenced and clean them up correctly at the end. The obvious candidate is the daemon itself. That is exactly what early Docker did, and it is why restarting Docker used to kill every container on the host. ## The shim as the answer containerd instead starts a small **shim** process per container (`containerd-shim-runc-v2`), double-forked so it is *not* a child of containerd. The shim: 1. invokes the OCI runtime binary (`runc create`, then `runc start`) — or, for other runtime handlers, whatever binary implements that handler; 2. becomes the parent of the container's init process; 3. owns the stdio: pipes into the logging path, or the TTY master for interactive containers; 4. exposes a small ttrpc API over a socket so containerd can start execs, resize TTYs, send signals and receive exit events; 5. reaps the process, reports the exit code and container-exit event, and tears down mounts and the cgroup afterwards. The v2 shim API allows a shim to serve **multiple containers in one sandbox**, which is what pod-level and VM-based runtimes (Kata) use; for plain Docker it is effectively one per container. ## What the decoupling buys **Manager restarts are non-events for workloads.** Since containerd is not in the container's process tree, `systemctl restart containerd` leaves containers running. On start-up containerd rediscovers the shims by their socket addresses recorded in its state directory, reconnects, and re-subscribes to exit events. Anything that exited while containerd was down is reconciled from the shim's recorded exit status. **Docker Engine gets the same property, but it is a setting.** Restarting `dockerd` only leaves containers running if `"live-restore": true` is set in `/etc/docker/daemon.json`. Without it, the daemon stops containers on shutdown. Live-restore carries caveats worth naming: it is not supported together with swarm mode, and a daemon restart that changes bridge/network configuration or a major daemon version upgrade may still require containers to be recreated. Logs continue to be buffered by the shim while the daemon is away. **Zero-downtime engine upgrades** follow directly. You can patch containerd or Docker on a node while application containers keep serving traffic, which matters enormously on Kubernetes nodes where kubelet and the runtime get updated independently of workloads. **Crash isolation.** A daemon panic does not take down customers' processes. The blast radius of a bug in the control plane is the control plane. ## Costs and failure modes - **Per-container process cost.** A few megabytes of RSS and a PID per container; at hundreds of containers per node this is real but small. - **Orphaned shims.** If containerd dies uncleanly or state is wiped, shims can survive with no manager to talk to. They keep containers alive but invisible to `docker ps`/`ctr tasks ls` until reconciliation, and in bad cases must be cleaned up manually after stopping the container. - **Version skew.** The shim binary in use is the one that was running when the container started; upgrading containerd does not upgrade in-flight shims, so old shims linger until their containers are recreated. That is normally fine but is a real consideration when a shim-level bug is being patched. - **Log gap.** While the daemon is restarting, the shim buffers; a very long daemon outage with a very chatty container can hit buffer limits. ## How to see it `ps -ef --forest` on a Docker host shows a flat set of `containerd-shim-runc-v2` processes, each with the container's process tree beneath it, and `containerd`/`dockerd` sitting entirely to one side. Killing `dockerd` and watching `curl` against a published port keep succeeding is the clearest possible demonstration — and a good thing to have actually tried before the interview.