What does `docker run --gpus all` require on the host, and what does it actually do to the container?

### `--gpus` is a request, not a mechanism Unlike `--device`, which the daemon can satisfy on its own, `--gpus` is a *device request*. Docker records it on the container (`docker inspect --format '{{json .HostConfig.DeviceRequests}}'` shows `Driver`, `Count` or `DeviceIDs`, and `Capabilities: [[gpu]]`) and expects some component on the host to know what a GPU is. If nothing does, the run fails immediately with: ``` docker: Error response from daemon: could not select device driver with capabilities: [[gpu]]. ``` That error is a host-configuration message, not an image problem — it appears just the same with a perfectly good CUDA image. ### What the NVIDIA Container Toolkit installs On an NVIDIA host the fulfilling component is the NVIDIA Container Toolkit. Installing it does three things: it puts an OCI runtime on the host that wraps `runc`, it registers that runtime with the daemon (an entry under `runtimes` in `/etc/docker/daemon.json`), and it installs a hook that runs after the container's filesystem is prepared but before its first process starts. The hook does the injection: * creates the control nodes `/dev/nvidiactl` and `/dev/nvidia-uvm` and one `/dev/nvidiaN` per requested GPU inside the container; * bind-mounts the host's user-space driver libraries — `libcuda.so.<driver version>`, `libnvidia-ml.so.<driver version>` and friends — into the container and runs `ldconfig`; * bind-mounts host binaries such as `nvidia-smi`. This is the single most important fact about GPU containers and the source of most interview follow-ups: **the driver comes from the host, the CUDA runtime comes from the image.** A container image must never install the NVIDIA driver; it installs the CUDA toolkit/runtime and the application. That is also why `nvidia-smi` run *inside* the container prints the host's driver version — it is literally the host's binary talking to the host's kernel module. ### Selecting GPUs and capabilities ``` docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi docker run --rm --gpus 2 ... # any two GPUs docker run --rm --gpus '"device=0,1"' ... # specific indices docker run --rm --gpus '"device=GPU-0e1f…"' ... # by GPU UUID docker run --rm --gpus 'all,capabilities=compute,utility' ... ``` The quoting on `device=` is awkward because the value itself must reach the daemon quoted. Under the hood these map onto the environment variables the toolkit reads — `NVIDIA_VISIBLE_DEVICES` and `NVIDIA_DRIVER_CAPABILITIES` — which the vendor's CUDA base images already set (typically `compute,utility`). Capabilities decide which slices of the driver get mounted: `compute` for CUDA, `utility` for `nvidia-smi`/NVML, `video` for hardware encode/decode. A container that only needs NVENC does not need `compute`. ### The older spellings Before Docker 19.03 there was no `--gpus`; you ran `--runtime=nvidia`, or used a `nvidia-docker` wrapper, or set the NVIDIA runtime as the daemon's default. Those still work where they are configured, and you will meet them in old runbooks, but `--gpus` is the current spelling and the one to give in an interview. ### CDI: the vendor-neutral successor The Container Device Interface (CDI) replaces per-vendor hooks with a declarative spec file. A tool on the host (for NVIDIA, `nvidia-ctk cdi generate`) writes a JSON/YAML spec into `/etc/cdi` or `/var/run/cdi` describing named devices such as `nvidia.com/gpu=0` and exactly which nodes, mounts and environment variables each one implies. Docker Engine gained CDI support in 25.0, initially behind a `cdi` entry in the daemon's `features` map, and CDI devices are requested through **`--device`, not `--gpus`**: ``` docker run --rm --device nvidia.com/gpu=0 my-inference:2.9 ``` That detail — GPUs arriving through `--device` once CDI is in play — is a common gotcha for people who learned only the `--gpus` spelling. ### What `--gpus` does *not* do It controls *visibility*, not partitioning. Two containers each started with `--gpus all` see the same physical GPU and compete for its memory; one filling the card makes the other fail to allocate. There is no per-container GPU memory limit equivalent to a memory cgroup. Placement across many GPUs and many hosts is a scheduling problem that belongs to an orchestrator, not to `docker run`. The single-host answer is discipline: hand each container an explicit `device=` list rather than `all`, so two jobs cannot silently land on the same card. Finally, verify rather than assume. `docker run --rm --gpus all <cuda-base-image> nvidia-smi` is the one-line smoke test that proves the toolkit is installed, the driver is loaded, the nodes were injected, and the container can see the cards — and it is worth running as part of node bootstrap before any real workload is scheduled onto a host.