What does a minimal vllm/vllm-openai docker run need to serve a Hub model?

## The image's contract `vllm/vllm-openai` is the official image whose entrypoint is the OpenAI-compatible API server. That single fact shapes the command line: you do not pass a shell command, you pass **engine arguments**, and they land on the server directly. The image listens on port 8000 by default, so `-p 8000:8000` makes it reachable. ## The four things a minimal run must supply **GPU access.** `--runtime nvidia --gpus all` (or `--gpus '"device=0,1"'` to select) exposes devices through the NVIDIA container toolkit. Without it the container starts and then fails during engine init because no CUDA device is visible. This is the single most common first failure and it is worth being able to say what the toolkit does: it injects the driver libraries and device nodes into the container. **Shared memory.** vLLM runs on PyTorch, which passes tensors between processes through shared memory — most visibly when tensor parallelism spawns one worker per GPU. Docker gives a container 64 MiB of `/dev/shm` by default, which is not enough; the run hangs or dies with a bus error. `--ipc=host` shares the host's IPC namespace and is what vLLM's own documentation shows; `--shm-size=16g` is the more contained alternative when you would rather not share the namespace. **A weights cache.** Mounting `~/.cache/huggingface` to `/root/.cache/huggingface` means the model is downloaded once per host rather than once per container start. Skipping it is not a correctness bug, it is a cold-start and disk bug: a 70B checkpoint is well over a hundred gigabytes, it lands in the container's writable layer, it is thrown away on every restart, and it turns every crash-loop into a bandwidth incident. In production this is a persistent volume, a pre-warmed node-local cache, or weights baked into an image. **Credentials for gated repos.** `--env "HUGGING_FACE_HUB_TOKEN=..."` lets the download authenticate. Without it, gated families fail at download with an authorization error that reads confusingly like a missing-model error. The token is a secret: it belongs in a secret store or an orchestrator secret, never in the image or in shell history. ## Engine arguments Everything after the image name is server configuration, and it is where the deployment's real decisions live: `--model` or the positional model id, `--tensor-parallel-size`, `--max-model-len`, `--gpu-memory-utilization`, `--quantization`, `--kv-cache-dtype`, `--served-model-name` to control the name clients pass in the `model` field. Because they are ordinary process arguments, they can come from an orchestrator's `args` list and be templated per environment. ## Verifying it came up The server exposes `/health` for liveness-style checks and `/v1/models` to confirm which model name it is serving; `/metrics` returns Prometheus text. A server that answers `/health` has finished loading — and loading is slow, because it means reading tens of gigabytes of weights from disk into VRAM and capturing CUDA graphs. Anything watching the container needs to tolerate minutes of startup, which is why a naive health check with a short timeout will kill a perfectly good server in a loop. ## Things people get wrong Treating the image like a stateless web app is the theme of every mistake here. It is not stateless: it holds a large model in VRAM, it wants a warm on-disk cache, and it takes minutes to become useful. It does not autoscale in seconds, it does not survive an aggressive liveness probe during load, and running two of them on one card without lowering each one's memory budget will simply OOM. The image is a convenient package for a heavyweight process, not a lightweight service. ## Variants The published tags are versioned as well as `latest`; pinning an exact version is the right default for a deployment, because engine argument names and defaults do move between releases. Building your own image on top is common when you want to bake in weights, add a startup wrapper, or pin CUDA and driver expectations to your fleet.