What does a minimal TGI docker run command need to serve a Hub model?

## What you are actually starting TGI is not a Python library you import; it is a server distributed as a container image. Inside that image the entrypoint is the `text-generation-launcher` binary, which supervises three things: a **downloader** that fetches the model weights from the Hugging Face Hub into a cache directory, one or more **shard** processes that hold the model on GPU and run the forward passes, and a **router** that owns the HTTP surface, validates requests, and batches them onto the shards. This matters for the command line. Because the entrypoint is the launcher, `docker run` arguments split at the image name: `--gpus`, `-p`, `-v`, `-e`, `--shm-size` are Docker's and go before it; `--model-id`, `--num-shard`, `--quantize`, `--max-total-tokens` are the launcher's and go after it. Putting `--model-id` before the image name is the single most common first-run mistake, and Docker's error message about an unknown flag is not obvious. ## The minimum viable launch ``` docker run --gpus all --shm-size 1g -p 8080:80 \ -v $PWD/data:/data \ ghcr.io/huggingface/text-generation-inference:3.3.5 \ --model-id HuggingFaceH4/zephyr-7b-beta ``` **`--gpus all`** hands the container GPU devices through the NVIDIA container runtime. Without it the container starts, the launcher runs, and the shard dies when it cannot find a CUDA device. On a multi-GPU box you can narrow this to `--gpus '"device=0,1"'`, or set `CUDA_VISIBLE_DEVICES` inside the container, when you want to leave other cards for other workloads. **`-p 8080:80`** publishes the port. The container serves HTTP on **80**; the host port is your choice. Every curl example that hits `http://localhost:8080/generate` assumes this mapping, and forgetting that the container side is 80 (not 8080) is a frequent source of connection-refused confusion. **`-v $PWD/data:/data`** is the weight cache. On a cold container the launcher downloads the repo — for a 7B model that is roughly 14 GB at bf16, for a 70B far more. Mounting a host directory at `/data` means the second start is seconds instead of minutes, and it means a crash-looping container is not re-pulling gigabytes each time. On Kubernetes this is the same argument for a persistent volume or a node-local scratch disk rather than the pod's ephemeral filesystem. **`--shm-size 1g`** raises the container's `/dev/shm`. Docker defaults it to 64 MB. A single-shard run usually survives that, but as soon as you shard across GPUs the shard processes coordinate through shared memory and the run hangs or dies with an opaque error. Setting it unconditionally is the cheap habit. **`--model-id`** takes a Hub repo id such as `meta-llama/Llama-3.1-8B-Instruct`. It can also take a **local path** to a directory of weights inside the container — useful when your weights come from an artifact store rather than the Hub, and mandatory in air-gapped environments. `--revision` pins a branch, tag, or commit sha so a redeploy cannot silently pick up a changed repo; leaving it unpinned means "whatever main is today", which is not what you want in production. ## Confirming it came up Startup is not instant: download, then load and shard the weights, then a warmup pass that probes how much memory the server can actually use. The launcher logs the stages, ending with a line reporting the shards ready and the router connected. Two routes make this scriptable — `/health` for readiness (this is what a Kubernetes readiness probe should hit, not the generation route) and `/info`, which reports the model id, revision, dtype and the effective token limits the server settled on. Reading `/info` after a deploy is the fastest way to catch "we shipped the wrong revision" or "the limits are not what the manifest said". A first smoke test is a POST to `/generate` with `{"inputs": "...", "parameters": {"max_new_tokens": 20}}`; a 200 with a `generated_text` field means the whole chain works. ## Image variants and pinning The published image has hardware variants — the default CUDA image, plus `-rocm` for AMD GPUs and `-gaudi` for Intel Gaudi, and a `latest-trtllm` variant. Pin an explicit tag (`:3.3.5`) rather than `latest`: an inference server upgrade can change default token limits, kernel selection and supported quantization values, and you want that to be a deliberate change with a test behind it rather than something that arrives on the next node restart. ## What this launch does not yet handle A gated repo needs a token in the environment. A model too large for one card needs sharding. A model whose context you want to bound needs explicit token limits. Those are separate flags layered onto exactly this command — the shape above does not change.