When would you choose TGI over vLLM for a production LLM endpoint?

## Why this question is asked An interviewer asking 'TGI or vLLM?' is not testing whether you memorised a benchmark table. Benchmarks go stale within a release cycle, and both projects ship constantly. They are testing whether you evaluate infrastructure on mechanics and organisational fit, or on which project has more stars. ## What they share Start by refusing the false dichotomy. Both servers: - Batch continuously, admitting and retiring requests between decoding steps rather than waiting for a fixed batch to fill. - Manage the KV cache in blocks so memory is not fragmented by variable-length sequences. - Expose an OpenAI-shaped chat completions API, so client code is largely portable. - Shard a model across GPUs on one node. - Stream tokens over server-sent events, expose Prometheus metrics, and ship as a container you point at a model. If your answer's differentiators are in that list, you have not differentiated. ## Where they genuinely differ **Ecosystem gravity.** TGI is Hugging Face's own server. Model ids, gated-repo tokens, tokenizer configs and chat templates all come from the same place your team already gets its models. For an organisation whose entire model workflow is Hub-shaped, that alignment is worth more than a percentage of throughput. **Implementation and operational shape.** TGI splits a Rust router in front of Python model shards, which gives the request-handling layer predictable overhead under high request rates. That is a real architectural difference, though it rarely dominates end-to-end latency compared with the GPU work itself. **Built-in features.** Grammar-constrained decoding (JSON Schema and regex) is a first-class TGI request parameter, and speculation is a launcher flag that works on ordinary checkpoints via n-gram lookup. If those are on your requirements list, TGI gives them to you without extra components. **Pace of model and quantization support.** vLLM's release cadence and contributor base tend to bring new architectures and new quantization formats sooner. A team that must serve a brand-new open-weight model the week it drops feels this acutely; a team standardised on a stable model family does not. **Vocabulary of the knobs.** This is where interviewers catch bluffing. TGI's prefill budget is `--max-batch-prefill-tokens` and its admission threshold is `--waiting-served-ratio`. vLLM's are `--max-num-batched-tokens` and `--max-num-seqs`, with `--gpu-memory-utilization` reserving the cache fraction. Quantization is `--quantize` in TGI and `--quantization` in vLLM, and the accepted values are not the same set. Mixing these up in an interview signals you have configured neither. ## How to actually decide 1. **Write the requirements first.** Which models, which context length, which SLO for TTFT and inter-token latency, which structured-output needs, which GPUs you can actually buy. 2. **Check hard constraints.** Does the candidate server support your architecture, your quantized checkpoint format and your hardware today? A hard no ends the comparison immediately, and it is the most common real deciding factor. 3. **Benchmark on your traffic.** Replay your own prompt-length and output-length distribution at your real concurrency. Published benchmarks use prompt mixes that are unlikely to be yours, and the ranking flips with the mix. 4. **Compare goodput, not peak throughput.** Tokens per second while still meeting your latency SLO is the number that maps to cost per served request. A server that wins on raw throughput while violating your TTFT target has not won. 5. **Price the switch.** Migration is not free: different flags, different metric names, different tuning intuition, different failure modes on call. A 10% throughput gain rarely repays retraining an on-call rotation. 6. **Consider not choosing one forever.** Both sit behind the same client contract, so keeping the deployment abstraction thin - a chat-completions-shaped interface, config not code - keeps the option to re-evaluate next year cheap. ## The third option A complete principal-level answer notes that these two are not the only shape. A Triton deployment with compiled TensorRT-LLM engines can win on per-GPU efficiency, at the cost of a build step that pins the engine to a specific GPU and precision, and of orchestration for multi-model or multi-framework serving. That is a different operational commitment, not merely a faster version of the same thing. ## What interviewers listen for The shape of a strong answer: name what is shared, name two or three real differences with correct per-project vocabulary, state that the decision is workload-benchmarked rather than reputation-driven, and price the migration. Candidates who declare a universal winner are, in this domain, telling you they have deployed exactly one thing.