How do TGI's /generate and /v1/chat/completions routes differ?

## Two front doors onto one engine A TGI process holds exactly one model and exposes more than one HTTP surface onto it. The routes are not different engines or different capabilities — they are different request contracts, and the real distinction is **who is responsible for turning a conversation into a prompt string**. ## The native route `POST /generate` takes a body of the form `{"inputs": "<prompt text>", "parameters": { ... }}`. The `inputs` string is used as-is. TGI applies no chat template, inserts no system preamble and adds no role markers. Whatever you send is what the model continues. That is a feature when you want it. Instruction-tuned models are trained against a specific formatting — special tokens marking the system, user and assistant turns — and a client that knows this formatting can control it exactly: constructing few-shot prefixes, prefilling the start of the assistant's reply, or serving a base model where the very idea of a chat turn does not apply. It is a footgun when you do not know it: posting bare user text to an instruction-tuned model produces plausible-looking, subtly worse output, and nothing in the response tells you the template was skipped. The response is `{"generated_text": "..."}`. Passing `"details": true` returns additional per-generation detail such as the finish reason and token-level information, which is what you want when you are debugging why a generation stopped where it did. `POST /generate_stream` is the same contract with incremental delivery: instead of one JSON body at the end, the connection stays open and the server emits events carrying successive pieces of the generation, ending with the completed text. Same request shape, different delivery. ## The Messages API `POST /v1/chat/completions` takes `{"model": "tgi", "messages": [{"role": "user", "content": "..."}], ...}`. Here the server does the prompt construction: it renders your message array through the chat template that ships with the model's tokenizer, so the role markers are correct by construction for whatever checkpoint is loaded. Swap the model id in the launch command and the template changes with it; your client code does not. The response comes back in the OpenAI shape — a `choices` array with a message and a finish reason, plus usage information — and `"stream": true` produces the OpenAI incremental delta format rather than TGI's native event shape. The `model` field deserves a note: TGI serves one model per process, chosen at launch by `--model-id`. The field exists because the OpenAI contract requires it, and examples commonly pass the literal `"tgi"`. It is a label, not a selector — you cannot ask a running TGI for a different model. Multi-model routing is a layer above the server. ## Why the compatibility layer matters The practical value of `/v1/chat/completions` is migration. A client library already pointed at a hosted OpenAI-compatible endpoint usually needs one change — its `base_url` set to `http://your-host:8080/v1` — to talk to your self-hosted model instead. No prompt-format research, no rewrite of the streaming handling, and the same code path can be switched back by config. That is why teams standardise on it even when they own both ends: it keeps the option of moving between a hosted provider and self-hosted weights a deployment decision rather than a code change. The cost is that the compatibility is a shape, not a promise of identical behaviour. Provider-specific fields, and behaviours around things the server does not implement, are where the analogy leaks. Test the specific fields your client sends rather than assuming a hosted provider's full surface is reproduced. ## Choosing between them - **Use the Messages API** for ordinary chat traffic, and especially when existing OpenAI-shaped tooling, SDKs or gateways sit in front of the server. Let the server own the template; that is one fewer thing to get wrong per model swap. - **Use `/generate`** when you need the prompt string to be exactly what you wrote: base models, custom few-shot layouts, prefilled assistant openings, or evaluation harnesses that must reproduce a documented prompt byte for byte. A useful rule: if you find yourself hand-writing the model's special role tokens into a `messages` content field, you wanted `/generate`. If you find yourself reimplementing a chat template in client code, you wanted the Messages API. ## Operational notes Both routes hit the same router, so the same validation limits apply to both, and an over-long request is rejected identically whichever door it came through. Neither route authenticates callers — TGI is a model server, not a gateway; if the endpoint needs auth, rate limits per tenant, or request logging, that belongs in a proxy or ingress in front of it. And because both routes share one queue and one set of shards, a heavy batch job on `/generate` and interactive chat on `/v1/chat/completions` are competing for the same GPU; separating those workloads means separate deployments, not separate routes.