Through llama-server, why does one message get a clean reply on /v1/chat/completions but a rambling continuation on /completion?
answer
- one route renders roles, one does not
- raw prompt string, BOS only
- template embedded in the GGUF
- Jinja engine on by default
- a route that shows the rendered prompt
basics
~20 s/completion sends your prompt string to the model raw, so it just continues the text. /v1/chat/completions and /v1/messages first render the messages through the GGUF's chat template with Jinja, on by default in llama.cpp 0.5.0.
solid answer
~40 s`/completion` is llama.cpp's native, non-OpenAI endpoint: it tokenizes `prompt` exactly as sent, adding only a BOS token when the model's metadata asks for one, and continues it. No role markers and no end-of-turn token are added, so a chat-tuned model treats your message as text to extend. `/v1/chat/completions`, and the Anthropic-style `/v1/messages`, render `messages` through the chat template embedded in the GGUF, falling back to ChatML when the file has none. In llama.cpp 0.5.0 the Jinja engine that runs those templates is on by default (`--jinja`). Override the template with `--chat-template` (a built-in name) or `--chat-template-file`, and use `POST /apply-template` to see the exact prompt the chat route would send.
code
bash · 8 lines# what the chat route would actually send to the model
curl -s http://127.0.0.1:8080/apply-template \
-H "Content-Type: application/json" \
-d '{"messages":[{"role":"user","content":"Summarise this ticket"}]}' \
| jq -r .prompt
# which template llama-server loaded from the GGUF
curl -s http://127.0.0.1:8080/props | jq -r .chat_template | head -20go deeper
Recall that /completion takes a raw prompt while the /v1 chat routes take messages and apply the model's template.
Explain where the template comes from, the ChatML fallback, what --jinja changes, and how /apply-template exposes the rendered prompt.
Diagnose garbage replies and text-only tool calls as template problems, and fix them with --chat-template-file rather than prompt hacks.
Decide which route a team standardises on and how template overrides are versioned with the model files they belong to.
## Two kinds of route on one server llama-server exposes the same model through several HTTP routes, and they differ in one decisive way: whether the server builds the prompt for you. | Route | Request shape | Chat template applied? | |---|---|---| | `POST /completion` | llama.cpp native, a `prompt` string or token array | no | | `POST /v1/completions` | OpenAI legacy completions, a `prompt` | no | | `POST /v1/chat/completions` | OpenAI chat, a `messages` array | yes | | `POST /v1/messages` | Anthropic Messages, `messages` plus `system` | yes | | `POST /v1/responses` | OpenAI Responses, converted to a chat request | yes | The fields each hosted contract defines belong to those contracts; what matters here is that the chat-shaped routes go through a **chat template** and the prompt-shaped ones do not. ## What /completion does with your prompt `/completion` is documented as **not** OpenAI-compatible. It takes `prompt` as a string, a token array or a mix, inserts a BOS token at the start only when the prompt begins with a string and the model's `tokenizer.ggml.add_bos_token` metadata is true, and generates. It adds no system or user markers and no end-of-turn token. A chat-tuned model reading "Summarise this ticket" with no turn structure sees the start of a document, so it may write the next paragraph, invent a second question, or keep going until it hits a limit. The route is still useful, because it exposes llama.cpp's own fields such as `n_predict`, `cache_prompt`, `id_slot` and `grammar`, but you have to bring the formatting yourself. ## How the chat routes build the prompt For `/v1/chat/completions` and `/v1/messages`, the server renders the message list into one prompt string before tokenizing it: 1. By default it uses the **template embedded in the GGUF's metadata**; `GET /props` returns it as `chat_template`. 2. If the file carries no template, it falls back to **ChatML**. 3. With `--jinja` (on by default in llama.cpp 0.5.0) the template runs through the Jinja engine, the same template language model authors ship. 4. With `--no-jinja`, only the commonly used built-in templates are accepted. How a given model family lays out its turns and special tokens is that family's template material; the server's job is to apply whichever template it was given. ## Overriding the template - `--chat-template NAME` picks a built-in template by name, for example `chatml` or `llama3`. - `--chat-template-file FILE` loads a custom Jinja template, the usual fix when a community GGUF ships a broken or missing one. - `--chat-template-kwargs '{...}'` passes extra variables to the template, and a request may add its own `chat_template_kwargs`. - If the embedded template fails to parse, the server stops at startup and suggests `--no-jinja --chat-template chatml` as a fallback. ## Tool calls and the Anthropic route Tool calling depends on the Jinja path: function calling in llama-server is documented as working when it is started with `--jinja`, and `tools` on `/v1/messages` requires it. When the embedded template is not recognised as a native tool-call format, the server uses a generic format, reports the chat format as `Generic` in its log, and may spend more tokens; supplying the model's proper template with `--chat-template-file` usually restores the native one. `/v1/messages` also accepts the `x-api-key` header, so Anthropic-style clients authenticate without changes. ## Mistakes that look like a bad model - **Pointing a chat client at `/completion`.** The client's message arrives with no turn structure, and the model's continuation is mistaken for poor quality. - **Building the template by hand in the client.** It drifts from the template in the GGUF the first time the model file is replaced; let the chat route render it. - **Overriding with the wrong built-in name.** `--chat-template chatml` on a model trained on another format can produce replies that wander or run on, because the turn markers differ from the ones the model was trained on. - **Disabling Jinja to silence a parse error.** `--no-jinja` gets the server to start, but tool calling and template-driven reasoning go with it; a corrected `--chat-template-file` is usually the better fix. ## Debugging a prompt that comes out wrong 1. `POST /apply-template` with your `messages` returns the rendered `prompt` without running inference. Read it: are the role markers and the final assistant header there? 2. `GET /props` shows which template the server loaded. 3. If you need `/completion` for its native fields, render with `/apply-template`, edit the string if you want to prefill the reply, and send that string as `prompt`.
- A model's tool calls come back as plain text in content through llama-server: what do you check?First that the server was not started with `--no-jinja`, since tool calling runs through the Jinja path. Then the startup log: a chat format reported as `Generic` means the embedded template was not recognised as a native tool-call format, so a generic fallback is in use. Supplying the model's proper template with `--chat-template-file` usually restores native calls.
- When is llama-server's /completion the better route despite the missing template?When you need exact control of the prompt: prefilling the start of the assistant's reply, running a base model that has no chat template, or using native fields such as `id_slot`, `cache_prompt` or `n_probs`. Render the conversation with `/apply-template`, edit the string, and send it as `prompt`. Code infilling has its own `/infill` route.
- /completion applies the model's chat template just like the chat route does.
- --jinja is off by default, so templates must be enabled first.
- Every model needs its template passed with --chat-template at startup.
- A GGUF without an embedded template makes the chat route fail.
- Tool calls work the same whether or not Jinja is enabled.
Assumes llama.cpp 0.5.0, where llama-server's --jinja is enabled by default and tool use on /v1/messages requires it.