Self-hosted Qwen returns tool calls as plain text in vLLM — which flags fix it?

## The gap between the model and the API contract The OpenAI-shaped API says an assistant message carries a structured `tool_calls` array. Open-weight models do not produce structured fields — they produce tokens. Everything between those two facts is the serving layer's job, and it is not automatic. Qwen's instruct models are trained in the Hermes-style convention: when the model decides to call a function, it writes, inside the ordinary assistant turn, something like <tool_call> {"name": "get_weather", "arguments": {"city": "Lima"}} </tool_call> and results come back into the conversation rendered by the template into its own tool-response tags. If the server does no extraction, that text is simply the assistant's content. Your client sees `tool_calls: null`, `finish_reason: "stop"`, and a body of text containing angle-bracket tags — which is precisely the bug report "tool calling doesn't work on my self-hosted Qwen". ## The two flags vllm serve Qwen/Qwen3-8B \ --enable-auto-tool-choice \ --tool-call-parser hermes `--enable-auto-tool-choice` allows the model to decide for itself whether to call a tool — the OpenAI `tool_choice: "auto"` behaviour — rather than rejecting or ignoring the request's tool settings. `--tool-call-parser` names the extraction strategy: which surface syntax to look for and how to map it into the response fields. `hermes` is the parser matching the `<tool_call>`-tag convention that Qwen's own template uses. Neither flag alone is enough; they are enabled as a pair. With them set, the server does four things: strips the tags out of `content`, emits `message.tool_calls[]` with a generated `id`, `function.name` and `function.arguments` as a JSON *string*, sets `finish_reason` to `tool_calls`, and applies the same extraction to streaming deltas so partial arguments accumulate the way OpenAI clients expect. ## The template half of the problem Parsing is only the response side. The request side needs the `tools` array rendered into the prompt so the model knows what it may call. That is the chat template's job, and Qwen's shipped template handles it — the tool schemas go into the system turn and `role: "tool"` messages are rendered back into the conversation. When a checkpoint's template lacks tool support, serving frameworks ship replacement Jinja templates you point at with a `--chat-template` argument. The failure signature here differs from the parsing one: the model never attempts a call at all, because it was never told the tools exist. ## Variant-specific parsers A parser is tied to a surface syntax, not to a vendor. Qwen's coder-specialised releases were trained to emit function calls in a different, more XML-ish encoding than the Hermes JSON-in-tags form, so pointing the Hermes parser at them yields either nothing extracted or malformed arguments. Those checkpoints need the parser that matches their own format. The general rule when self-hosting: check the model card for the tool-call format, then select the parser named for it — and validate with one real round trip rather than assuming. ## Debugging it in production The diagnosis path is short because the failure modes are distinguishable: - `content` contains tag syntax and `tool_calls` is null → parser not enabled or wrong parser for this checkpoint. - Model answers in prose and never attempts a call → tools were not rendered into the prompt (template gap), or the request omitted `tools`. - Arguments arrive as invalid JSON, or truncated → often a `max_tokens` cut mid-call, or a small model genuinely producing malformed JSON; validate arguments before executing and be ready to re-prompt. - Works non-streaming, breaks streaming → an incremental-parsing edge case; test both paths before shipping. ## Why this bites people Because the API is OpenAI-shaped, teams assume the semantics are complete out of the box. They are not: tool calling on a self-hosted open-weight model is an agreement between three independently configurable things — the model's trained syntax, the chat template that advertises the tools, and the server-side parser that recovers structure from text. Any one of them mismatched degrades silently into plain text rather than raising an error, which is why an end-to-end tool round trip belongs in your deployment smoke test.