skip to content

A parser reading JSON from llama-server's /v1/chat/completions started failing overnight — which request fields and server settings do you check first?

level: seniorimportance: should knowfreq 6%

answer

  1. how the reply ended
  2. an object is not your schema
  3. where the schema must be nested
  4. a full slot stops generation
  5. thought text left in content

basics

~10 s

Check finish_reason first: length means max_tokens or the slot's context cut the JSON off. Then check response_format: json_object without a schema allows any object, and a json_schema type needs its schema nested in json_schema.schema.

solid answer

~40 s

Start with `finish_reason`. If it is `length`, generation stopped before the object closed: `max_tokens` ran out, or the slot's context filled, and with context shift off by default the server simply stops. Next, read the request's `response_format`. `{"type": "json_object"}` with no schema constrains output only to *some* JSON object, and for `{"type": "json_schema"}` llama-server reads the schema from `json_schema.schema`; put it elsewhere and you are back to any object. Then check what changed on the server overnight: a restart with `--reasoning-format none` leaves a thinking model's thoughts in `message.content`, and a smaller `-c` or a new GGUF changes what fits. For non-JSON formats, send a GBNF string as `grammar`, but never with `json_schema` in the same request, and never alongside tools.

code

json · 21 lines
json
{
  "model": "your-model",
  "max_tokens": 1024,
  "messages": [
    {"role": "user", "content": "Extract the invoice number and total."}
  ],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "invoice",
      "schema": {
        "type": "object",
        "properties": {
          "number": {"type": "string"},
          "total": {"type": "number"}
        },
        "required": ["number", "total"]
      }
    }
  }
}

go deeper

for a junior

Recall that finish_reason length means the reply was cut off, and that json_object alone does not enforce your fields.

for a middle

Explain the response_format shapes llama-server reads, where the schema must sit, and when to use a grammar instead.

for a senior

Trace an overnight break to truncation, a misplaced schema, a reasoning-format change or a smaller slot, and make the client fail loudly.

for a principal

Decide where output validation lives, in the server's constraints, the client's checks or both, and how server flag changes get reviewed.

## Two different failures look the same to a parser When `JSON.parse` throws, the text you received is one of two things: **valid JSON of the wrong shape**, or **JSON that never finished**. llama-server can produce both from a request that worked yesterday, and the response tells you which one before you look at a single brace. How grammar-constrained decoding works, and what it can and cannot guarantee in general, is the structured-output material's subject; this answer is about which llama-server settings decide what you got. ## First check: how the generation ended `finish_reason` on `/v1/chat/completions` is `stop` when the model finished on its own, `tool_calls` when it called a tool, and **`length`** when the server stopped it. A schema-constrained reply that ends with `length` is cut off mid-object, because the constraint shapes each token but cannot make the model finish early. Two limits cause it: 1. **`max_tokens`** in the request (`n_predict` on the native route) ran out. 2. **The slot's context filled.** `--context-shift` is disabled by default, so when prompt plus reply reaches the slot's `n_ctx`, the server stops generating and marks the result truncated. A restart that shrank the context, from a changed `-c`, more `-np` slots or `--fit` choosing a smaller window for a new GGUF, makes long inputs hit this overnight. The response's `usage` and `timings` show how many tokens were used, so you can see how close you were. ## Second check: what the request actually constrained On `/v1/chat/completions`, llama-server reads `response_format` like this: | `response_format` sent | What llama-server enforces | |---|---| | `{"type": "text"}` or absent | nothing | | `{"type": "json_object"}` | any JSON object | | `{"type": "json_object", "schema": {...}}` | that schema | | `{"type": "json_schema", "json_schema": {"schema": {...}}}` | that schema | | `{"type": "json_schema", "schema": {...}}` | any JSON object: the schema is not read | | any other `type` | rejected with an error | The fifth row is the easy one to miss: for the `json_schema` type the server reads the schema only from inside `json_schema`, and an absent or empty schema is treated as "any object". The reply is valid JSON, so nothing errors, but required fields go missing. The same request may also carry a top-level `json_schema` field or a `grammar` field, which are llama.cpp extensions; sending both is rejected with `Cannot use both json_schema and grammar`. ## Third check: what changed on the server - **Reasoning output.** `--reasoning-format` defaults to `auto`. Restarted with `none`, a thinking model's thought text is left in `message.content`, so a parser reading `content` now sees prose before the JSON. - **Server-wide constraints.** `--grammar-file`, `-j` and `-jf` constrain generations for the whole server; a flag added or dropped in a restart changes every client at once. - **Tools.** A request that offers tools cannot also carry a custom `grammar`; the server refuses it with `Cannot use custom grammar constraints with tools.` - **The model file.** A new GGUF can bring a different chat template and trained context, which can change both the output and the slot size. ## When the format is not JSON A **GBNF** grammar, the BNF-like format llama.cpp uses for constraints, covers what a JSON schema cannot: a fixed label set, a CSV line, a small command language. Send it per request as the `grammar` string, or load it for the whole server with `--grammar-file`. An invalid grammar is rejected with `Failed to parse grammar`, and a schema the converter cannot handle is rejected with an error naming `json_schema`, so both fail loudly rather than silently. ## Mistakes behind most overnight breaks - **Copying a `response_format` shape from another server's docs.** The field names look alike across servers; llama-server reads only the shapes in the table above. - **Raising `max_tokens` without checking the slot.** If the slot's context is the limit, a larger `max_tokens` changes nothing. - **Stripping the reply with string hacks.** Removing a leading `<think>` block in the client hides a server flag change instead of fixing it. - **Testing with short inputs only.** Truncation appears with the longest documents, which are the ones that arrive at night in batch jobs. ## Making the next failure loud 1. Treat `finish_reason` `length` as a retry or an error, never as input to the parser. 2. Validate the parsed object against the same schema client-side. 3. Log the server's flags and the GGUF file name with each deployment, so an overnight restart is visible in the diff.

  • When do you send llama-server a GBNF grammar instead of a JSON schema?
    When the output is not JSON: a fixed set of labels, a CSV line or a small command language. Send it as the request's `grammar` string, or load it server-wide with `--grammar-file`. A request cannot carry both `grammar` and `json_schema`, and a custom grammar is refused when the request also defines tools.
  • How do you make a llama-server JSON client fail loudly instead of silently?
    Check `finish_reason` before parsing and treat `length` as a retry or an error. Validate the parsed object against the same schema on the client, because `json_object` without a schema accepts any object. Record the server's flags and GGUF with each deployment, so a restart with a different `-c` or `--reasoning-format` shows up in review.

  • response_format json_object makes llama-server enforce my schema's fields.
  • Schema-constrained output from llama-server can never be cut off mid-object.
  • A grammar and a JSON schema can be combined in one llama-server request.
  • For type json_schema, a schema placed directly under response_format is enforced.
  • --reasoning-format none strips the thinking text out of the reply.