Output Formatting

## What a format instruction actually is When you write "Respond with a single JSON object and nothing else", you have added tokens to the prompt. That shifts the model's probability distribution toward JSON-shaped output; it removes nothing from the vocabulary. Every production failure follows from that fact. The model remains free at each decoding step to emit a friendly preamble, to wrap the object in a ```json fence because that is how JSON appears throughout its training data, to append "Let me know if you'd like me to adjust anything", to emit a trailing comma, or to hit the output-token cap halfway through the object. Under easy inputs it complies almost always — which is exactly the trap. A one-percent failure rate is invisible when you eyeball ten outputs and constant when a menu-ingestion pipeline processes thousands of restaurants a night. ## Level 1 — instruction wording This is where most people start, and it does help. Effective wording is concrete rather than emphatic: give the literal shape you want (an example object with the exact keys), name the field types, say explicitly "no markdown fences, no commentary before or after", and specify what to emit when a value is absent (`null`, not omission). Wording alone is the only lever available when you have no control over the serving stack, and it is worth doing well even when you do — the model still has to choose values, and a clear shape spec reduces semantic mistakes as well as syntactic ones. But it is probabilistic. Two things reliably degrade it: long inputs that push the instruction far from the generation point, and unusual inputs that make the model want to explain itself. ## Level 2 — schema-based structured output Most providers expose a mode where you supply a JSON Schema (sometimes via the same mechanism used for tool arguments) and the response is constrained to conform. Practically this eliminates the fence, the preamble and the malformed-object class of failures in one move, because the surrounding prose is simply not part of what the model is allowed to produce. It also documents the contract in a machine-readable artifact your validator can reuse, so the prompt and the parser cannot drift apart. Constraints are real: schemas typically support a subset of JSON Schema (deeply recursive or exotic constructs are often unsupported), the schema costs tokens on every request, and enumerating a large enum inflates it further. ## Level 3 — grammar-constrained decoding The strongest level operates on the sampler. At every step the decoder computes which tokens could still continue a string in the target language — described by a grammar, a regular expression or a compiled schema — and masks the logits of all the others to negative infinity. Invalid output is then not unlikely, it is unreachable. This is how local runtimes and inference servers implement structured generation, and it generalizes beyond JSON: you can constrain to a CSV row shape, a specific enum, a date format, or a tiny DSL. The costs are that you need control over decoding (so it is available on self-hosted or feature-supporting stacks, not through every hosted endpoint), grammar compilation adds startup overhead, and an over-tight grammar can force the model down a path it would not have chosen, which shows up as worse content rather than worse syntax. ## What none of the levels buy you All three control *shape*. None controls *truth*. A grammar-constrained extractor will happily emit `{"dish_name": "Pad Thai", "price_cents": 1450, "allergens": ["peanuts"]}` for a menu that never listed a price and lists shellfish too. Worse, forcing a required non-nullable field pressures the model into inventing a value, because the alternative — saying nothing — is not representable. So structured output raises two obligations rather than removing them: make absence expressible in the schema (nullable fields, an explicit `not_stated` status), and keep validating semantically after parsing, since only your code knows that a price of 1450 cents is implausible for that dish. ## The practical recipe Pick the strongest level your stack supports. Define the schema once and share it between the request and the validator. Make every genuinely optional value nullable, and provide an explicit escape hatch for "unknown" so the model never has to fabricate. Parse defensively anyway — a tolerant pre-parse that strips a stray fence is cheap insurance when you are on wording-only control. Validate, fail closed on violations, and track the parse-failure and validation-failure rates as first-class metrics: they are your early warning that a model upgrade, a prompt edit or an unusual input distribution has broken the contract.