What can still go wrong when constrained decoding guarantees schema-valid output?

## Why this question is asked Once a team turns on a strict schema mode, parse errors vanish from the dashboards and the pipeline looks solved. The interviewer wants to know whether you understand what the guarantee actually covers, because the failures that remain are quieter and more damaging than a `JSONDecodeError`. ## Failure 1 — valid and wrong A constraint is a statement about the *language* of the output. It says the record has a `solvent`, that it is one of 34 approved names, and that `yield_percent` is a number. It says nothing about whether that solvent is the one the abstract described, or whether the number belongs to the reaction in question rather than the one in the sentence before. So the defect class shifts from malformed to plausible-but-false: yields above 100, temperatures below absolute zero, a solvent silently copied from the previous document because the current one omitted it, a field filled from the paper's introduction rather than its results. None of these trip a schema check. Range checks, cross-field consistency, referential checks against your own reference tables, and a sampled human audit are the controls that catch them, and they belong above the decoder rather than inside it. ## Failure 2 — truncation at the token ceiling This is the edge case interviewers most like, because it is where people's mental model breaks. Suppose a grammar-constrained extraction over a dense abstract runs long and hits the maximum-token limit while the object is half built. The constraint has been honoured at every single step: every token emitted was a legal continuation. But the automaton never reached an accepting state, so the output is a **legal prefix**, not a legal object — `{"solvent": "THF", "temperature": "65 °C", "yield_pe` and nothing more. The important consequences: - The constraint cannot save you. It can forbid the end-of-sequence token until the object closes, but it cannot conjure the remaining budget; when the ceiling wins, generation stops regardless. - **Detect it from the finish reason, not the parser.** A length-capped stop is a distinct signal from a validation failure, and conflating them sends a truncated record into a repair path where it does not belong. - **Never synthesise the closing braces.** Appending `"}` makes the object parse and quietly invents a value — you have manufactured a data error that now looks clean. - Mitigate at design time: bound array lengths in the schema, keep the record flat and the key names short, reserve headroom between the expected output size and the ceiling, split a large extraction into several smaller constrained calls, and alert on the truncation rate as a first-class metric. A related trap is a schema so large that the *keys themselves* dominate the budget: a 200-key record spends a substantial share of its tokens on field names, leaving less for values than anyone estimated. ## Failure 3 — the abstention path is masked away Masking removes options, and "I don't know" is an option. If the abstract never states a temperature and the schema requires a pattern-constrained string, the decoder must emit something matching that pattern. The model is not lying by choice; the honest answer was unreachable. Every constrained field therefore needs a legitimate out: a nullable type, or an explicit `not_reported` / `unknown` enum member. Then monitor its rate — that rate is one of the few cheap, high-signal quality indicators an extraction pipeline has, and a sudden drop usually means someone tightened a schema. ## Failure 4 — the quality cost of rigidity Masking renormalises the distribution over allowed tokens, and forcing the model directly into a rigid payload denies it the space to work a problem out in prose first. Published work since 2024 has reported degradation on reasoning-heavy tasks under strict formatting, while other results find the effect small when the schema is well shaped; as of mid-2026 this is genuinely unsettled, and the honest interview answer says so rather than picking a side. What is not contested are the mitigations: - put a free-text reasoning or evidence field **before** the structured fields, so the model can think inside the schema — and it doubles as an audit trail, since you can check the quoted evidence against the source, - or split into two passes: unconstrained analysis, then a constrained extraction over that analysis, - and shape the schema for compliance — flat, enum-heavy, no deep nesting or exotic composition. ## The summary an interviewer wants Constrained decoding eliminates one failure class completely and leaves three: semantic error, truncation, and forced fabrication, plus a possible quality tax. Its real value is that it converts a broad, noisy problem into a narrow one you can instrument — and instrumenting it means tracking truncation rate, unknown-value rate, and business-rule violation rate, not just parse success.