How does TGI's grammar parameter force a response to match a JSON schema?

## What the feature is Most applications that call an LLM want a data structure back, not prose. The naive approach - asking politely in the prompt and parsing with a retry loop - fails a few percent of the time, and those failures cluster exactly where you least want them: long outputs, unusual inputs, models under quantization. TGI's guidance feature removes the failure mode at the decoding layer instead. You pass a `grammar` object in the request's `parameters`: - `{"type": "json", "value": { ...JSON Schema... }}` constrains output to that schema. - `{"type": "regex", "value": "..."}` constrains output to a regular expression, which is the right tool for narrow formats - a date, an enum, a phone number, a single classification label. ## The mechanism: masking, not retrying The important mental model is that this happens *inside* generation, once per token, not after generation as validation. 1. The schema or regex is compiled into a finite state machine over the model's tokenizer vocabulary. 2. At each step the server knows the current state - say, 'we have emitted `{"name": "` and are inside a string value'. 3. It computes the set of vocabulary tokens that could legally extend from that state and sets the logits of every other token to negative infinity. 4. Sampling proceeds normally over what remains. Because an illegal token can never be selected, there is no such thing as a malformed response to recover from. Contrast the two alternatives candidates usually propose: prompt-and-hope (fails stochastically), and generate-then-validate-then-retry (doubles latency and cost on every failure, and can loop). Masking pays a small constant cost per step and cannot fail structurally. ## What it does *not* guarantee This distinction separates a middle answer from a strong one. - **Types and shape: guaranteed.** Required fields will be present, an integer field will hold an integer, an enum field will hold one of its members. - **Truth: not guaranteed.** A schema demanding `"invoice_total": number` will get a number even when the document had no total. The constraint has removed the model's ability to say so. - **Quality can drop.** If the schema fights the model's natural output order or forces fields it has no evidence for, you are pushing probability mass toward tokens the model considers unlikely. The mitigation is designing schemas that let the model opt out: nullable fields, an explicit `"unknown"` enum member, a `confidence` field, or a top-level union between a result object and an error object. ## Costs and operational notes Compiling a schema into its state machine is CPU work on the router side. A workload that sends a handful of stable schemas amortises this quickly; one that generates a fresh schema per request pays it repeatedly, and pathological schemas - deep nesting, many alternations, unbounded string patterns - make both compilation and the per-step legality check more expensive. Keep schemas as flat and as bounded as the task allows. Also remember the constraint is per-request, not per-server: two clients hitting the same TGI deployment can use different grammars, and requests without a `grammar` field are unaffected. And a grammar interacts with your token budget - a verbose schema with long field names spends output tokens on structure, which is real money at scale and real latency for the user. ## What interviewers listen for The strong answer names the layer: *token masking during decoding*, not post-hoc validation. Then it draws the structural-versus-semantic line without prompting, and closes with schema design as the mitigation for forced-field hallucination. Candidates who describe it as 'the server retries until the JSON is valid' have the wrong mechanism entirely, and that misconception predicts bad decisions about latency and cost.