skip to content

When an LLM's JSON fails schema validation, how should the calling code respond?

level: middleimportance: must knowfreq 63%

answer

  1. output is untrusted input
  2. never silently coerce a bad value
  3. fail closed at the boundary
  4. quote the validator error back
  5. cap the retries, then fall back

basics

~20 s

Fail closed: reject the response, never coerce or scrape it. Then optionally send one targeted repair prompt quoting the exact validator error, cap retries at one or two, and fall back to a deterministic path or human review while logging the failure rate.

solid answer

~50 s

Treat model output as untrusted input crossing a boundary. Parse it, validate it against the schema — types, required fields, enum membership, cross-field invariants — and on any violation **fail closed**: reject the whole response rather than salvaging the parts that looked fine. The common anti-pattern is silent coercion: a triage classifier returns a sixth tag outside your four-value enum and someone maps it to the nearest allowed value by string similarity, converting a loud failure into permanently wrong data. If repair is worth it, send one *targeted* prompt — the offending output plus the exact validator message plus the allowed values — not the original prompt again. Cap attempts at one or two, because some inputs simply never produce valid output and an unbounded loop burns cost and latency. Beyond the cap, take the deterministic fallback or queue for human review, and emit the failure as a metric.

code

python · 14 lines
python
import json
from jsonschema import Draft202012Validator

SCHEMA = {
    "type": "object",
    "properties": {"tag": {"enum": ["billing", "outage", "howto", "abuse"]}},
    "required": ["tag"],
    "additionalProperties": False,
}

def validate(raw: str) -> dict:
    payload = json.loads(raw)                        # raises on malformed JSON
    Draft202012Validator(SCHEMA).validate(payload)   # raises on a sixth tag
    return payload

go deeper

for a junior

Always parse and validate the model's response before using it, and let an invalid response raise rather than patching it up. Know that valid JSON is not the same as a valid value.

for a middle

Explain the layers of validation — parse, structure, domain rules like enum membership, cross-field invariants — and why failing closed with a capped, targeted repair beats coercion or an unbounded retry loop.

for a senior

Show the operational picture: metrics per failing rule tagged by model and prompt version, a deterministic fallback path, and the judgment that a persistently high repair rate means the schema or prompt is wrong rather than the loop is working.

for a principal

Own the boundary policy across services: where model output is allowed to flow, what the retry and cost budget is, how failures escalate to humans, and the rule that schema conformance never substitutes for encoding and authorization checks downstream.

## The contract at the boundary The string a model returns is data from an untrusted source that happens to be expensive. The discipline is the same one you would apply to a request body from an unknown client: parse it, validate it against a declared schema, and let nothing downstream see it until it has passed. What makes LLM output distinctive is that it is *usually* right, which tempts engineers into treating validation as a formality and into repairing rather than rejecting. ## Validate more than the shape A complete check has four layers: 1. **Parseable** — the string is well-formed JSON at all. 2. **Structural** — required keys present, types correct, no unexpected keys if you declared `additionalProperties: false`. 3. **Domain** — enum membership, numeric ranges, string formats, referential checks against your own data (does this SKU exist?). 4. **Cross-field** — invariants that no single field can violate alone, like `price_status` being `not_stated` while `price_cents` holds a number. Layers 3 and 4 are where the interesting failures live. A classifier constrained to a four-value enum by prompt wording will, on an ambiguous ticket, confidently invent a fifth or sixth label — and the response is valid JSON with a string in the right slot. Only enum validation catches it. ## Fail closed, and what that rules out Failing closed means a violation stops the record. Three tempting alternatives are all worse: - **Coercion.** Mapping the invented tag to the nearest allowed value by edit distance or embedding similarity turns a detectable error into silently wrong data that no metric will ever surface. The model produced a value outside the label space precisely because it did not believe any allowed label fit; overriding that judgment destroys information. - **Partial acceptance.** Keeping the fields that validated and nulling the rest sounds pragmatic, but the invalid field is evidence that the model misread the input, and the "good" fields come from the same misreading. - **Regex scraping.** Pulling values out of malformed text with pattern matching is the least robust parser you can build, and it will succeed just often enough that nobody notices when it starts matching the wrong thing. The correct outcomes are: repair, deterministic fallback, or escalate — all of them explicit, all of them counted. ## Targeted repair, not a blind retry If you repair, make the second call carry information the first did not have. A repair prompt should contain the invalid output, the precise validator message, and the constraint that was breached — "`tag` was `\"escalation\"`; allowed values are billing, outage, howto, abuse; return the corrected object only". That is a different, much easier task than the original, and it succeeds far more often than resending the same prompt and hoping sampling lands differently. Send only the failing part where the payload is large; there is no need to re-extract ninety valid rows to fix one. ## Cap the loop Retry budgets exist because some inputs never yield valid output — a menu image that is unreadable, a ticket that genuinely fits no category, a prompt-schema mismatch you introduced in the last deploy. An unbounded repair loop turns those into unbounded cost and latency, and under load it amplifies exactly when you can least afford it. One or two attempts captures nearly all the recoverable cases. After that, take the fallback path: a deterministic rule, a default with an explicit `needs_review` marker, or a human queue. Note that a repair loop is also a poor fix for a systematic problem — if five percent of records need repair, the schema or the prompt is wrong and the loop is hiding it. ## Instrument it Emit counters for parse failures, validation failures broken down by which rule fired, repair attempts, repair successes and fallbacks taken, tagged with the model version and prompt version. These are your regression detector: a prompt edit or a model upgrade that breaks the contract shows up here first, usually as a spike in one specific rule. Watching only end-to-end success hides the fact that a growing share of your throughput is being carried by retries. ## The security angle Validation is also a containment boundary. Model output may echo content from untrusted sources — a scraped menu, a customer's ticket text — so a field that flows into a shell command, a SQL statement, a file path or a rendered page must be validated and encoded on its own terms, not trusted because it arrived inside well-formed JSON. Schema conformance says the string is in the right slot; it says nothing about whether the string is safe.

  • Why is a targeted repair prompt more effective than simply resending the original prompt?
    Because it changes the task. The repair call carries the invalid output, the exact validator message and the allowed values, so the model is doing a small correction with the error in view rather than re-attempting the original extraction blind. Resending the same prompt relies on sampling noise landing differently, which at low temperature may reproduce the identical failure.
  • Your repair loop succeeds on five percent of records every day. Is that healthy?
    No — that is a systematic defect the loop is masking. A five percent violation rate usually means the schema forbids something real (no way to express "unknown"), the enum is missing a genuine category, or the prompt and schema disagree. Fix the contract; a repair loop should be catching rare accidents, not carrying a routine share of throughput.
  • Is silently mapping an out-of-enum value to the closest allowed one ever acceptable?
    Only when the mapping is an explicit, reviewed business rule with its own audit trail and metric — for example, folding a known synonym into a canonical label. Doing it as a generic string-similarity fallback is not: the model emitted an out-of-space value because it judged that no allowed label fit, and overriding that produces confidently wrong data nobody can find later.

saying these in an interview costs you the question

  • Coercing an out-of-enum value to the nearest allowed label
  • Scraping fields out of malformed output with regexes
  • Retrying in an unbounded loop until something validates
  • Validating only that the response parses as JSON
  • Trusting model output downstream because the schema matched

context

resolving links…