Why validate an LLM's JSON output against a strict schema before your code uses it?
answer
- parse is not the same as validate
- the model is an untrusted producer
- types, enums, required fields
- fail loudly at the boundary
- schema-valid can still be semantically wrong
basics
~20 sParsing only proves the text is syntactically JSON. Validation proves the fields your code reads actually exist, with the right types and allowed values — so bad output fails loudly at the boundary instead of corrupting logic downstream.
solid answer
~50 sParsing and validating are two different checks. A JSON parser answers one question: is this text well-formed? It happily accepts an object with a misspelled key, a quantity returned as the string `"12"`, a missing required field, or an invented field nobody asked for. Schema validation answers the question your program actually cares about: does this value have the shape the rest of the code assumes? So you validate once, at the boundary where model output enters the system, and pass a typed object inward — never a raw dictionary. Treat the model as an untrusted producer, because it is non-deterministic: the same prompt can yield a different key spelling or a truncated object on the next call. Validation failure should be an explicit, logged, handled outcome — repair, escalate, or reject — not a `KeyError` three layers deep in business code.
go deeper
Be able to say plainly that parsing checks syntax while validation checks shape, and that you validate model output before using it. Naming a few things a schema enforces — required fields, types, enum values — is enough at this level.
Explain the two layers: structural validation against the schema, then business-rule checks a schema cannot express, such as cross-field constraints and lookups against your own reference data. Be ready to say where in the code the boundary sits.
Show the operational side. Talk about logging the raw output with the validator error, measuring per-field failure rates, and treating validation failure as expected traffic with an explicit handling path rather than an exception nobody catches.
Own the position that the model is an untrusted producer inside your own system, and that validation is a contract you keep regardless of provider capability. Be ready to argue what belongs in the schema versus a separate rules layer, and who owns each when they drift.
## Parsing is not validating When a model returns text you intend to consume programmatically, two independent checks stand between that text and your business logic. The first is **parsing**: turning bytes into an in-memory value. A JSON parser answers exactly one question — is this text syntactically well-formed JSON? Balanced braces, quoted keys, no trailing comma. That is all. A parser will cheerfully hand you `{"quantitiy": "12", "mode": "aeroplane"}` because that text is perfectly valid JSON. It is also useless to a program expecting `quantity` as an integer and `mode` drawn from a fixed set. The second is **validation**: checking the parsed value against a declared schema. This is the check that answers the question your code actually depends on — does this value have the shape the rest of the program assumes? ## What a schema check actually buys A schema language (JSON Schema being the common one in this space) lets you assert, mechanically: - **Presence** — which fields are `required`, so a silently omitted field is caught rather than read as null. - **Types** — `"integer"` versus a stringified number, `"boolean"` versus the word `"yes"`. - **Allowed values** — `enum` membership, so a status field cannot arrive as a synonym the model preferred. - **Format** — `pattern` for identifiers with a fixed shape, numeric `minimum`/`maximum` for ranges. - **Closed shape** — `additionalProperties: false` rejects invented keys instead of letting them pass unnoticed. Each of these turns a class of silent corruption into a loud, localized failure with a precise message naming the offending path. ## Why the model boundary needs this more than an ordinary API boundary You validate input from any external system, but a language model is a harsher producer than most. It is non-deterministic: the same prompt at the same settings can return a different key spelling, a different enum synonym, or a differently nested object across calls. Output can be **truncated mid-object** when it hits a token limit, producing text that is not even parseable. And a model asked to fill a field it has no grounds for will often invent a plausible-looking value rather than omit it — the failure mode that most needs a mechanical check, because it looks entirely reasonable. Even where a provider offers a strict schema mode that constrains generation, validation on your side remains the check you own: it survives a provider without that mode, a self-hosted model, a schema-mode fallback path, and a truncated response. It is the safety net, not the duplicate. ## Schema-valid is still not the same as correct This is the point that separates a rehearsed answer from a real one. A schema constrains **structure**; it cannot constrain **meaning**. Consider a freight forwarder turning a broker's email into a customs declaration. A schema can guarantee that `commodity[2].hs_code` is a six-digit string, that `incoterm` is one of the eleven allowed codes, and that all fourteen required fields are present. It cannot tell you that the HS code is not a real tariff line, that gross weight came back lower than net weight, that the declared value is off by three orders of magnitude, or that the incoterm is incompatible with the transport mode. So production pipelines run two layers: **structural validation** against the schema, then **semantic validation** — cross-field constraints, referential checks against your own reference data, and plausibility bounds. Both layers produce the same kind of artifact: a precise, machine-readable error naming what failed and why. That artifact is what makes an automated repair attempt possible at all. ## Fail closed, at the edge The operational discipline is: validate once, immediately after parsing, and convert to a typed domain object right there. Downstream code should never see an unvalidated map. If validation fails, that is a first-class outcome to handle explicitly — log the raw output alongside the validator error and a request identifier, then choose between a repair attempt, escalation to a human, or a clean rejection. What you must not do is let the raw parsed value flow inward and discover the problem as a type error deep in business logic, where the original output is no longer available to diagnose. ## The cost argument Validation is cheap in the only currency that matters here. Checking an object against a schema takes microseconds against a generation that took seconds and cost real money. There is no throughput argument for skipping it, and the debugging asymmetry is enormous: a validator error names the exact failing path, while an unvalidated bad field surfaces later as a corrupted record with no trace of where it came from.
- If a provider already enforces the schema during generation, why keep your own validator?Because the check you own is the one that survives everything: a model or deployment without strict mode, a fallback path, a truncated response, and — most importantly — the business rules no schema can express. Provider enforcement narrows the structural failures; it does not make a wrong HS code, an impossible weight, or an inconsistent field pair go away. Keeping validation also means your failure handling is identical regardless of which model served the request.
- What do you log when validation fails, and why does it matter?The raw model output verbatim, the full validator error including the failing path and expected-versus-actual, the schema version, the model and settings used, and a correlation id. Without the raw output you cannot tell a schema-design problem from a one-off sampling artifact. Aggregated over time these logs give you a per-field failure rate, which is the signal that tells you to fix the schema rather than lean harder on retries.
- Should validation errors be raised as exceptions or returned as values?Either works, but the boundary must be explicit and total: every call site handles the failure path. Returning a result value tends to be safer here because the failure is expected traffic, not an exceptional condition — a small percentage of calls will fail validation as a matter of course. What matters is that no code path can accidentally proceed with an unvalidated object, which usually means converting to a typed domain object at the boundary and never passing the raw map inward.
Parsing is checking that a form was filled in with a pen rather than smeared; validation is checking that the boxes contain a date, a country code, and a number.
saying these in an interview costs you the question
- Says a successful JSON parse means the output is valid
- Assumes a strict provider mode removes the need to validate
- Reads model output fields directly without any check
- Thinks schema validation catches wrong or implausible values
- Lets validation failures surface as type errors deep in business code