A Postman jsonSchema assertion is green on every run. What has it actually proved about the reply?
answer
- It compared a reply against your own copy
- Shape passing is not the same as correct
- Nothing links the copy to a published definition
- A widened schema still reports green
basics
~20 sA green Postman jsonSchema assertion proves only that the reply satisfied the schema object the script itself supplied. That schema is a copy held on your side, so a pass says nothing about the provider's published definition.
solid answer
~40 sIt proves one thing: the parsed body satisfied the schema object the script passed in, on that run, as judged by the validator behind the assertion. The schema is a **copy** — the collection format has no field pointing at a payload definition, so whatever the script built is what you validated against, and nothing keeps it in step with what the provider publishes. Three claims it never makes: that values are *correct* rather than merely well-shaped; that any counterparty agreed to this shape; and that the shape has not changed since the last release. The dangerous outcome is not a red build but a green one that means almost nothing, because the copy was quietly widened every time it complained. Review the schema, not just the presence of the assertion.
go deeper
Recall the narrow claim: a pass means the reply matched the schema the script supplied. It does not mean the values were right, and it does not compare anything against a definition someone else publishes.
Explain why the schema is a copy rather than a reference, and walk through how a provider change can pass straight through a green assertion when the copy was never updated.
Demonstrate the operating discipline: one copy per payload, an owner and a refresh path for it, loosening treated as a reviewed decision, and reporting that says which document was validated against.
Own the instrument choice. Decide when a runtime shape assertion is enough, when agreement with a counterparty needs contract tooling, and when change detection needs a gate over the published specification.
## What the assertion actually asserted A green `jsonSchema` assertion in a Postman test script carries one guarantee, stated precisely: **the value under test satisfied the schema object that the script handed in, as judged by Ajv, on that run.** Every broader claim people attach to it is added by the reader. Three separate things sit behind that sentence, and a senior answer separates them: | The thing | Where it comes from | What can silently change it | |---|---|---| | the value | the reply body, auto-parsed by the response chain | the service | | the schema | a JavaScript object built inside the script | whoever edits the collection | | the verdict | Ajv, configured by the optional `ajvOptions` argument | the options passed at the call site | ## The schema is a copy, not a reference The collection format has no field that points at a payload schema. `info.schema` names the collection format the file is written in — it is metadata about the file, not about any reply. So the document `jsonSchema()` validates against is whatever the script produced: a literal, or a value read out of a variable, or something fetched during the run. In every case it is a **copy**, held on your side, with nothing keeping it in step with the definition the provider publishes. That has consequences a long-lived collection will eventually meet: - the provider tightens a type and your copy still permits the old one; the gate stays green through a real change; - the provider adds a required field; unless your copy says so, the assertion is indifferent to its absence; - somebody widens the copy to unblock a red build, and the loosening is now permanent and invisible; - eight requests carry eight pasted copies, and only three get updated. ## What the check cannot see, by construction 1. **Correctness of values.** Shape is not truth. A `total` of `-1` passes a schema that says `number`. A schema check never says the answer was right, only that it was the right *sort* of thing. 2. **Agreement with a counterparty.** Nobody on the provider's side ever saw your schema. Consumer-driven contract tooling exists precisely to make that agreement real; a runtime shape assertion is a different instrument and does not substitute for it. 3. **Change across releases.** Comparing this version of a published specification with the last one, and reporting what broke, is the job of a specification-diff gate. Your assertion sees one reply at one moment. 4. **Anything the schema does not mention.** A field the schema is silent about is a field the check has no opinion about. ## The judgment to demonstrate The useful move in an interview is to say what the assertion *is for* and then say what you put beside it: - treat the in-script schema as **test data that goes stale**, and give it an owner and a refresh path, the same way you would any other fixture; - keep one copy per payload rather than one per request, so drift is a single edit rather than a hunt; - when you tighten or widen the schema, do it deliberately and in review — a widened schema is a weakened gate, and it looks identical to a passing build; - be explicit in reporting: "the reply matched our copy of the shape" is the true statement, and it is weaker than "the reply matched the published definition"; - reach for the assertion where it is strong — catching a whole class of shape regressions cheaply, in the same run that already makes the call — and reach for something else where it is not. ## The failure mode to name The dangerous outcome is not a red build; it is a green one that nobody trusts to mean anything. That happens when the schema is written so loosely that it cannot fail — an object type and nothing else, or a copy that was pruned every time it complained. At that point the assertion still runs, still reports green, and still costs review attention while asserting almost nothing. Reviewing the schema itself, not just the presence of the assertion, is the discipline that keeps the check honest.
- A red schema assertion is unblocked by loosening the schema. Why is that worse than it looks?Because the build goes green and stays green. A loosened schema is a permanently weakened gate that is indistinguishable, in every report afterwards, from one that is genuinely passing. Loosening should be a reviewed decision with a reason attached, not the fastest way to clear a build.
- What can a runtime schema assertion never tell you that a specification-diff gate is meant to?Whether the shape changed between releases. The assertion sees one reply at one moment against one document you were holding. Reporting what broke across two versions of a published specification is a different instrument's job, and the two answer different questions.
saying these in an interview costs you the question
- Treats a green schema check as contract verification
- Believes shape validation also confirms the values are right
- Widens the schema to unblock a build without review
- Assumes the copy in the script tracks the provider automatically
- Counts assertion presence rather than reviewing the schema itself
- Reads extra fields in the reply as an automatic failure