A frontend suite that stubs every network call stayed green through a backend release that broke the app in production. Explain how this mock drift happens and what you would add so the suite fails instead.
answer
- green means agrees with the stub
- the snapshot is never re-taken
- types catch shape, not values
- validate the fixture against the schema
- something must compare schema to reality
basics
~20 sStubs freeze the contract as it was when they were written, and nothing rechecks them, so a fully stubbed suite verifies the app against its own stale assumption. Detection requires deriving stubs from the schema, validating stub payloads against it at test time, and keeping a few real calls.
solid answer
~50 sGreen meant "the app agrees with the stub", not "the app agrees with the API" — a fully mocked suite has no path back to the real contract, so it cannot notice when that contract moves. I close it in layers. Generate fixture types from the published schema so a shape change breaks the build. Validate every stub payload against the schema at test runtime, so a fixture that no longer conforms fails loudly rather than being quietly accepted. Pin the schema in CI and fail when the deployed one differs from the pinned copy, which is what actually detects the drift rather than reacting to it. Then keep a small number of tests that hit a real running instance, because only a real response proves the stub was ever accurate. The mocked suite stays fast; a thin real-network layer keeps it honest.
code
javascript · 12 lines// One test whose only job is to prove the fixtures still conform.
import Ajv from 'ajv';
import userSchema from './schema/user.json' with { type: 'json' };
import { makeUser } from './fixtures';
const validate = new Ajv().compile(userSchema);
test('user fixture conforms to the published schema', () => {
const ok = validate(makeUser());
expect(validate.errors ?? []).toEqual([]);
expect(ok).toBe(true);
});go deeper
Understand that a stubbed test proves the app handles the stub, not that the stub matches the real API. Be able to explain why the suite could stay green while production broke.
Explain the mechanics of each detection layer: generated types catching shape changes at build time, schema validation of fixtures catching runtime violations, and why each one alone is insufficient.
Show that you would design the detection rather than react to the incident — a job that compares the pinned schema against the deployed one, a thin real-network layer, and a clear statement of the drift classes that still slip through.
Own the tradeoff as policy: how much the organisation mocks, what re-verification budget that obliges it to fund, and who is accountable when the published schema and the running service disagree.
## What actually happened Every assertion in that suite was of the form "given this response, the app does X". The response came from a stub written by the frontend team. So the suite proved a conditional: *if* the API returns this, the app behaves. It never evaluated the antecedent. When the backend changed, the antecedent became false and every conclusion became worthless — while every test stayed green, which is worse than failing, because the team believed it. This is mock drift: the stub is a snapshot of the contract taken on the day it was written, and no process re-takes the snapshot. ## Why it is so easy to miss Drift has no symptom inside the suite. There is no error, no slowdown, no warning. It surfaces first in production, and often not immediately — a renamed optional field may only break the branch that shows a subscription banner. Teams also accumulate drift asymmetrically: one stub is corrected during a bug fix while eleven others keep the old shape, so the suite ends up holding several mutually inconsistent versions of the same endpoint. ## Layer one: stop hand-writing the shape Generate types from the OpenAPI or GraphQL schema and annotate fixtures and factories with them, regenerating in CI against the deployed schema and failing on a diff. This catches the largest class of drift — renamed, removed and retyped fields — at build time, and it costs nothing per test. Its ceiling matters: types vanish at runtime, so any stub built dynamically, loaded from a JSON file, or cast escapes the check entirely. ## Layer two: validate stub payloads at test time The cheapest runtime check is to assert that what your stub is about to return is something the schema permits. Convert the schema to a validator once (a JSON Schema validator such as Ajv, or a hand-kept parser schema) and run the fixture through it inside the mock layer or in a dedicated fixture test: ```javascript const valid = validateUser(userFixture); if (!valid) throw new Error(`fixture violates schema: ${JSON.stringify(validateUser.errors)}`); ``` The key design choice is to run this on the fixture, not only on what the app receives — that way it fails in a fixture test with a clear message rather than as a mystery component failure. This catches what types cannot: fixtures assembled at runtime, values that violate declared formats or enums, and required fields dropped by a careless override. ## Layer three: detect that the contract moved, not that your fixture is bad Both layers above compare the fixture to a *copy* of the schema. If the copy is stale, both pass. So one job must compare the copy to reality: fetch the deployed service's schema on a schedule and in CI, diff it against the pinned copy, and fail or open a ticket when they differ. That is the step that turns drift from a production discovery into a build notification. ## Layer four: keep a few real calls Schemas describe intent; services have behaviour. A small number of tests that exercise a real running instance — a smoke path against a deployed environment, or component tests run once against a real backend in a nightly job — are the only thing that proves the service honours its own spec. They are slow and less reliable, which is exactly why there should be few of them, and why the mocked suite carries the coverage. ## What still slips through Be honest about the residue, because a strong candidate names it: - **Additive change.** The API starts sending a new field the app should handle. No stub is invalid; nothing fails; the feature is simply absent. - **Semantic change with a stable shape.** A `status` string keeps its type but changes meaning, or a currency amount switches from cents to units. - **A spec that lies.** If the schema is hand-maintained and the implementation drifted from it, every layer here validates against fiction. - **Behaviour, not payload.** Ordering, pagination boundaries and idempotency are contract too, and no fixture check covers them. ## How to frame the answer The framing that lands is: mocking buys speed and determinism by cutting the connection to the real system, so anything you mock, you must independently re-verify. The cost of a fast suite is a drift-detection budget, and a team that pays it can mock aggressively without lying to itself.
- How do you validate stubs against a schema without turning every test into an integration test?Compile the schema into a validator once and run the fixtures through it in a single dedicated conformance test, or inside the mock layer where responses are constructed. It stays in-process and adds milliseconds; the tests themselves keep stubbing normally and never touch the network.
- Which direction of drift do schema-validated stubs still miss?Additive and semantic change. If the API starts sending a new field, every existing fixture remains valid and nothing fails — the feature is just missing. Likewise a field that keeps its type but changes meaning, such as an amount switching from cents to units, passes every structural check.
- The team's answer to this incident is "write more end-to-end tests". What is wrong with that?It solves the right problem with the wrong instrument. A handful of real-network tests is exactly what proves the contract, but scaling them up trades a fast deterministic suite for a slow flaky one. Keep the mocked suite for coverage and spend the real-network budget on detection, not duplication.
A stub is a photograph of the API taken the day you wrote the test. The suite keeps checking the app against the photograph long after the subject has moved, and photographs never look out of date on their own.
saying these in an interview costs you the question
- Assuming a green mocked suite proves the API contract
- Believing more mocked tests reduce integration risk
- Refreshing fixtures only after a production incident
- Treating the stub as the source of truth for the API
- Claiming end-to-end tests alone make fixture checks unnecessary