Your data-access suite is green while data-access defects keep reaching production. How do you decide which realism to buy back into the suite?

When data-access defects keep escaping a green suite, the instinct is to add tests. The better move is to work out *which property* the suite lacks, price each candidate, and buy deliberately — because every unit of realism is paid for in feedback time, and a suite slow enough to be skipped protects nothing. ## Step 1: classify the escapes, do not guess Take the last several data-access incidents and label each one with the reason the suite could not have caught it: - **No boundary was crossed** — the test kept one unit of work open, so nothing was detached and nothing re-read. - **No volume** — the defect's symptom is proportional to rows, and the fixture had three. - **No second observer** — the assertion read back through the same tracked set that wrote the state. - **No schema fidelity** — the mapping was only ever exercised against a schema the tests themselves produced, not the one that is deployed. - **Not reproducible in-process at all** — competing writers, real data distribution, plan changes as a table grows. The distribution is usually lopsided, and it is rarely the class people assume. Two escapes from the same class justify a structural change; one escape from a class nothing else has ever hit does not. ## Step 2: price each property | Property to buy | What it costs | What it buys | |---|---|---| | Boundary crossing as the default test shape | Small: setup cleanup and a little runtime | Detachment, re-read, commit-time behaviour | | A second observer on key fields | Small: verbose assertions | Engine-supplied values, symmetric mapping errors | | Volume-shaped tests on chosen paths | Large per test: seeding and runtime | Scaling of the work a path does | | Mapping validated against the deployed schema | Moderate: build wiring | Drift between mapping and the real schema | | Concurrency scenarios | Large and often flaky | Lost updates, contention behaviour | The first two are close to free and belong in the default convention. The last two are targeted purchases with named owners, not blanket policy. ## Step 3: buy depth on representative paths, not breadth everywhere One read path exercised fully — realistic rows, boundary crossed, statements counted, state re-read outside — teaches more than every path being slightly more realistic. Pick the paths by exposure: the ones on the hottest request, the ones whose data grows without bound, the ones where a hand-off across a boundary actually happens. Uniform realism is how a suite becomes slow without becoming informative. ## Step 4: name what you are not buying, and cover it elsewhere State the classes the suite will not attempt, and where the signal comes from instead: 1. **Real data distribution and plan behaviour** — from observing the running system, not from synthetic rows that are uniform and freshly written. 2. **True concurrency under load** — from a load exercise, or from the production error signal on version conflicts and lock waits. 3. **Slow growth effects** — from trend monitoring on table size and operation latency. Writing this down matters as much as the tests: an unstated boundary means every escape triggers an argument about whether the suite should have caught it, and the argument is usually settled by adding another slow test. ## Step 5: protect the feedback loop as a first-class constraint Set an explicit budget for how long the suite may take and treat it like any other resource. When a new realistic test is added, something either fits in the budget or displaces something. Practical levers: seed volume with direct bulk statements rather than through the mapper's write path; keep the volume tier separate from the fast tier so the fast one still runs on every change; and delete tests whose class of defect is now structurally impossible rather than accumulating them. ## The judgment being assessed An interviewer asking this wants to see three things: that you reason from the actual escape history rather than from a checklist; that you can price a testing property in feedback time and spend accordingly; and that you are willing to say a class of defect will not be caught before production and to name the signal that will catch it there. Answering "add integration tests" without saying which property they add, on which paths, and at what cost to the loop, is the answer that fails.