Test Data Seams

Almost every automated case contains two activities that look alike in code and mean completely different things when they fail. Keeping them in separate parts of the case is one of the cheapest structural decisions available, and one of the few whose payoff is visible in every red run afterwards. ## Two kinds of work in one case The first activity is **building the premise**: obtaining the records the case needs so the behaviour under test is even reachable. The second is **exercising and checking**: doing the thing, then verifying what the product did. The premise is not under test. Nobody wrote the case to find out whether an account can be created; they wrote it to find out what happens to an account in a particular state. That distinction is what makes the placement rule more than tidiness. A failure while building the premise is a statement about the suite's supporting machinery or the target it runs against. A failure after the premise is established is a statement about the product. Those two statements have different owners, different urgency and different fixes. ## Why a mixed case cannot be triaged | | Break while provisioning | Break in a check | | --- | --- | --- | | What it means | The case never got to run | The product did something wrong | | Who acts | Whoever owns the seam or the target | Whoever owns the behaviour | | What the run should report | Blocked, or errored | Failed | | What a mixed case reports | Failed | Failed | | Cost of the confusion | Time spent reading product code for a data problem | Real defects diluted in noise | The last row is the expensive one, and it compounds. Once a run reports thirty failures of which most are provisioning trouble, the honest response — read every failure — stops being affordable, and the suite starts being skimmed. Separating the two kinds of work is what keeps a red count meaningful. ## Blocked is a status, not a nicety A result store that carries a blocked status distinct from failed lets a run say something true that a single status cannot: *two cases found defects, thirty could not obtain their records*. That sentence changes what happens next. It routes work correctly, it stops a provisioning outage from reading as a product regression, and it keeps failure trends honest over time — a suite whose blocked count is climbing has an infrastructure problem that a pure pass rate hides. The structure of the case is what makes the status assignable. A run can only classify a failure by where it happened, so the boundary has to exist in the case before the reporting can use it. ## The honest exception Sometimes a record genuinely has to be created partway through, after an earlier check. A case verifying that a second order appears in the history has to create that second order between two assertions. That is not a violation of the rule, because the creation is part of the scenario being exercised rather than a premise being established. The repair is to name it for what it is: an action step, reported as an action. What must not happen is the creation being tucked inside the assertion block for brevity, where a failure will be reported as a failed check on behaviour nobody tested. The test to apply is a single question: **if this call fails, has the product told me anything?** If yes, it is part of the scenario. If no, it is a premise and it belongs in the setup step. ## Making the boundary structural Style guides do not hold this line; structure does. A few habits that make it stick: - **Give the case named sections.** Whatever the harness offers — distinct blocks, distinct methods, a lifecycle step that runs before the case body — the boundary should be visible without reading the calls. - **Let a setup failure raise a different kind of error from an assertion failure.** That is what a run needs in order to assign a status without parsing messages. - **Keep the setup step free of checks.** A verification inside setup reintroduces exactly the ambiguity the split removed. If a premise genuinely needs verifying, let it fail as a setup error, not as an assertion. - **Keep the checking section free of creation.** Anything created there should be an action the case is deliberately taking, named as one. - **Watch the length of the setup step.** A block that grows past a handful of lines usually means the provisioning vocabulary is missing a request the case had to assemble by hand. ## What it buys in the end The gain is not aesthetic. It is that a reader who opens a red run can tell, without opening any code, whether the product is broken or the data was. That reader is usually you, on a Monday, looking at a run from Friday night — and the version of the case that mixed its work is the one you will spend the morning on.