Why must a harness capture failure evidence during the failing run rather than let an engineer reproduce it?

## A failure belongs to one execution, not to the case When a case fails, what failed is not "the case" in the abstract. One **execution** of it failed: at one instant, against one state of the product, on one machine, with one set of data. Everything that would explain the failure lives inside that execution — the values in scope, the interface as it stood, the responses the system returned, the elapsed times, the identifiers that tie the case to the system's own records. The moment the case finishes, most of that starts to disappear. Teardown rolls back or discards the data the case created. The session that was operating the application is closed and whatever it held is released. The parallel test worker takes the next case and reuses its slot. On a disposable executor the whole working directory goes away with the machine that had it. The report a human eventually reads is written after all of that has happened. So the harness gets exactly one chance to preserve the evidence, and it is inside the failing run. ## What a re-run actually gives you The instinct to "just run it again and watch" treats a re-run as a replay. It is not one; it is a second, independent execution that happens to use the same instructions. | What you hope a re-run reproduces | What actually changes | |---|---| | The same data | New records, new identifiers, a store that has moved on | | The same timing | Different load, different scheduling, different latency | | The same build | Possibly a redeployed system with different configuration | | The same target | A different deployed instance, or a local one that does not match it | | The same outcome | An intermittent failure may simply pass | That last row is the decisive one. A failure that appears in one run of forty is the failure you most need evidence for, and it is exactly the one a re-run will not hand you. A suite whose diagnosis strategy is "run it again" can only diagnose its deterministic failures — the ones that were easy anyway. ## What the harness captures on the failing path 1. **The comparison itself.** The value the case expected, the value that arrived, and the path at which they diverged. Not a bare statement that two things differed. 2. **The interaction log.** The ordered steps the case performed, each with its own outcome and its own elapsed time, so a step that quietly failed earlier is visible. 3. **A snapshot of observable state.** A rendering of the interface, a dump of the relevant process, or a serialisation of the responses — whichever matches what the case was driving. 4. **The identifiers.** Whatever lets the reader find the system's own record of the same work rather than searching by time. 5. **The circumstances.** The clock and time zone, the target the case was pointed at, the worker that ran it, and how far into the run it happened. ## Where the capture is wired, and why that is the design decision Capture belongs to the **harness**, not to the case. A harness lifecycle step that runs after every case and inspects the outcome sees every failure in the suite. Capture written inside individual cases covers only the cases whose author remembered to add it — and the cases that fail undiagnosably are disproportionately the ones nobody has looked at recently. Making capture something every case inherits also makes it uniform: every failure produces the same bundle in the same shape, so a reader does not learn a new format per case. Two constraints on that code, and both are learned the hard way: - **It must not run before the outcome is known.** Gathering evidence for a case that is about to pass is pure cost, paid on every green case of every run. - **It must be defensive.** Capture runs when the system is already unhealthy: the target may be unresponsive, the interface half-drawn, the process out of memory. Each capture step needs its own error boundary and its own deadline, so a slow or failing capture degrades the bundle instead of hanging the worker — and never replaces the case's real failure with a diagnostic error of its own. A third, subtler constraint: capture should read from what the case already accumulated rather than driving the application again. Asking the product for more information after a timing failure can perturb exactly the timing you are trying to explain. ## The test of whether it works The criterion is behavioural, not architectural. Can somebody who was not present, who does not have access to that environment, and who did not write the case, say what went wrong from the bundle alone? If the honest answer is "I would have to run it", the capture is incomplete however many files it produced. The same criterion guards the opposite failure. A hundred files nobody opens is not diagnosability; it is noise with a storage bill. The goal is not maximum evidence — it is enough evidence, arranged so the first thing the reader sees is the thing that broke.