A failing test saved the generative step's input and output but nothing about the call itself. What can an investigation not conclude?

## Input and output describe the ends, not the call An investigator arriving at a failed test of a feature that answers from a generative step has exactly one question: what actually happened? The input and the output describe the two ends of that attempt. They say nothing about the attempt itself. Four facts do, and each of them can change underneath a team without anybody editing a line of the product's own logic. 1. **The material supplied to the generative step** — whatever the product retrieved, assembled or attached on that attempt. It is assembled at run time, so it is different on every attempt by construction. 2. **The instruction text and its version.** The wording is an artefact of the product like any other, and it is edited far more often than product code because editing it is cheap and feels safe. 3. **The model identifier.** Which model answered, at which version. A product that follows a moving alias is being changed by someone else on a schedule it does not control. 4. **The generation settings.** How much variation is permitted, any ceiling on output length, any stop condition. These decide whether two attempts are even comparable. ## What each absence costs | Fact missing from the record | What the investigation can no longer conclude | |---|---| | The supplied material | Whether the step answered badly, or answered reasonably from material the product assembled wrongly | | The instruction text version | Which wording produced the failure, so no later edit can be credited with fixing it and no rollback can be aimed | | The model identifier | Whether the release was even exercising the model it ships against, or whether the failure predates a change of model | | The generation settings | Whether a repeat is the same call at all, so a non-reproduction proves nothing | Read that table the other way and it becomes a design rule: **record whatever, if absent, would let someone answer "we cannot tell" and close the investigation.** ## The failure mode this prevents The characteristic bad outcome is not a wrong conclusion. It is an **unfalsifiable** one. Somebody says the generative step was having a bad day; nobody can show otherwise, because the evidence does not contain the facts that would contradict it. The failure is closed as unreproducible, the same shape of failure returns three weeks later, and the second investigation starts from the same standing start as the first. The second bad outcome is a **false all-clear**. A wording is edited, the test passes, and the change is credited with the fix. If the record never said which wording ran when it failed, the pass is equally consistent with the failing wording having been fine and the attempt simply having gone the other way. You have not fixed anything; you have observed a different sample. ## Recording them where they cannot fall out of step Facts recorded from a second source drift away from the facts that were used. Two habits keep them honest: - **Read the value from the same object the call used, at call time.** Take the instruction version from the artefact that was passed to the step, not from a configuration file read separately by the reporting code. Take the model identifier from what the call was actually made with, and where the response reports back which model served it, record that too and record the mismatch when they differ. - **Make the record fail loudly when a fact is absent.** An evidence writer that quietly omits an unset field produces records that look complete. Writing an explicit absent marker, or refusing to accept an evidence record with a hole in it, converts a silent gap into a build-time complaint while somebody can still fix it. ## What this still does not settle Recording the four facts makes an investigation possible; it does not perform it. The record will not tell you whether the failure is a defect in the product or a limit of what the feature can do, and it will not tell you whether the case should be repeated or the assertion loosened. Those are separate decisions, taken by people, with this material in front of them. What the record guarantees is narrower and more useful than a verdict: that when those decisions are taken, they are taken about a specific call that specifically happened, rather than about a reconstruction assembled from memory a fortnight later.