A generated specification report renders scenarios that never executed as documented behaviour. How do you fix it?

### The failure A generated specification report is trusted because readers assume its pages describe what the system did on a real run. That assumption breaks the moment the report renders a scenario the run never executed with the same visual weight as one that passed. The reader sees prose describing behaviour; nothing verified that behaviour; and the document is now confidently asserting something no evidence supports. This is worse than a stale hand-written page, because the artefact carries an implicit claim of freshness. Scenarios end a run in more states than pass and fail: - **Passed** — every step executed and the assertions held. - **Failed** — a step raised or an assertion did not hold. - **Not implemented** — the scenario text exists but one or more steps have no automation behind them, so the runner reports them as undefined or pending. - **Skipped** — an earlier step failed, or a precondition hook aborted the scenario. - **Filtered out** — the run selected a subset by tag, and this scenario was never a candidate. - **Absent** — the scenario is in the repository, but the results file the generator read predates it. Only the first two are evidence. Every other state must be visible in the published document. ### The fixes, in order of leverage **1. Render status per scenario, and make "not executed" loud.** The generator should have no default that lets an unknown status render as ordinary prose. Treat any scenario without a pass-or-fail result as a first-class, visually distinct state — not-run — never as neutral body text. A scenario with undefined steps in particular is frequently mistaken for a passing one, because the text reads perfectly. **2. Stamp provenance on the artefact.** The header should carry the revision, the finish time, the environment, and the selection expression the run used. A reader who can see "generated from a run filtered to the smoke tag" knows immediately that the document describes a slice, not the system. A report with no provenance cannot be audited and cannot be trusted a week later. **3. Publish the counts, not just the pages.** Total scenarios in the repository, executed, passed, failed, not run. If those five numbers are on the front page, the misreading is much harder to sustain. **4. Gate the publish step.** If more than a small share of scenarios did not execute, the publishing step should fail rather than emit a misleading artefact. A document that is missing for a day is a smaller problem than a document that overstates the system for a month. **5. Never carry a previous run's status forward.** The tempting optimisation — "this scenario was not in today's filtered run, so show yesterday's green" — is exactly how a report ends up asserting behaviour on a revision that never demonstrated it. If it must be shown, label it with the revision it last passed on and mark it as not current. ### A worked example The four-person team on the parcel-tracking gateway runs a fast subset on every commit, filtered by tag, and the full suite nightly. Their published report was generated from the *commit* run. Of 214 scenarios in the repository, 118 were candidates under that filter; the remaining 96 were rendered as ordinary pages with no status marking. Among those 96 was the scenario for serving a status lookup after a cache expiry — which, on the nightly run, had been failing for eleven days because of a stale-cache read. The published specification of record was describing behaviour the system had not exhibited for a week and a half, and the document gave the reader no way to know it. The fix they shipped: generate the published artefact only from a full run; stamp the revision and the filter on the header; and fail the publish step when the not-run share exceeds a small threshold. ### The boundary worth stating Note what this is not. Counting how often the suite is red over time, or how long the pipeline takes, is a different conversation about suite and pipeline health. This is narrower and about the *document*: whether a reader can distinguish what the last run demonstrated from what it merely described. Getting that distinction right is what makes the artefact usable as a specification of record rather than as decoration. ### How to answer well Enumerate the statuses; say that anything other than passed or failed is not evidence and must render distinctly; insist on provenance in the header; and offer the publish gate as the mechanism that stops a misleading artefact from shipping at all. Mentioning that you would rather publish nothing than publish an overstatement is a strong senior signal.