A team quotes a monthly trend of failures from a scheduled promptfoo red-team job, and this month it dropped by half. The report does not say whether the suite was regenerated or whether responses were replayed from cache. What do you check to explain the drop, and what should each run record so the question is answerable next time?

### Why this happens at all A red-team number acquires authority by being repeated. Six months on, the chart is quoted in a review by someone who never saw the job configuration, and nothing in the chart records that the suite behind month three is not the suite behind month six. The defect is not statistical noise; it is missing metadata about the instrument. Every layer of a promptfoo red-team run can move independently of the target, and each one moves the number. ### The diagnostic order, cheapest first **1. The denominator — did the graded population change?** Compare the case count and the case identities against last month's run. If the job runs `promptfoo redteam run`, generation happens on every invocation and the cases are model-written afresh each time, so a large move in either direction is expected from the suite alone. This is a one-line comparison and it ends most investigations. **2. Did calls actually happen?** Compare run duration and recorded provider usage. promptfoo reports token counts per result; a run served from its response cache shows near-zero usage and finishes in a fraction of the usual wall clock, and it describes whichever deployment answered originally, not today's. A partial hit is the nastier version: some cases replayed, some live, producing a number no single system ever generated. **3. The grader.** The object deciding pass or fail is usually itself a model — a model-graded assertion with a rubric and a judge model. If the rubric text was edited, or the judge model was upgraded underneath you (a provider default moving is enough), identical responses get scored differently. Diff the assertions and pin the judge model explicitly so this stops being a variable. **4. The target.** Only now is "the application got safer" supportable, and it still needs a deployment identity you can point at: a model version, an image tag, a prompt revision. "Same endpoint name" is not an identity. ### What it costs to answer versus to prevent Answering after the fact is expensive: an afternoon of one engineer's time, plus possibly a full cold rerun of the suite (hundreds to low thousands of metered calls, tens of minutes) to reproduce last month — and if the old suite was never persisted, the question is simply unanswerable and the point has to be dropped. Preventing it is a few dozen bytes written alongside each result. The asymmetry is the whole argument. ### What a defensible run record contains ``` suite: hash of the generated case file, case count target: provider, endpoint, model/deployment version identifier grader: assertion set version, judge model and its version runtime: cache mode, duration, request + token counts, promptfoo version result: absolute pass/fail counts, not only a rate ``` Stored with the result — in the eval output, in the artefact store, in the row that backs the chart — not in a chat message or someone's memory. ### Where the number misleads Two readings do most of the damage. First, **a rate hides its denominator**: failures falling from 60 to 30 is a halving whether the case count held at 600 or fell to 300, and only absolute counts make the difference visible. Report counts and the denominator alongside any percentage. Second, **a continuous line implies a continuous instrument**. Drawing one line across a regeneration invites the reader to interpret an instrument change as a behaviour change, and they will, because that is what a line means. Break the series at every suite change and annotate the break. A third, quieter one: even when everything checks out, the improvement is an improvement *on those cases*. Expect and pre-empt the follow-up question — does the gain survive a freshly generated exploratory suite? A drop that appears only on the pinned cases is a tuning artefact. ### What you check before crediting the application Confirm case count and suite hash match; confirm the run was cold and the provider recorded the expected traffic; diff the grader configuration and confirm the judge model version; then obtain the target's deployment identity from whoever owns it and confirm it actually changed. If all four hold, say what changed and by how much, in counts, and mark the deployment on the chart so the next reader does not have to repeat this.