Suite as an Artifact

### The two phases, and which one people think they are measuring promptfoo's red-team mode splits into generation and evaluation. `promptfoo redteam generate` reads your red-team configuration — the `purpose` or application description, the `plugins` list (harm categories: what to test for) and the `strategies` list (transformations: how to deliver it), with `numTests` controlling how many cases each plugin contributes — and calls a model to expand all of that into concrete adversarial test cases, writing them to a file (`redteam.yaml` by default, or whatever `--output` names). Evaluation then sends each case to the `providers` under test and applies the per-plugin graders. `promptfoo redteam run` does both in one command, which is exactly the shape that gets pasted into CI and causes the problem. Generation is a model call. Two runs of the same configuration produce different wording, different scenarios, and — because strategies expand base cases and some expansions are conditional — sometimes a different total count. The configuration is deterministic; the suite it produces is not. ### What pinning buys - **A stable denominator.** The same cases in the same number every run, so a rate means something across runs. - **Reproducibility.** A reported failure names a concrete case a developer can rerun locally in seconds. - **Reviewability.** A refresh becomes a diff someone approves, not an invisible change in the instrument. - **Attribution.** When the number moves, the suite is off the list of candidate causes, and the target is the only thing left. ### What pinning costs A frozen adversarial suite is a fixed target, and prompts get tuned until those exact strings stop working. The score improves, real robustness does not — the ordinary overfit-to-the-eval failure, and it is fast, because a system prompt edit that hard-codes a refusal for the pinned scenarios is a day's work. The suite also stops representing what the plugin and strategy catalogue has grown to cover since it was written, so its coverage claim silently expires. And it is a committed file of adversarial content: it needs the same access thinking as any other sensitive artefact, and reviewers need to know that a large diff of attack strings is expected rather than alarming. ### What each option costs Regenerating every run pays for generation calls on top of evaluation calls, every night, forever, plus the latency and the external dependency if generation runs through a hosted generation service rather than locally. Pinning removes the generation phase from the nightly job entirely, which is a real but modest saving — the evaluation calls dominate — and removes most of the run-to-run variance in wall clock and case count. The genuine cost of pinning is not money; it is the discipline of scheduling and reviewing refreshes, which is engineer time nobody budgets for. ### Where the number misleads With a regenerated suite, a month-over-month delta has two causes and no way to separate them. Worse, the suite-only noise is not small: plugins sample a space of possible attacks, so two generations of an identical configuration against an *unchanged* target can differ by several points of failure rate purely from which scenarios happened to be written. A three-point improvement quoted from such a job is comfortably inside the noise of the instrument, and reading it as progress is the specific mistake. With a pinned suite, the misleading reading is the opposite one: a pass rate on a frozen file is not a probability that a real attacker fails. It is the fraction of a few hundred model-written strings that one grader declined to flag. "We pass 100% of the red-team suite" means those strings no longer work, and nothing at all about strings nobody generated. ### What you check Before believing a delta, hash the suite file and confirm both runs used the same one, then confirm the case counts are identical; if you regenerated deliberately, run old and new suites the same day so the level shift is visible. Before believing a pinned improvement, run a freshly generated exploratory suite against the same target: if it still finds failures at roughly the old level while the pinned score climbs, you measured tuning, not safety. ### The operating pattern Keep a pinned suite as the regression baseline, refreshed as a reviewed commit on a deliberate cadence. Run a separate, freshly generated exploration job whose purpose is finding *new* categories of failure, and promote what it finds into the pinned suite as added cases rather than replacing it wholesale. Report the two apart: the pinned suite answers "did we regress", the exploratory run answers "did we miss something". One blended number answers neither.