You own adversarial-testing coverage for a dozen LLM applications, all scanned with promptfoo's red-team mode. How do you govern which harm-class plugins each application runs, and what breaks when the available catalogue changes between quarters?

At portfolio scale the hard problem stops being "can we run promptfoo" and becomes "can anyone believe the numbers". Selection governance is the answer to the second question. ### A three-layer selection **Baseline.** The harm classes every application in the portfolio runs, expressed as a shared `redteam.plugins` fragment that each application's `promptfooconfig.yaml` composes in. Map each baseline class to an entry in the organisation's risk register, because the risk register is the language a launch decision is actually argued in; a selection justified only by "it is in the tool" does not survive an auditor. **Per-application additions.** Classes added because this system has tools, data reach, a caller population or regulated content the baseline does not assume, plus `policy` entries carrying its own business rules. **Explicit exclusions.** Every catalogue class not run carries a capability argument, a named owner and a review date. An exclusion with no reason is a defect, not a configuration. Exclusions decay faster than anything else in this file, so any change to an application's tools or data access re-opens all of that application's exclusions at once. ### Comparability, and the ranking trap Two applications with different plugin lists have different denominators, so their pass rates are not on the same axis and must never be ranked side by side on a dashboard. Ranking is the failure mode that quietly corrupts the whole programme: it gives every team an incentive to narrow its own selection, and once selection is a performance lever the taxonomy stops describing risk. Publish per-application results with their resolved plugin list attached, and resist any single portfolio-wide safety score — a compressed number is exactly the artefact that loses the context that made it meaningful. ### Catalogue drift, and the version trap underneath it New harm classes appear between promptfoo releases, and existing collection aliases absorb them. This is the mechanism worth naming precisely: `plugins: [harmful]` or `plugins: [owasp:llm]` resolves to a **version-dependent** set. Upgrade promptfoo and that one unchanged line can expand to more plugins than it did last quarter, generating more cases, exposing more surface and lowering the pass rate — with no diff in any application's config to explain it. A team can spend a week hunting a regression that was a dependency bump. Three effects arrive together whenever the catalogue moves: applications that adopt new classes get worse-looking numbers for reasons unrelated to their behaviour; applications that do not adopt them look better than their peers for no merit; and any trend line spanning the change compares two different measurements. The disciplined handling: pin the promptfoo version and record it with every result; prefer resolved plugin lists over aliases in the stored artefact even if aliases are used in the config; re-baseline on an announced date; run one overlap period where both the old and new sets are scanned so the step change is quantified rather than argued about; then treat pre- and post-baseline results as separate series. The same discipline applies in reverse when a class is dropped. ### What this costs at twelve applications Twelve applications, twenty-five classes each, five cases per class is roughly fifteen hundred base cases per cycle before strategies multiply them, and each case is a target call plus a grader call. Nightly, that is a standing bill and a couple of hours of machine time — affordable. What is not automatically affordable is the triage: findings that nobody reads are the same as findings that were never generated, except more expensive. Budget the reviewer time before you budget the tokens, and let that budget, not the catalogue, set how broad each selection can honestly be. ### What I would instrument Selection drift per application over time; the count and age of unreviewed exclusions; the promptfoo version each result was produced under; and — the cheapest early warning in the whole programme — whether any application's resolved plugin list or per-class case count *shrank* between consecutive runs. A rising pass rate next to a shrinking selection is a denominator change wearing the costume of an improvement. ### What I would resist A mandate to enable everything everywhere, which converts the programme into a noise generator, flatters every aggregate with easy passes on inapplicable classes, and destroys triage. And a single number reported upward, which is how a narrow scan becomes a broad clearance in someone else's slide.