In a promptfoo red-team configuration, attack plugins and attack strategies are two separate lists. If you enable one more strategy, how does the generated test-case count change, and why does that matter for a suite that reruns on every release?

### The two lists are different kinds of thing A promptfoo red-team configuration holds two independent lists inside its `redteam` block: `redteam.plugins` and `redteam.strategies`. A **plugin** is a case generator for one harm class — it decides *what* a test case tries to make the target do. A **strategy** is a transform that runs over cases which already exist — it decides *how* that same attempt is dressed up before it is sent. They are separate catalogues with separate semantics, and the most common configuration mistake in this tool is reading them as one long list of "attacks" you tick boxes in. ### What generation actually does, in order `promptfoo redteam generate` runs in two stages. Stage one asks an attacker model to write `redteam.numTests` cases for each entry in `redteam.plugins`, using `redteam.purpose` — your prose description of the application — as the context that makes those cases plausible for your app rather than generic. Call the result the **base set**: roughly `numTests × plugins` cases. Stage two hands that entire base set to each entry in `redteam.strategies`, and each strategy emits its own variant of every base case. The generated test file that falls out therefore contains the base set plus one transformed copy of the base set per enabled strategy. So for single-shot framings the arithmetic is `numTests × plugins × (1 + strategies)`. Enabling one more strategy does not add a handful of cases; it adds another complete pass over everything the plugins produced. Ten plugins at ten tests each is 100 base cases; three strategies makes it 400. ### The two strategy families cost very differently | family | calls per case at eval time | reproducible? | |---|---|---| | single-shot transform (rewrites the case once, then sends it) | 1 target call + 1 grader call | yes — same input, same request | | iterative / multi-turn (chooses the next message from the target's reply) | up to the strategy's turn cap in target calls, plus an attacker-model call per turn, plus grading | no — the trajectory differs run to run | A multi-turn framing with a five-turn cap is therefore on the order of ten model calls per case rather than two, and the exact number is not knowable before the run because the loop can stop early when it succeeds and can run to the cap when it does not. ### What a run costs, and which half repeats Generation is paid once per `generate` and is the cheap half: one attacker-model call per case written. The recurring cost is `promptfoo redteam eval`, where every case in the generated file is one call to your target plus — in the default configuration — one call to a grader model that decides whether the response counted as a failure. That is the number to budget, and it is paid **in full on every scheduled rerun**, because a rerun does not regenerate; it replays the same file. Wall-clock is that call count divided by your concurrency setting, then floored by the provider's rate limit, which is usually the real constraint when the red team shares a quota with production traffic. ### Where the number misleads Three readings go wrong. First, the generated case count gets quoted as **coverage**. It is not: strategy copies re-express harms you already had, so 400 cases across ten plugins still covers ten harm classes. The coverage denominator is the plugin list, never the case list. Second, the headline pass rate moves when the strategy list moves. Add a framing your target happens to survive and the average rises while nothing about the target changed; add one aggressive framing across every plugin and it falls the same way. A rate quoted without the plugin-and-strategy list that produced it is not comparable with last month's rate. Third, "generation was one-off, so the reruns are basically free" is exactly backwards — generation is the cheap half, and the multiplied eval is what you pay every release, twice over once graders are counted. ### What to check before you commit to a configuration Count the cases in the generated file before running anything — `generate` prints the total, and the file itself is inspectable. Estimate calls as `base × (1 + single-shot strategies) + base × multi-turn strategies × turn cap`, then double for graders and add headroom for retries. Do one deliberately small run with a low `numTests`, read the real token spend and elapsed time off it, and scale linearly. Finally, confirm your concurrency does not exceed the target's rate limit: over it, calls turn into errors, errors can be graded as refusals, and you get a cheap fast run with a clean report — both of them wrong.