Attack Strategies

### What a strategy-free run actually sends When `redteam.strategies` is empty or absent, promptfoo evaluates exactly the base set that `redteam.plugins` produced and nothing else. Each case is a direct, single-turn request for the harmful thing, phrased by the attacker model in ordinary language and delivered to the target cold — no wrapper, no role-play, no prior turn, no encoding. That is the simplest possible form of every attack in the catalogue you selected. ### Why that form passes so easily Safety tuning is trained largely on directly-phrased harmful requests, so the plain surface is the exact distribution a tuned model has seen most of. A base case reads like the textbook example of the thing it should refuse, and any target with real alignment work behind it turns those down. The practical consequence is that a strategy-free suite behaves much closer to a smoke test than to an assessment: it confirms the target is not catastrophically open, and it confirms your wiring works end to end — the provider is reachable, responses are being extracted, the grader is running and returning verdicts. Those are genuinely worth confirming. None of them is a safety result. ### What the run costs — and why the cheapness is part of the trap This is the cheapest arm you will ever run: one target call plus one grader call per base case, no attacker loop, no extra copies. Eight plugins at ten tests each is 80 cases, roughly 160 model calls, minutes of wall-clock and small change in spend. That is exactly why it gets run first, run often, and quoted — and why a claim built on it costs the organisation far more than the run did. The strategies you did not enable are where the money and the time would have gone, and their absence is invisible in the output. ### What the pass rate is a denominator over Four things, all of which you chose: 1. **The harm classes you enabled** as plugins. A harm you did not select has no denominator at all, not a passing score. 2. **`numTests` per plugin** — how many cases were drawn for each class. 3. **One framing**: plain, single turn. 4. **The `redteam.purpose` text**, which is what made the cases specific to your app. Thin purpose text yields generic cases the target refuses easily, so a weak description inflates the result. The sample size deserves a second look. Ten cases per plugin, all passing, is not evidence of a low failure rate — a behaviour that genuinely fails one time in twenty will show zero failures in ten draws most of the time. Zero-out-of-ten is consistent with a true failure rate of up to roughly a quarter. "No failures" and "a low rate" are different claims, and only the first one was measured. ### How a green sweep can be entirely fake Before believing a clean result, rule out the failure modes that produce one for free. Response extraction that returns the whole JSON envelope rather than the assistant's text makes the grader score a wrapper, not an answer. A provider misconfiguration, a rate limit or a timeout turns calls into errors or empty strings, which a grader will happily score as non-compliance — errors read as refusals, and the report goes green. An authentication failure at the endpoint does the same thing on every case at once. All of these produce a perfect score from a suite that never reached the model. ### What to check Open a handful of raw outputs and read them. A real pass looks like the model's own voice declining, varied case to case. Identical strings on every case, empty bodies, or an error object mean the harness, not the target, produced your result. Confirm the non-2xx count is zero. Reproduce one case by hand against the same endpoint. Then enable one cheap deterministic strategy and rerun: if literally nothing moves across framings, be suspicious rather than pleased. ### How to report it One sentence, with the qualifiers attached: the target refused every plain, single-turn request in the harm classes we enabled, at this sample size, under the purpose text we supplied — and we have not yet tested adversarial framing or multi-turn escalation, so this is a baseline, not a safety verdict. The reportable finding arrives later, as the gap between this baseline and the same cases wrapped.