Red-Team Generation

It supports one narrow claim: the target refused those harms when asked plainly, in a single turn, in the generator's own phrasing. It says nothing about the same asks under an adversarial framing or across several turns, which is where real failures live. Report it as passing the plain baseline, never as safe.

promptfoo's red-team mode reads that description twice: the generator writes attack cases from it, and the graders judge replies against it. A one-line description yields generic attacks that miss your app's real surfaces, and gives the graders no stated rule to fail an answer against, so weak replies get marked pass.

It sets how many adversarial cases promptfoo generates for each enabled plugin, so total volume scales with plugins times that count. A small default gives each harm class only a handful of tries, so a failure the app commits rarely can easily never fire, and the run then reports a clean pass.

You can conclude that on this run, for those three harm classes, at that suite size, nothing failed. You cannot conclude the assistant is safe. Harm classes with no plugin enabled generated no cases, so the report is silent about them. Silence is a missing denominator, not a pass.

Strategies do not add cases, they re-express them. Each strategy takes the cases the plugins generated and emits a transformed variant of every one, so the suite scales with plugins times strategies rather than growing by a fixed amount. Every scheduled rerun re-pays that multiplied count in target calls, grader calls and wall-clock time.

By default promptfoo generates red-team cases through its hosted generation service, so the description of your application leaves your machine. Treat it as text you would hand a vendor: no secrets, no customer records, no internal hostnames. There is a documented switch that turns remote generation off and generates locally instead.

No. promptfoo's graders judge each reply against the description you supplied. If it never says what the assistant must refuse, who may ask, or what records it reaches, an off-policy answer breaks no declared rule and is scored a pass. Rewrite the description with roles, data and refusals, then re-run.

Work from what the application can actually do: its tools, the data it can reach, who talks to it, and what it must never say. Enable the harm classes it could genuinely commit. Enabling everything costs generation and inference on irrelevant classes and buries real failures in noise nobody triages.

It means the refusal is keyed to the surface form of the request rather than to its intent: change the framing and the same harm goes through. Pool it into one overall pass rate and that signal disappears, because the plain copies dilute the wrapped failures. Report per plugin and per strategy instead.

Treating each case as an independent try, the chance of missing every time is 0.95 to the tenth, about 60 percent. So a clean run of ten mostly reflects the sample size. To be roughly 95 percent sure of seeing a one-in-twenty behaviour you need about sixty cases.

Not before separating two causes: the application genuinely cannot commit that harm, or it can and the failures are being judged against the wrong expectation. Only the first justifies disabling. Disabling deletes the class from every future run too, so the next report is silently narrower while the headline number improves.

Check that the target definition carries conversation state between turns. If every request hits the endpoint as a fresh conversation, a multi-turn strategy's gradual escalation is thrown away and each turn lands as an isolated plain ask, so it under-reports by construction rather than because the target is strong.

Do not thin everything evenly - that leaves every harm class too small to conclude anything while still costing money. Split the schedule: a small nightly tier on the surfaces that change, and a deep sweep with high case counts before releases. Stop regenerating cases you could reuse, and label the thin tier as a smoke test.

Split by what each strategy is for. The per-release run should be a cheap, stable regression arm: deterministic framings, fixed cases, fast enough to block a release. Expensive iterative and conversational framings go to a scheduled deeper run, where a slow, stochastic result is investigated by a person rather than gating a deploy.

Treat each description as a reviewed, version-controlled artifact owned with the application. When an app gains a tool, role or data source and the text does not, promptfoo's generator stops attacking the new surface and its graders stop failing it, so the pass rate rises while real risk grows. Diff it every release.

Treat it as allocating a fixed recurring budget, not picking a setting. Give high counts to the harm classes where a miss is expensive and to apps with real exposure, and thin counts elsewhere. A uniform number overspends on harms an app cannot commit while under-sampling the ones it can, and hides both behind one green score.

Set a baseline of harm classes every application must run, plus per-application additions derived from its capabilities, with every exclusion carrying a written reason and an owner. Record the enabled set with each report. When the catalogue grows, new classes change the denominator, so cross-quarter comparisons are invalid unless you re-baseline deliberately.