garak

A probe that mints prompts each run does not send the same prompts twice, so one lower count is a different sample, not a result. Replay the specific saved prompts that failed before against the patched system, and rerun the generating probe several times on both builds before claiming an improvement.

Ask which probes ran and whether their prompts are a public fixed list. A public list can be filtered or tuned against, so a pass shows those strings are handled, not that the behaviour is absent. Ask for a rerun that includes probes minting prompts at run time, with the probe selection and logs attached.

Spend repeats where they decide something: a high count on the few probes gating the release, a low one elsewhere, and pool several cheap runs rather than one big one. Never buy repeats by pinning the generator's decoding to be deterministic when production samples, that measures a system you do not ship.

Split the sweep into several small runs, each with a named probe selection and its own report, ordered so the families you must report on finish first. Keep the target configuration identical across runs so results stay comparable, log the selection per run, and treat a died-mid-probe result as incomplete, never as a pass.

Decline when nobody will own the ruling criterion over time. A custom pair needs re-labelling as the target changes, and a drifting criterion quietly moves the number while looking stable. For a one-off question, test by hand or with a small held-out set; reserve custom plugins for risks you will re-run every release.

Treat the rates as a trend within one pinned probe and detector set, never as an absolute safety figure. Pin that configuration across releases, hand-audit a sample of failures and passes each cycle to estimate detector error, publish the audited numbers with sample sizes, and never let a rate alone gate a release.

Name what each choice measures. The bare model measures the model; a staging deployment measures the app's prompts, retrieval and filters; production measures the live system plus its abuse controls, at the cost of polluted logs, alerts and real spend. Default to a staging build that mirrors production, and state the choice in the report.

Group hits by the behaviour they demonstrate rather than writing one finding per hit. Read several per group, confirm the detector was right, and keep one finding with a verified example and the count behind it. Drop groups where every sampled hit was an artefact, and state what you sampled.

A relative score is a position against reference data that ships with the tool, so it moves when that reference data or the probe's prompts change, even with your model untouched. Sampling noise and a silently updated hosted endpoint also move it. Pin the tool and reference bundle per series, and trend absolute scores.

Set it per tier, not once: a small count for fast pre-merge scans, a much larger one for the release gate, and freeze it so rates stay comparable between runs. Publish the count next to every rate, and say plainly that a clean scan at a low count is weak evidence, not proof of absence.

Spend a first slice broadly and shallowly across families to find where signal exists, then reinvest the rest depth-first on the families that hit and the ones the deployment's threat model cares about. Tell the stakeholder exactly which families ran, which were never run, and that unrun means untested.

Spend the budget on breadth first: buffs add no new behaviour, they re-send prompts you already have. Buy buffed runs for a narrow slice where input normalisation or a filter is the thing under test, or to audit whether your own detectors are brittle. Treat them as an occasional audit, not a gate.

Gate the pipeline on a small pinned set of fixed-prompt-list probes, because only deterministic prompts give a comparable pass/fail signal per build. Run prompt-generating probes on a slower cadence for fresh coverage, reviewed by a human. Pinned probes go stale and invite tuning; generated ones produce noisy counts that cannot block a build.