A garak run over a hand-picked subset of probes finishes with nothing flagged. How do you write that up so it is not read as "the model is safe", and what denominator do you attach when you use the word coverage?

The failure this question tests is a single leap: from *the tool printed no failures* to *the system is safe*. Every clause of the write-up exists to block that leap, and the discipline is to make each bound explicit rather than trusting the reader to infer it. ### Fix the denominator Three different things get called "coverage", and a garak run measures only the first: | claimed coverage | what it would mean | does the scan measure it? | |---|---|---| | probe coverage | probes run / probes in the catalogue you selected from | **yes** — this is the only one | | attack-surface coverage | share of the deployment's reachable paths exercised | no — garak talks only to the generator you wired up | | behavioural coverage | share of real-world eliciting behaviours reproduced | no — probes sample families, they do not enumerate them | So state the number with its units visible: *N of the M probes in the catalogue we selected from*, produced by diffing `garak --list_probes` against the probes present in the report. The moment that fraction is relabelled "80% coverage" in a slide, it has silently become a claim about the product rather than about the tool. Attack-surface is the substitution that hurts most in practice. If the deployment has retrieval, tool or function calling, multi-turn session state, or a front-end guard, and the scan pointed at a bare model endpoint, none of those paths were touched — and in real incidents that is where the failure lives. ### Fix the configuration A result is only meaningful against the configuration that produced it, and every part of that configuration drifts. Record the endpoint and model version, the system prompt in force (with its version), decoding settings, `--generations`, whether a guard sat in front, and the date. A clean scan against a raw model says nothing about the guarded production stack; a clean scan against the guarded stack says nothing about the model underneath. Those are two different results and they support two different decisions. ### Fix the epistemics of a zero Every zero in the report is a **detector's** decision, not an observation of the model. A detector that missed and a model that genuinely refused produce identical rows. So "no hits" is evidence bounded by detector sensitivity, and the bound is not small: detectors keyed to particular phrasings can miss a model that complies in a different register, and a probe whose calls all errored also reports as unalarming. The verification is cheap and you should budget for it: pull the raw attempts behind at least one clean probe out of the JSONL report and read a sample of the responses yourself. Thirty to sixty minutes per engagement catches the two failures that would otherwise invalidate the whole write-up — attempts that never reached the model, and a detector criterion that does not match how this model phrases compliance. Compare each probe's attempt count against its expected prompts x generations figure while you are in there. ### Use the tool's own framing, not a stronger one garak reports per-probe pass and failure rates, an absolute score, and a **calibration Z-score** positioning the target against a comparison set of models. A Z-score is a *relative position*: it says this target does better or worse than comparable models on that probe. It is not a threshold, not a margin of error, and not a pass mark — the comparison set could itself be uniformly weak. Reporting "scored above average" as though it were "meets bar" is how a relative statistic becomes a certification nobody meant to issue. ### The deliverable sentence > "garak <version>, probes X, Y, Z at N generations, against endpoint E with system prompt version V and guard G, on date D: no detector hits. Families A, B, C were not run and remain untested. Retrieval and tool-calling paths were out of scope. One clean probe was spot-checked at the raw-attempt level." Boring, bounded, and it survives a later incident. The cost of getting this wrong is not abstract: a write-up that reads as a clean bill of health gets used to gate a release, and when something surfaces afterwards the argument is about what you claimed, not about what you ran. Unrun families go in their own explicit list, under the heading *untested* — never under *passed*, and never omitted, because an omitted family reads as fine to every downstream reader.