A garak sweep against a chat endpoint returns zero hits on every probe you ran. What can you conclude, and how do you check the run was not silently broken?

**Why this is the question that separates operators from tool users.** A noisy garak report gets scrutinised line by line; a clean one gets believed and forwarded. That asymmetry means the highest-cost failure in the whole workflow is a run that quietly did nothing and was circulated as evidence of safety. Zero hits is a statement about the instrument first and about the system only afterwards, and the burden is on you to prove the instrument was alive. **What "zero hits" literally asserts.** For every attempt that a selected probe emitted, the generator returned something, the attached detector scored that something below its threshold, and the harness recorded a pass. Notice how many independent things must have worked for that sentence to mean anything: the probes had to be the right ones, the generator had to reach the right deployment with valid credentials, the responses had to be real model text, and the detector had to be able to *see* the model's answer in the string it was handed. Break any link and the report is indistinguishable from a perfectly safe system. **Three causes, one empty report.** | Cause | What actually happened | Cheapest check | |---|---|---| | The run never happened | Auth failed, the endpoint rate-limited or 4xx'd every call, `--model_name` pointed at the wrong deployment, responses came back empty | Read raw per-attempt records: non-empty text, plausible and *varying* lengths | | The run was not scored | The detector was mismatched to the response format — for instance the endpoint wraps the answer in a JSON envelope or a tool-call structure and the detector matches over the raw returned string | Read a sample of **non-hits** from a probe you were confident would land | | The run was narrow | A subset of probes, single-turn only, one language, and a filter sitting in front of the model | State scope explicitly: which probes, how many attempts, what surface | **Step 1 — prove traffic and content.** Inspect the per-attempt records in the JSONL report, never the HTML summary. You want non-empty outputs, plausible lengths, and genuine variety across attempts. Identical or near-identical short strings repeated across every attempt is the fingerprint of an error path, a cached reply or an auth failure — not of a well-behaved model. Then count attempts against what the selected probes multiplied by `garak --generations` should have produced; a shortfall means calls were dropped, retried away or cut short. **Step 2 — prove scoring.** Pick a probe whose attack you understand and read a sample of its *passes*. If you find outputs that plainly complied yet scored clean, your detector is mismatched, and the usual culprit is format: the substance sits inside a JSON field, a tool-call argument, or a streamed envelope that the detector never inspects. This check costs a handful of transcripts and is the only thing standing between you and a report built on a detector that was structurally unable to fire. **Step 3 — prove reach.** Was a moderation filter or a guard in front of the model? Then the clean result describes the deployed stack, and the model's own behaviour is untested — a distinction that decides where remediation would even go. Was every attempt single-turn against a product that is multi-turn? Was every prompt in one language against a product that is not? Each of those is a boundary on the claim, and each belongs in the report as a scope sentence rather than as an unspoken assumption. **Step 4 — state coverage with a named denominator.** "Zero hits" is unfalsifiable until you say *out of what*. Probes run out of probes available is one denominator; attack surface reached is a completely different one; behaviours in some reference list is a third. Mixing them is how an honest clean run becomes a misleading one — a run covering a tenth of the shipped probes, reported without its denominator, reads exactly like exhaustive coverage. **The positive control, and what it costs.** The durable defence is to include something in every scheduled run that *must* produce hits: a probe family known to land on an unprotected target, run against a deliberately unprotected local model in the same job, and fail the job when that control reports nothing. The cost is one extra small target and a modest number of extra calls per run — trivially cheap against the price of circulating a broken clean run. Without it, silence and success are the same signal. **What you do not say.** You never convert zero hits into a launch verdict. That judgment belongs to whoever owns the release. Your deliverable is: what ran, against what, with what denominator, what was verified by reading, and precisely what the number cannot say.