When you start a garak scan against a chat endpoint and do not name any probes, what set of attacks actually runs, and why is that neither nothing nor the whole probe catalogue?

### What a probe is, and how garak's catalogue is shaped garak is an open-source LLM vulnerability scanner. Its unit of attack is a **probe**: a Python class that owns a fixed set of prompts embodying one attack idea, plus a declaration of which **detectors** — the classifiers that decide whether a reply counts as a hit — garak should run over the responses. Probe classes are grouped into modules by family (`garak.probes.<module>`), each module holding one or more classes. garak's selection flag, `--probes` (short form `-p`), takes a comma-separated list at **either** granularity: a bare module name runs the whole family, and `module.ClassName` runs exactly one probe. `garak --list_probes` prints the catalogue for the version you actually have installed, which is the only authoritative statement of what exists — the catalogue grows between releases, so a list you memorised last year is wrong. ### What runs when you name nothing Neither nothing nor everything. With no `--probes` given, garak falls back to a built-in default selection: a curated subset sized so that a first run finishes in minutes and shows you the report format, the per-probe pass and failure rates, and the scoring block. That default exists for **orientation**, not assurance. It is the tool's tutorial mode, and it is smaller than the full catalogue by a large factor. Two further filters sit underneath it and surprise people. First, a garak probe class carries an `active` attribute; a class flagged inactive is skipped when you select its module wholesale and runs only when you name that class explicitly. Second, some probes depend on something the machine may not have — a dataset download, a local support model, a key for a helper service — and will be skipped or will fail at load rather than announcing themselves loudly in the summary. ### What the two extremes cost Running the whole catalogue is the other pole, and it is expensive in a way the flag does not advertise. Prompt counts per probe span orders of magnitude: some probes carry a dozen handwritten prompts, others expand a public dataset into thousands. On top of that, garak's `--generations` flag re-sends every prompt N times and defaults to a small number greater than one, so the request count is already a multiple of the prompt count before you choose anything. Against a hosted, metered chat endpoint a full sweep is realistically tens to hundreds of thousands of requests — hours to overnight of wall-clock, and a bill that lands in real money rather than rounding error. It is a planned exercise with a written estimate, an owner's approval and a spend cap on the API key, never a default. ### Where the number misleads **Absence is silent.** A probe family that never ran does not appear in the report as a warning, an error, or a zero — it simply is not there, and a reader skimming the summary cannot distinguish "we tested encoding attacks and found nothing" from "we never sent an encoding prompt". This is the single most common way a garak result is over-read. **A zero-hit row is not proof of health either.** A probe whose calls all errored — rate-limited, timed out, or hitting a misconfigured generator — can still render a row that looks unalarming. The report shows what the detectors concluded about the attempts that came back, not that the attempts were real. So "we ran garak and it was clean" is not a finding. It is a sentence with no denominator, and it will be read as a clean bill of health by everyone downstream. ### What you would check 1. Run `garak --list_probes` and save that listing with the engagement notes — it pins the catalogue to the installed version. 2. Record the exact `--probes` string you passed, the `--generations` value, and the generator configuration the calls hit (`--model_type` / `--model_name`, the system prompt in force, decoding settings, any guard in front). 3. After the run, diff the probes present in the report against the selection you asked for, and explain every gap: inactive class, missing dependency, or typo in the module name. 4. Spot-read a handful of raw attempts behind one clean probe to confirm the responses arrived non-empty and non-errored. The statement you are then entitled to make is bounded and boring: *these probes, at this repeat count, against this configuration, on this date, produced no detector hits; everything else in the catalogue is untested.* Untested is a different word from safe, and the difference is the whole job.