When you start a garak scan against a chat endpoint and do not name any probes, what set of attacks actually runs, and why is that neither nothing nor the whole probe catalogue?
answer
- default set, not full catalogue
- module or single class
- some probes only run when named
- report enumerates what ran
- sample, not an assessment
basics
~20 sgarak runs a built-in default selection of its probes, not the entire catalogue. The full catalogue is far larger and would cost hours of metered calls. Some probes also only run when you name them. So a default run is a sample, and you must read the report to see which probes actually ran.
solid answer
~50 sgarak's probes are organised into modules, each holding one or more probe classes, and you can select at either granularity: a whole module, or one class inside it. Give no selection and you get a curated default set, which exists so the tool is useful in minutes rather than hours; it is a starting sample, not an assessment. Two things bite juniors here. First, the catalogue is much larger than the default, so "I ran garak" says almost nothing until you say *which* probes. Second, some probes are flagged so a broad sweep skips them unless you name them explicitly — a family can be absent from your results without any error appearing. The only reliable statement of what ran is the run's own report, which enumerates the probes attempted; list the catalogue first and diff it against that report before writing any coverage sentence.
go deeper
Should know that garak selects probes by module or class, that leaving the selection empty runs a default subset rather than everything, and that the report is where you check what ran.
Should add that prompt counts vary enormously per probe, that some probes are skipped unless named, and should be able to reconstruct the actual selection from the report.
Frames selection as choosing the denominator for the coverage claim, and builds the catalogue-versus-report diff into the engagement's evidence trail.
Sets the organisation's policy: which selections constitute a release-gating scan versus a smoke run, who approves a full sweep against a paid endpoint, and how untested families are recorded.
### What a probe is, and how garak's catalogue is shaped garak is an open-source LLM vulnerability scanner. Its unit of attack is a **probe**: a Python class that owns a fixed set of prompts embodying one attack idea, plus a declaration of which **detectors** — the classifiers that decide whether a reply counts as a hit — garak should run over the responses. Probe classes are grouped into modules by family (`garak.probes.<module>`), each module holding one or more classes. garak's selection flag, `--probes` (short form `-p`), takes a comma-separated list at **either** granularity: a bare module name runs the whole family, and `module.ClassName` runs exactly one probe. `garak --list_probes` prints the catalogue for the version you actually have installed, which is the only authoritative statement of what exists — the catalogue grows between releases, so a list you memorised last year is wrong. ### What runs when you name nothing Neither nothing nor everything. With no `--probes` given, garak falls back to a built-in default selection: a curated subset sized so that a first run finishes in minutes and shows you the report format, the per-probe pass and failure rates, and the scoring block. That default exists for **orientation**, not assurance. It is the tool's tutorial mode, and it is smaller than the full catalogue by a large factor. Two further filters sit underneath it and surprise people. First, a garak probe class carries an `active` attribute; a class flagged inactive is skipped when you select its module wholesale and runs only when you name that class explicitly. Second, some probes depend on something the machine may not have — a dataset download, a local support model, a key for a helper service — and will be skipped or will fail at load rather than announcing themselves loudly in the summary. ### What the two extremes cost Running the whole catalogue is the other pole, and it is expensive in a way the flag does not advertise. Prompt counts per probe span orders of magnitude: some probes carry a dozen handwritten prompts, others expand a public dataset into thousands. On top of that, garak's `--generations` flag re-sends every prompt N times and defaults to a small number greater than one, so the request count is already a multiple of the prompt count before you choose anything. Against a hosted, metered chat endpoint a full sweep is realistically tens to hundreds of thousands of requests — hours to overnight of wall-clock, and a bill that lands in real money rather than rounding error. It is a planned exercise with a written estimate, an owner's approval and a spend cap on the API key, never a default. ### Where the number misleads **Absence is silent.** A probe family that never ran does not appear in the report as a warning, an error, or a zero — it simply is not there, and a reader skimming the summary cannot distinguish "we tested encoding attacks and found nothing" from "we never sent an encoding prompt". This is the single most common way a garak result is over-read. **A zero-hit row is not proof of health either.** A probe whose calls all errored — rate-limited, timed out, or hitting a misconfigured generator — can still render a row that looks unalarming. The report shows what the detectors concluded about the attempts that came back, not that the attempts were real. So "we ran garak and it was clean" is not a finding. It is a sentence with no denominator, and it will be read as a clean bill of health by everyone downstream. ### What you would check 1. Run `garak --list_probes` and save that listing with the engagement notes — it pins the catalogue to the installed version. 2. Record the exact `--probes` string you passed, the `--generations` value, and the generator configuration the calls hit (`--model_type` / `--model_name`, the system prompt in force, decoding settings, any guard in front). 3. After the run, diff the probes present in the report against the selection you asked for, and explain every gap: inactive class, missing dependency, or typo in the module name. 4. Spot-read a handful of raw attempts behind one clean probe to confirm the responses arrived non-empty and non-errored. The statement you are then entitled to make is bounded and boring: *these probes, at this repeat count, against this configuration, on this date, produced no detector hits; everything else in the catalogue is untested.* Untested is a different word from safe, and the difference is the whole job.
- You need to prove which garak probes ran. Where does that evidence come from?From the run's own report, which enumerates the probes attempted and their per-probe results — not from the command you believe you typed. Diff the report against the catalogue listing and against your intended selection.
- Why is 'we ran garak on it' an unacceptable line in a security report?It names no denominator. Without the probe selection, the repeat count and the target configuration, the reader cannot tell whether a whole attack family was tested or skipped.
- Is running the entire catalogue the safe default choice?No. Prompt counts per probe differ by orders of magnitude, so a full sweep against a metered endpoint is an hours-long, billable job. It is a planned exercise with an estimate attached, not a default.
saying these in an interview costs you the question
- Believing the default run covers the whole probe catalogue.
- Reporting 'garak found nothing' without stating which probes ran.
- Assuming a missing probe family would have shown up as an error.
- Treating a full sweep as free or as the obvious default against a hosted paid endpoint.