Running a Scan

### What the setting is, and where it lives garak's unit of work is the **attempt**: one prompt, sent once, scored once. Three objects meet at that unit. A **probe** — a Python plugin class, the kind `garak --list_probes` prints — owns a fixed set of prompts. A **generator** — selected with `garak --model_type` and `--model_name` — owns the actual call to the system under test. A **detector** owns the verdict on the text that comes back. The repeat count sits between the probe and the generator: it is `garak --generations N` on the command line (short form `garak -g N`), or `run.generations` in the YAML you hand to `garak --config`. It decides how many times each of the probe's prompts is sent, and therefore how many attempts each prompt produces. Nothing else about the run changes. The prompt list belongs to the probe class; which probes run is fixed by `garak --probes`. Raising the repeat count re-sends what you already selected, and each returned response is passed to that probe's detectors independently — so a single prompt can yield a mix of clean and failing attempts. ### Why the knob exists at all The thing under test is a sampler. A hosted chat endpoint at ordinary decoding settings does not return the same text twice, and a guardrail sitting in front of it may fire on one draw and not the next. One generation per prompt is one Bernoulli draw: it reliably detects behaviour that fails every time, and is close to blind to behaviour that fails occasionally. The repeat count is the only knob that buys resolution on the second kind. ### What it costs Everything downstream of the probe is linear in N. | what scales with N | how | | --- | --- | | model calls | prompts x N, plus retries | | tokens billed | the same multiple, on both prompt and completion | | wall clock | x N, unless you also raise `garak --parallel_attempts`, which against a rate-limited endpoint mostly buys backoff | | detector work | one verdict per attempt | | triage volume | roughly x N on any probe that fires — false positives included | Concretely: a selection totalling 2,000 prompts at `-g 1` is 2,000 calls; at `-g 10` it is 20,000. At a rough 400 prompt and 200 completion tokens per call that is about 12M tokens instead of 1.2M — the difference between pocket change and a visible line item on a metered endpoint — and at a sustained three requests per second it is roughly two hours of wall clock instead of eleven minutes. None of that spend buys a single new prompt. ### Where the number misleads Two readings go wrong, and both are common. First, **absolute hit counts are not severity**. The digest and the `eval` records in garak's report carry counts as well as rates, and the counts scale with N. A probe showing 40 hits at `-g 10` is not four times worse than one showing 10 hits at `-g 10`, and it is not worse at all than a `-g 1` run showing 4 — one flaky prompt repeated ten times manufactures ten hits. Read the rate, and read the repeat count that produced it, before you rank anything. Second, **more repeats is not more coverage**. Depth and breadth are separate budgets drawn on the same wallet. A team that spends its whole query allowance raising N on the probes it already ran knows more about those prompts and exactly nothing more about the families it never selected. If the honest gap is "we have never run that probe family at all", the repeat count is the wrong knob and buying it is a way of feeling thorough without becoming thorough. ### What to check before you believe a run 1. **The count actually used.** Read `run.generations` out of the run's own recorded configuration or report header, never from memory: garak's default has not been the same value across releases, so a remembered default is a guess dressed as a fact. 2. **The generator's decoding settings.** If the target is configured to be effectively deterministic, high N returns near-duplicates at full price and the run is not sampling the variance you paid for. 3. **That attempt totals reconcile.** For a probe with P prompts the scored total should be P x N. If it is short, attempts errored, timed out or were dropped by the transport — and a dropped attempt is missing data, not a pass.