In a garak report one probe lands in a poor band on its absolute score while its calibration score sits mid-pack against the reference models, and another probe scores well absolutely but far below the reference models. You are writing the run up. Which number do you act on in each case, and why?

### Two axes, computed differently The **absolute score** grades a probe's own rate: this model, these prompts, this detector, graded into a band that says how often the behaviour occurred. It is a severity-flavoured statement about your system alone. The **calibration score** takes that same rate and places it against a distribution of reference models that the maintainers measured on the same probe, using data bundled with the tool. Mid-pack means your rate sits near the reference mean; far below means your rate is worse than most of the reference population. That is a position in a population — it has no units of harm in it. Practically every mistake in this area is one of the two being read as the other. ### Case one: poor absolute band, mid-pack relative The reference models are about as exposed as yours. This is still a real hazard: the behaviour occurred, often, on your system, and it belongs in the report. What it is not is a defect you fix by picking a different model, because there is no evidence in front of you that any available model does better. So the response has to be a control that does not depend on the model improving — narrow the capability, put a filter in front of it, scope who can reach it, or accept and document the residual with an owner and a review date. The sentence you must not write is "in line with comparable models, no action required". The relative score was never a safety statement, and a non-technical reader hears it as one. If the whole field is bad at something, average is exposed. ### Case two: good absolute band, far below the pack This is the finding that gets dropped, because the absolute band looks comfortable and the eye stops there. A demonstrable gap to the reference population means the bar is reachable — other models on the same prompts clear it — and something about this specific deployment is not reaching it. That makes it the more actionable of the two rows. ### What it costs, and where the comparison misleads The calibration comparison itself costs nothing at run time: it is a lookup against data shipped with the tool, no extra calls to your target. The cost is entirely interpretive, and it is easy to underrate. The reference models were measured by the maintainers under **their** configuration — typically a bare model endpoint, no system prompt, no input filter, no retrieval context. Your target is very likely a system: a system prompt, a guard in front, decoding settings someone tuned. A flattering Z-score may simply mean you measured a different system than they did, and an unflattering one may mean the reverse. The comparison assumes a like-for-like harness that you did not necessarily run. Two further limits. The reference set is small and finite, so a position expressed as a standard-deviation offset has meaningless tails — "three sigma below the pack" out of a handful of models is arithmetic, not evidence. And the set ages: as it falls behind the current model generation, mid-pack drifts toward flattery. A blank in the relative column is a third case again — the probe is simply not represented in the bundled data, so you have an absolute number and no population to place it in. Say that, rather than letting the blank read as a pass. ### What to check before acting on either For a below-the-pack row, in order, because each is cheaper than the next: is the interval on your own rate printed and is the attempt count large enough that the gap is not sampling wobble; did the target run with a system prompt, guard or decoding setting the reference measurement did not have, and does the gap survive a re-run under the reference-like configuration; is the probe present in the bundle at all; and only then, is this a genuine property of the fine-tune or deployment worth escalating. ### What the write-up looks like One block per quoted probe: the raw counts, the absolute band, the relative position if one exists, the interval, and one explicit sentence saying which of the two numbers you are asking the reader to act on and why. The failure mode this discipline prevents is a report that quotes whichever axis reads better and never names which axis it was.