A garak report can position a probe's result against previously measured reference models as well as giving an absolute score. Why does a probe and detector you wrote yourself not get that comparative reading, and what do you present instead?

### What the comparative reading is, mechanically Alongside an absolute score, a garak report can position a probe's result against a bag of previously measured reference models: roughly, how far your target's score sits from the mean of that bag, in standard deviations. That is a *positional* statement, and a positional statement is only meaningful if everything except the target was held fixed when the reference data was collected — the same prompt set, the same ruling criterion, comparable run conditions. The calibration data ships with the tool and is keyed by plugin identity: this probe, scored by this detector. A probe and detector you wrote yesterday have no key in that data. Not because custom plugins are second-class, but because the reference models were never run against your prompts and nobody ever scored their replies with your criterion. There is literally nothing on the other side of the comparison. So the report gives you the absolute number and stops. ### Why the absence bites An absolute rate is close to uninterpretable on its own. Is 22% high? Compared with what — a model that scores 5% on this family, or one that scores 60%? Without position, the reader supplies their own reference silently, and it is usually wrong in whichever direction their prior runs. For shipped families the tool supplies the reference; for custom families you must manufacture it, or explicitly refuse to imply one. ### The three honest closures 1. **Build your own reference spread.** Run the identical custom probe and identical detector against two or three other targets you are permitted to test: an open-weights model you host yourself, a second vendor endpoint, your own build from before a mitigation. This is the closest analogue to what the tool does and my default. Cost it honestly: it is the full prompts x `--generations` bill per target, and the slowest endpoint's rate limit sets the schedule for the whole spread, so three targets is not three times the wall clock — it is often much worse, because one of them throttles. 2. **Use a within-target baseline.** The same probe and detector against your own system with a mitigation off and on, or before and after a release. This answers "did we improve", which is usually the decision actually on the table, and it costs two runs against an endpoint you already control. 3. **Report the absolute number as absolute.** Criterion written out, error measured, no implied position. Boring, and honest. ### Where the number misleads Three specific misreadings, in rising order of damage. The first is **column contamination**: a report that lists shipped families with their positional readings and the custom family beside them, formatted identically. A reader scans down and treats them as commensurable. One is positioned against measured reference data; the other is an unvalidated criterion's output against one target. Different footnotes at minimum, different sections preferably. The second is **consistency mistaken for correctness**. Building your own spread makes the criterion *consistently applied*, not right. If the detector is wrong, it is wrong in the same direction on all three targets, and the spread comes out looking tidy and internally coherent — which is exactly what a wrong-but-stable ruler produces. Validating the detector and building the spread are two separate obligations; discharging one does not discharge the other. The third is **stale reference data**, which applies to shipped families too and is worth voicing because it is the same class of error. A bag of reference models measured some releases ago flatters anything current: your target looks unusually good against a comparison set that has since been superseded. A positional reading needs a date on it, the way a benchmark score does. ### What I would check Confirm from the report whether a positional reading was produced at all for the pair, rather than assuming a blank means a good result. If I am building a spread, hold prompts, detector and `--generations` identical across every target, and record model names, versions and dates alongside the numbers — a spread whose targets were run with different generations settings has different denominators and is not a spread. Confirm I am permitted to test each endpoint before sending anything. And keep the custom family's numbers in their own section, with the detector's measured error stated next to the rate.