A chat model declines one variant of an attacker's probe and answers another: what does either single observation prove?

## Why the same input gives different replies Generation samples. Unless a deployment pins decoding to the single highest-probability token — and consumer chat products generally do not — the reply is drawn from a distribution over continuations. Refusal lives inside that distribution rather than beside it, so an input near the boundary can produce a decline on one draw and an answer on the next without anything having changed. And sampling is not the only source of movement. The system behind the chat box is not guaranteed to be the same artefact from one probe to the next: a deployment can change, and the surrounding instructions the product wraps around the conversation can change too. From outside, none of that is announced. All the prober sees is that an input which behaved one way an hour ago behaves differently now. ## What a single observation licenses Precision about direction matters more here than anywhere else in a mapping pass. | Observation | What it establishes | What it does not | | --- | --- | --- | | One decline | this draw produced a refusal | that the neighbourhood is reliably covered | | One answer | this draw did not refuse | that the neighbourhood is uncovered | | One hedge | this draw sat between the two | any rate at all | Every row says the same thing: the unit of evidence is a rate, and a rate needs repeats. An attacker who records binary outcomes from single probes is building a map out of coin flips, and will read noise as structure — usually as "this topic is open" on the strength of one lucky draw, which then fails to reproduce for anybody else. ## Where the information actually is A probe that has declined on every one of ten attempts is deep inside covered territory; the eleventh attempt tells you almost nothing. A probe that has answered every time is deep outside it; same conclusion. The probes worth repeating are the ones whose outcomes disagree with each other, because that is where the decline rate sits away from the extremes and where an extra sample moves the estimate most. That is the whole allocation principle: sweep wide with single probes to find disagreement, then concentrate repeats on the disagreeing band. ## Why this is expensive rather than merely annoying On a consumer chat product the vantage point is an account and the account is metered. Every repeat is one message from a finite quota, so resolution and breadth trade directly against each other: ten repeats each across twenty neighbourhoods costs the same as one probe each across two hundred. There is a second cost too — an account that burns its quota on obviously repetitive probing is an account that may not survive to finish the map, and the map dies with the vantage point. ## The control probe Because the deployment can move underneath a mapping pass, a mapper carries at least one probe whose outcome is already well established and re-issues it periodically. If the control starts behaving differently, the change is in the system, not in the new probes, and everything measured since the last good control is suspect. Without a control, a mid-pass change looks exactly like a discovery. ## The misreadings to avoid The first is treating one decline as proof of coverage — the same crisp-boundary assumption that underlies the banned-list picture, just applied from the other side. The second is treating one answer as a finding: a construction that worked once against one deployment on one draw is not yet a reliable result, and reporting it as one is how a red-team report acquires items that nobody else can reproduce. The third is attributing variance to the wrong cause: before concluding that a phrasing difference moved the outcome, the mapper needs enough samples on both phrasings to show that the difference is bigger than the draw-to-draw noise. What survives all of this is modest and useful: a decline rate per probe, with a sample count attached, dated to a window. That is the honest unit a coverage map is built from.