A retrained ranker's per-segment quality report is all green — what does that establish about an adversary?

## What is being read A per-segment quality report compares a newly retrained model against the previous one, broken down by a handful of segments a team decided to track. It is the most granular instrument most teams have, and it is the one that gets quoted as "we checked and everything is fine." Understanding what it can and cannot bound is the whole of this question. ## Scope one: granularity The report constrains movement in the segments it prints. Every slice finer than those segments is inside one of them, averaged with everything else there. If a segment covers eleven million events and the affected slice covers eighty thousand, an effect that is severe on that slice moves the segment's number by roughly the slice's share of it — a rounding error against the threshold anyone would alert on. This is why a narrow adversary does not need much restraint. They are not sneaking under a threshold by careful tuning; the report's own structure averages them away. Restraint is required only when the effect they want is broad. ## Scope two: resolution Each printed number is an estimate from a finite sample, so it carries noise proportional to that segment's size. A small segment's figure bounces around run to run, which means two things at once: it reads green under mild damage as easily as under health, and it also raises false alarms, so teams widen its alert band, which lowers its sensitivity further. Splitting into more, smaller segments does not give you a free increase in coverage — it trades granularity against resolution, and there is a floor set by how much traffic each slice actually gets. ## The direction of the inference Stated honestly, all green establishes: *no effect exceeding the resolvable noise of each reported segment, in the segments reported, over the compared window.* That is genuinely useful. It rules out a crude broad degradation. It does not rule out an effect that is confined, one that is inside an unreported slice, or one small relative to a noisy segment's band. The error to avoid is the inversion: reading the absence of a signal from an instrument as evidence about a thing the instrument does not measure. A green report is consistent with no adversary and with a competent narrow one, and by itself it does not discriminate between them at all. ## The conversation this leads to Asked what the report is worth next quarter, the useful answer is comparative rather than absolute. It will keep catching broad degradation and clumsy or unlucky campaigns, which is a real class. It will not start catching a confined effect, because that is a structural property of averaging and not a tuning problem. Buying more segments buys resolution down to the point where each slice's traffic runs out, and buys nothing below it. Anything below that floor needs a different activity — evaluating behaviour on inputs chosen because you suspect them, rather than on a sample of ordinary traffic — with its own scope and its own limits. ## Triage, when a slice does look off The symmetric error is over-reading a red. A single segment moving is far more often an ordinary cause — a traffic-mix change, an upstream logging change, a seasonal shift, a genuine data-quality bug — than an adversary. The useful discipline is to ask what would distinguish those, which usually means checking whether the movement tracks something exogenous, whether it persists across retrains, and whether the affected population has any relationship to who could write into the corpus. Concluding "poisoning" from a moved number alone is as bad as concluding "clean" from a flat one; both replace a bound with a verdict. ## What to say when signing off If you are the person whose name goes on the check, phrase the conclusion with its scope attached: which segments were compared, over what window, with what alerting band, and the explicit statement that effects confined below that granularity are not covered. That sentence costs nothing and prevents the report from being quoted later as an assurance nobody made.