An auditor asks what share of your training corpus a human reviewed, and intake tripled while reviewer headcount did not - what do you tell them?

## What is actually being asked The auditor's question sounds like a metric request and is really a request for a **claim you will stand behind**. Whatever number you say becomes a sentence in a document you do not control, and the sentence will be read as assurance unless you attach the bound yourself. ## Decision one: which number, and with what beside it There are two candidates and they tell opposite stories. *Items reviewed* is flat and reassuring. *Share of intake reviewed* has fallen with growth. The flat count is the one that gets quoted, because it is stable and it makes the team look consistent - which it is. Report the share, with the trend and the reason: intake grew, staffed hours did not, nothing regressed. Volunteering that framing is what stops the finding from landing on the QA team, where it does not belong, instead of on the risk register, where it does. ## Decision two: the sentence you will and will not sign Write the bound yourself rather than leaving it to be inferred: - **Supported:** a stated share of intake received an item-by-item human verdict against the labelling guideline, and nothing individually implausible was found among those items. - **Not supported:** any statement about the unreviewed remainder. The review draws a share; against writes sized against that share, the expected number of poisoned items appearing in front of a reviewer can be below one, so a clean result is equally consistent with a clean corpus and with a sized attack. - **Also not supported:** any statement about patterns spanning items. Reviewers answer a per-item question, so a pattern that exists only across a set of individually reasonable items is not in the scope of what was checked. The hardest part of this is professional rather than technical. "We cannot assert that this corpus is unpoisoned" is an uncomfortable sentence to put in an audit record, and it is the true one. Saying it early, with the supported claim beside it, is far better than having it extracted later. ## Decision three: where the next unit of assurance spend goes The obvious move is to fund review back up to last year's share. Do the arithmetic in front of whoever is asking. Tripled intake means roughly triple the reviewer hours to hold the percentage, and it means that again next time the business grows. What that buys is a linear increase in the chance of *drawing* a poisoned item - and the drawn item still has to be recognised, which per-item review is structurally poor at, because each item was written to survive exactly that judgment. You are proposing a permanently growing cost for a linear gain against an adversary who can re-size their writes for almost nothing. That is a bad trade and you should say so rather than quietly accept the budget. The better answers point elsewhere. Who can write into the training queue at all is a much smaller and more auditable population than the corpus is, and it is where the exposure actually comes from. Whether anything in the pipeline ever looks at items *together* - by source, by arrival window - is a question about the unit of review, not its rate, and it is the half that a bigger sample does not touch. And there is a real strategic call about whether this deployment faces an actor who would size writes against a published QA rate at all: for many pipelines the honest answer is that review is priced against vendor sloppiness, that this is fine, and that the assurance claim should be scoped to say so. ## What good looks like in the room You give the falling share unprompted, you attach the bound to it in writing, you decline to convert it into a cleanliness claim, and you come with a view on where assurance should actually be bought rather than a request to triple a budget. An auditor can work with that. What they cannot work with - and what tends to become the finding - is a confident percentage with no statement of what it covers.