Reviewers sample collected training rows before each retrain — why is that not a bound?

## The answer that sounds like a control and is not Ask a team how they handle poisoning of a product that trains on its users, and the common answer is "we review the data before it is used." It is worth taking seriously, because review is genuinely useful — and then showing exactly what it does and does not establish. ## Review is bounded by hours; the other side is bounded by accounts and time Review capacity is a headcount multiplied by a throughput: how many rows a person can meaningfully judge per hour, times hours funded. That is a fixed number per week. Divide it by the batch size and you get a sampled fraction, and in any product with real traffic that fraction is small. The contribution side scales differently. It grows with the number of identities the contributor holds — a term priced in seats, not in effort — and with the length of the retrain window, and it resets at every retrain. Two quantities growing on different axes do not meet: doubling reviewer headcount doubles the sampled fraction once, while the other side is multiplied by whatever the account cost allows. This asymmetry is the substance of the answer. Anyone can say "review does not scale"; the credible version names *what* each side scales with. ## What a clean review actually establishes Be precise about the direction of the claim, because this is where candidates get marked down. A clean review establishes: - the **sampled** rows contained nothing the reviewers **recognised** as bad, - under whatever criteria they were given, - for **this** batch only. It does not establish that the batch is free of rows placed to change a behaviour, that the retrained model will behave like the last one, or that contributions were spread evenly across accounts. Each of those is a different measurement. ## The part that has nothing to flag Even a hypothetical full review runs into a harder problem. Reviewers catch rows whose label contradicts their content — that is what a labelling queue is good at. But when the product derives the label from behaviour, no contradiction exists. Content that was genuinely produced and genuinely accepted is genuinely labelled as accepted. Each row is individually unobjectionable; what carries the effect is which rows there are and how many, not any single one being wrong. This is the general shape of correctly-labelled contributions: there is no anomaly at the row level to detect, so a per-row inspection process is looking at the wrong unit. ## The measurements that actually say something If review is a rate rather than a bound, what does bound it? Quantities about the *composition* of the batch rather than the quality of individual rows: - what share of the batch came from any single account, and from the top handful of accounts; - what share of a **narrow context slice** — the rows touching the behaviour you care about — came from one channel; - how much of the batch came from accounts younger than one retrain interval; - whether contribution concentration changed between batches. Those are questions a data platform can answer, and each of them has an obvious design lever behind it: a cap on how much any identity may supply to one batch, a floor on how many distinct sources a context slice must draw from, and a cost attached to opening an account. None of that is a scanner; it is collection design. ## How to deliver this in an interview Do not dismiss review — say what it buys. It catches sloppy, obvious and accidental contamination, it gives you labelled examples of what bad looks like, and it is how you learn the criteria in the first place. Then state the limit plainly: it covers a sampled fraction, it is priced in hours, and it is blind by construction to rows whose labels are honest. Finish on the design question, which is the one the interviewer wants: *what fraction of a batch may one uncontrolled channel supply, and do we measure it?*