A clean-label poisoning result reproduces on one retrain in five — how do you triage it?

## The finding on the table Someone re-ran a suspected poisoning result against a classifier retrained from a sample-sharing feed. On one of five retrains, a chosen file was read the way an adversary would want; on the other four it was not. Overall validation accuracy did not move on any of the five. The instinct in a triage queue is to mark it not reproducible and close it. That instinct is wrong here, and the reason is mechanical. ## Why the flake is the signature A clean-label effect is *aimed*. The contributed rows carry correct labels, so the only leverage is where their content sits relative to the boundary the training run will learn, and the adversary chose them by reasoning against a stand-in model rather than the deployed one. Two approximations therefore stack: - **The stand-in is not the target.** It agrees near the region of interest only approximately, and it drifts further with every retrain the defender does. - **The target is not one function.** Each retrain reaches a different fit — a different seed, a different data order, a different mix of that period's honest arrivals. Two runs on nearly the same corpus can differ on exactly the marginal cases the attack is aiming at. Aim with an approximation at a moving target and you get a probability, not a certainty. One in five is a perfectly ordinary value for a real effect. The contrast is what makes the point land. Label-flipping asserts a falsehood in the label column; that pushes the fit in a direction on essentially every run, which is why the cheap, reviewable build is also the stable one. **Stability across retrains is evidence about which build you are looking at, not about whether the finding is real.** ## What to measure instead of "does it reproduce" Treat the rate as the quantity of interest. - **Rate over enough reruns to mean something.** One-of-five is a very wide interval; a rate estimated from five runs cannot distinguish a 20% attack from a 40% one. - **Does the rate move with volume?** If contributing more rows raises it, that is strong evidence of a real mechanism rather than a coincidence of one fit. - **Does the rate move with the honest data mix?** An effect that survives a substantially different period's arrivals is more robust than one that does not. - **What is unchanged?** Flat aggregate accuracy across all runs is consistent with a targeted effect and is not exculpatory. Reporting "accuracy was fine" as a reason to close is the classic direction-of-claim error: it says the average was preserved, which is what this class of attack wants. ## Then make it economic The defender's unit of exposure is not one retrain, it is the retry loop. A submitter who can contribute every cycle faces a repeated trial at whatever the per-retrain rate is, and often needs to win once. So the two numbers that turn this finding into a decision are **the per-retrain success rate** and **how often the adversary gets to retry** — the retrain interval and their submission access. A 20% rate against a weekly retrain is a very different exposure from a 20% rate against something refit twice a year. The corresponding cost on their side is the perishability described above: their stand-in decays, so each retry is aimed slightly worse than the last unless they can refresh it. That is a real limit and it belongs in the write-up next to the rate. ## How to write it up Three sentences that do not overclaim in either direction: 1. The effect was observed on a fraction of retrains, and that fraction — not a yes/no — is the finding. 2. Flat aggregate metrics are consistent with it and provide no evidence against it. 3. Exposure is the rate multiplied by the retry opportunities the submission path allows. What you must not write is "could not reproduce, closing", and what you must not write is "the model is backdoored". The first discards a real signal because it arrived as a probability; the second asserts a mechanism the evidence does not establish. A finding that lands once in five is a measured rate, and rates are exactly the kind of thing a security review is supposed to be able to carry.