How do you tell whether tightening a training-data outlier screen removed poison or your rare real records?

## Why the screen cannot answer the question about itself A sanitization stage emits one number that people treat as evidence: the fraction of the capture it dropped. Nothing about that number separates the two populations you care about. The dropped slice is a mixture of genuinely rare real records, measurement junk, and whatever poison was careless enough to look unusual. The retained bulk is a mixture of ordinary real records and any poison written to sit inside the distribution. Raising the threshold moves records from the second mixture to the first; it does not tell you which kind moved. So *we now drop three per cent instead of one* is not a finding. It is a change to a cost you have not measured. ## What is actually measurable **Characterise the removals.** Pull a sample of the newly dropped records and describe what they are, at whatever granularity your data allows - protocol, port range, duration band, source class, time of day. You are asking one question: is this slice dominated by legitimate behaviour the monitored source really produces? On a corpus captured from a live segment the answer is very often yes, because the tail of a traffic distribution is full of real-but-uncommon things: a monthly bulk transfer, an unusual protocol from one host, a long-lived session. If the removals look like that, the tighter cut bought nothing and spent real coverage. **Measure the cost where it lands.** Aggregate accuracy on a held-out sample drawn from the same distribution is the wrong instrument, because the tail is by definition a small fraction of that sample and so barely moves the average. Build an evaluation set that deliberately over-weights the rare behaviour the model exists to recognise, and read recall on it before and after the threshold change. That is where the bill from a tighter cut shows up, and it is the number the model's owner cares about. **Look for the effect you were trying to buy.** If tightening was motivated by a suspicion of poisoning, the point was to change the model's behaviour on whatever the poison was aimed at. If you have a hypothesis about that - a class whose detection degraded, a narrow condition that fires oddly - test the retrained model on it. If you cannot state what should have improved, you cannot claim the tightening improved anything. **Look where the screen does not.** The one signal that genuinely separates the populations is not in the feature values at all: it is where the records came from and how many arrived together. Volume from one source, a burst that appears across consecutive capture windows, a contributor whose records concentrate in one region of the space - none of these are visible to a per-record unusualness score, and all of them are visible in the pipeline that collected the data. This is the direction with headroom, and it is why the answer to a poisoning worry is more often about write access than about scoring rules. ## The second-order effect worth naming out loud An attacker whose goal is a worse monitor gets two paths to it, and the defender controls the second. The first is the poison itself. The second is your response: a screen tightened until it deletes the rare real traffic the model needed is a degraded monitor produced by your own hand, at no further cost to the attacker. When the proposal on the table is *turn the filter up*, someone should say this. The right question is not whether a tighter threshold catches more poison in a test, but what the same effort buys elsewhere - metering who can write into the corpus, keeping per-source provenance so removals can be traced, holding out an evaluation set the corpus cannot influence. ## How to report the result Write the finding as three lines, none of which is a drop count on its own: - what the newly dropped records were, from a sampled characterisation; - what the change cost on rare-class recall, from an evaluation set built to show it; - what, if anything, changed in the behaviour you suspected was poisoned. If the first line says *mostly real*, the second says *recall fell*, and the third says *nothing observable*, the tightening was a self-inflicted loss and should be reverted. Reporting it as *we now remove more anomalous records* would have hidden all three. ## The trap in one sentence A screen's removals cannot validate a screen, because the quantity it scores is not the quantity you are asking about - and the only honest way to price a tightening is against the rare real behaviour it deletes first.