An extraction run reports 97% agreement with a paid malware-verdict API - what do you ask before believing it?

## Why a bare agreement figure is not a finding Agreement - the fraction of inputs on which a fitted copy returns the same verdict as the target it was extracted from - is the right metric for extraction. It is also almost useless on its own, for exactly the reasons any evaluation number is useless without its columns. Three questions turn it into a claim. ## 1. Measured on what, and against what base rate? Agreement is a property of a *pool*. Say what the pool was and how it was drawn. Then say what the target itself answers on that pool, because that sets the floor: if the service returns benign for 97% of those inputs, a copy that always answers benign scores 97% agreement and has learned nothing. The failure mode is specific and common. An allocation that spends its whole budget on uniformly random or synthetic junk gets nearly-constant verdicts back; the copy learns the constant; and if the evaluation pool is drawn the same way, it reports the highest agreement of any allocation on the sheet. The tell is that the allocation with the *least* information per paid label produced the *best* number. That pattern is nearly always a base-rate artefact, not a discovery. ## 2. Was the evaluation pool fixed in advance and shared? Each allocation must be scored on the same pool, chosen before the campaign, drawn from the traffic the copy is intended to replace. If each row was scored on whatever it happened to submit, the rows measure different functions on different distributions and cannot be ordered against each other. This is where extraction write-ups fall apart most often, because the query pool is sitting right there and reusing it is free. ## 3. Aggregate or per class? A verdict service over real traffic is heavily skewed. A single aggregate agreement number is dominated by the majority class, and the entire question of whether the copy reproduces the target's *malicious* decisions can hide inside three percentage points. Break agreement out per class, or the number tells you about the easy side only. ## The cost columns, once the number survives Only after the above is it worth asking what the result cost: submissions, unit price, total spend, and how many submissions went to locating good inputs rather than to labelling them. That last one matters because with a top-1 verdict and no score, an attacker inferring which files sit near a boundary spends budget on the inference itself. A campaign that reports 50,000 submissions may have spent a meaningful slice of them on search, and comparing it to an allocation that spent all 50,000 on labelling a free in-domain corpus is comparing different things. ## What the sheet should look like A defensible reporting line names, for each allocation: the submission count, the unit price, the evaluation pool and how it was drawn, the target's base rate on that pool, aggregate agreement and per-class agreement. Anything less and the ranking of allocations is not established, whatever the headline says. ## Why this question is the one a red-teamer gets asked The chair here is the person writing the engagement report, or reading a teammate's. The deliverable is a claim about what a fixed budget bought - and the client's follow-up question is always some version of "so how good is the copy, really?". A number that collapses under "measured on what?" is worse than no number, because it will be quoted back. The honest report usually says something less dramatic and far more useful: on a fixed in-domain evaluation pool, with the target's base rate stated, this allocation reached this agreement overall and this much on the class that matters, for this spend. ## What a well-founded agreement number still does not establish Even a clean figure is a statement about the evaluation pool and nothing beyond it. It does not say the copy behaves like the target on inputs unlike anything in that pool; it does not say anything about the target's parameters; and it does not say the copy would remain in agreement after the service is retrained. Agreement is a snapshot of behavioural overlap on one distribution, priced in submissions - which is exactly what an extraction engagement is entitled to claim, and no more.