A report claims 94% agreement between an extracted copy and your code endpoint — what do you ask?

## Why a single agreement rate is almost never a claim An extraction result is normally reported as an agreement rate: the share of prompts on which the copy answered the way the original did. It is the right unit — extraction takes behaviour, so behaviour is what should be measured. But the number is a property of *a measurement*, and four choices inside that measurement can each move it by tens of points. A headline figure with none of them attached is not comparable to any other figure and should not be accepted. ## The four columns that must be attached **1. Agreement against what?** Two very different quantities get called the same thing. *Agreement with the original's replies* is the extraction measurement. *Correctness against ground truth* is a quality measurement, and a copy can score well on one and badly on the other. A copy that faithfully reproduces the original's mistakes has high agreement and mediocre quality; a strong independent model can have excellent quality and low agreement. Only the first says anything about what was taken. **2. Measured on which prompts?** Agreement is highest exactly where the attacker's queries landed. Evaluating on prompts drawn from the same pool the training replies were bought from measures the best case. The informative report gives slices: prompts like the query mix, prompts adjacent to it, and prompts deliberately unlike it. **3. Under what match rule?** For a classifier this is trivial — same label or not. For a code endpoint returning free-form text it is a decision: exact string match after normalisation, an equivalent patch, the same tests passing, or a judge's verdict. Those give different numbers on the same copy, and the reference is stochastic on top of that, since a sampled endpoint can answer one prompt two ways. Note the direction carefully: a deterministic copy emitting the endpoint's most likely answer can score higher against the endpoint than the endpoint scores against a second draw of itself, so self-agreement is not a ceiling — it is a note that the reference is noisy and the rule must say how that was handled. **4. Against what baseline?** Two models trained on similar public material already answer conventional prompts the same way. Whatever agreement the attacker's base model had *before the first paid query* was not bought by the extraction. A report that omits it credits the query budget with agreement that was free. ## Reading a result table Given per-slice numbers, the headline usually turns out to be the top row. The gap between the first and last rows is the actual finding: it says how narrow the copy is and, by implication, how narrow the attacker's query mix was. If the pre-query baseline row is high, most of the headline was never purchased at all. ## What the number cannot establish, however large - **Nothing about parameters.** Agreement is a relation between two functions. Reading out actual parameters is a separate attack with different preconditions, and no behavioural agreement rate is evidence for it. - **Nothing about the training corpus.** The copy was fitted on the attacker's prompts and the endpoint's answers to them. - **Nothing outside the eval distribution.** The claim's scope is the prompt mix it was measured on and no wider. - **Nothing about the copy's usefulness to its owner** unless the eval mix resembles the traffic they intend to serve. ## The limitations paragraph this produces A red-teamer writing the limitations section should be able to write, without hedging: *agreement was measured against the endpoint's sampled replies under a stated match rule, on prompts drawn from the queried mix and on two slices deliberately outside it; the base model's pre-query agreement is reported as a baseline; the result establishes behavioural imitation on the queried distribution only, and no parameters or training data were obtained.* Any claim broader than that sentence is not supported by an agreement rate, and any report that cannot fill in those clauses has published a number rather than a finding.