In a training-data extraction attack, why score each candidate span against a reference model that never saw the corpus?

## What the filter is for An adversary with generation access to a large text model can produce candidate spans without limit; generation is nearly free. What is scarce is a way to tell which candidates came out of the training data and which the model composed on the spot. Everything interesting in an extraction pipeline lives in that discriminator, and the discriminator is a score comparison. ## Why the target's own confidence is not enough The obvious filter is to keep the spans the model scores most confidently — lowest loss, lowest perplexity. It fails badly, and the reason is instructive: a language model assigns high confidence to text that is *intrinsically* likely, and intrinsic likelihood has nothing to do with memorization. Text that scores confidently under any competent model includes: - boilerplate and legal formulae that recur everywhere; - highly structured formats, where each character is nearly determined by the previous ones; - repetitive or degenerate sequences, which are trivially predictable; - extremely common phrasing and widely reproduced passages. Rank a million candidates by the target's confidence and the top of the list is dominated by exactly this material. It is a filter that mostly selects for *unremarkable*, and its false-positive rate is the reason naive extraction runs produce nothing usable. ## The comparison The repair is to remove the part of the score that any model would assign. Score the candidate twice: once under the target, once under a reference model that provably never saw the corpus under investigation. Keep spans where the target is far more confident than the reference. What this does conceptually is subtract intrinsic likelihood. Boilerplate is easy for both, so it cancels. A common quotation is easy for both, so it cancels. A span that is *hard in general but easy for the target specifically* does not cancel — and being unusually easy for one model in particular is precisely the fingerprint that fitting that span during training would leave. The reference does not have to be a second trained model, though that is the cleanest choice. Any yardstick for how compressible or predictable the text is in general serves the same purpose: a much smaller model, a model of a different lineage trained on a disjoint corpus, or a generic text compressor's code length for the span. All of them answer the same question — how surprising is this string in the abstract — so that the target's surprise can be judged relative to it. ## The preconditions, which is where candidates get caught **The reference must genuinely not have seen the data.** If the reference was trained on an overlapping crawl that also contains the archive, both models find the span easy and the gap collapses. You have not proved absence of memorization; you have destroyed your instrument. For a specific licensed archive this is arguable; for anything widely mirrored on the open web it is often impossible to establish, which is a real limit on what these runs can be pointed at. **The reference must be roughly comparable in capability.** Against a much weaker reference, the target is more confident about nearly everything, so nearly everything looks memorized. The gap then measures general capability difference, not corpus membership. Practitioners partly control for this by using several references, or by comparing a span's score to the same model's score on a perturbed or re-cased variant of the same text, so that capability is held constant. **A gap is evidence, not proof.** A large gap says the target's behaviour on this string is anomalous in the direction training would produce. It does not say the string is a training record. Confirmation still comes from matching the span against the genuine artefact, which is why an extraction run without a verification oracle produces ranked hypotheses rather than findings. ## Why this makes extraction and membership one family The comparison above is a membership-inference test. Membership inference asks, of a record you hold, whether the model behaves as though it trained on it, and the classic signal is exactly this: the fit is stronger on what the model saw, judged against what fit you would expect otherwise. Extraction differs only in where the record comes from — the model generated it rather than the adversary supplying it — and then runs the same test. That framing is useful in an interview because it predicts the failure modes. Membership inference is meaningless without a base rate to compare against; extraction filtering is meaningless without a reference to compare against. Both are relative measurements dressed up as absolute ones, and both fail the same way when someone quotes the raw number without the comparison it was made against. ## What a candidate should be able to state The filter is the attack. Its input is a confidence gap between the target and an uninvolved reference; its output is a ranked shortlist; its validity rests on the reference being independent of the data and comparable in strength; and its result is evidence that still wants verification against the real source before anyone calls it a leak.