Verbatim Recall

## The situation An adversary has ordinary generation access to a large text model behind a paid API. They do not hold its weights and they did not build its training corpus. They want to show that some specific body of text — a licensed archive, a customer table, a private document set — was in that corpus and can be pulled back out word for word. They generate large volumes of candidate continuations, and somewhere in the pile is a span that looks exactly like the thing they are hunting for. The question is what that span is worth as evidence. The honest answer is: on its own, almost nothing. ## Why a realistic span is weak evidence A generative model is trained to produce text that is *likely*, and likely text is well-formed text. Formats are among the easiest things for it to learn, because a format is repeated across enormous numbers of examples that are otherwise unrelated: the digit grouping of an account number, the punctuation of a postal address, the character classes and length of an API-style key, the cadence of a legal clause. Having learned the format, the model will produce novel, fluent, entirely invented instances of it at essentially zero cost, because generating is cheap. So a span that *looks right* has established one fact: the model can write text of that shape. That is not the fact anyone cares about. The fact people care about is whether this particular string came back out of the weights rather than being assembled on the spot. The two are visually indistinguishable, and confidently reading the first as the second is the standard failure of an inexperienced extraction report. The asymmetry matters for cost as well. Candidate generation is nearly free, so an adversary can always produce more candidates. What is not free is telling recall from invention. That discriminator is the entire attack; the generation is a prelude to it. ## The two ways a candidate becomes a claim **1. Verification against the genuine artefact.** Somebody who holds the real record checks whether the span matches it, character for character, over a length long enough that coincidence is implausible. This converts the candidate into a fact and it is the strongest thing available. Note who can do it: whoever holds the ground truth. An auditor, a publisher or the data owner can verify; an outside adversary who never had the archive frequently cannot, which is why the same technical pipeline produces a *finding* in one pair of hands and only a *suspicion* in another. **2. A confidence comparison against a reference.** The adversary scores the candidate under the target model and under a second model that provably never saw the data in question, and keeps only spans the target fits far better than the reference does. Text that is intrinsically easy — boilerplate, common phrasing, formulaic structure — is easy for both models and cancels out. Text that is easy for the target and hard for anything else is the interesting residue. This is evidence, not proof: it says the target's fit on that string is anomalous in the way training on it would produce. Notice what the second method actually is. It asks, of one specific string, whether the model behaves as though it had seen it. That is a membership question, asked about a string the model itself supplied. Extraction and membership inference are not neighbouring topics; the extraction pipeline has a membership test bolted to its output, and the test is the part that does the work. ## The direction of the claim Getting the direction right is most of the skill here. - A fluent, plausible span establishes that the model can produce that shape. - A span the target scores anomalously confidently, relative to a reference that never saw the data, is evidence consistent with recall. - A span verified against the genuine artefact establishes that this text was in the training data. - None of the three establishes how much *else* is recoverable, and a candidate the pipeline failed to confirm is not evidence that anything is absent. Whether a given span was memorized at all was settled during training and is a separate question from whether an adversary can elicit and prove it; this one is about the eliciting and the proving. ## What this means in practice A red-team report that lists thousands of suggestive candidate spans is not a finding, it is a pile of generated text. A report that lists a small number of independently verified spans, states the threshold and the reference used, and refuses to extrapolate to the rest of the corpus, is a finding somebody can act on. The evidentiary value of an extraction run is its precision, never its volume.