Candidate generation is nearly free for an extraction attacker — so what actually binds the run?

## The cost structure is upside-down from the intuition Asked what limits a training-data extraction campaign, most people reach for compute: how many candidate spans can you generate. That is the cheap half. Generation against a paid endpoint costs a small amount per span and parallelises trivially, so the candidate pool can be made as large as anyone wants. The expensive half is deciding which candidates are real. A well-run extraction pipeline therefore has a very particular shape: an enormous, nearly free generation stage; a scoring stage that ranks candidates by how much better the target fits them than an independent reference does; and a small, slow, expensive verification stage where a human with access to the genuine source confirms a shortlist. The middle stage exists only to make the last stage affordable. ## The threshold is a price, not a hyperparameter Everything about the run is set by where the score threshold sits. - **Lower it.** More candidates are flagged. Precision falls, because the additional candidates are drawn from the part of the distribution where the target's advantage over the reference is small. Each confirmed hit now costs more verification time, since the reviewer wades through more misses per success. - **Raise it.** The shortlist shrinks and precision climbs, but genuine hits with a modest confidence gap are discarded unseen and never appear in the report. The run is priced in verification, so the threshold is set from the verification budget backwards: how many candidates can a reviewer with archive access actually check, and what precision do they need for that to be worth their time. Choosing it from the candidate pool forward — flag the top ten thousand because we have ten thousand — is how these exercises turn into an unreviewable pile. ## Who can verify is part of the limit Verification requires the genuine artefact. That means the run's ceiling depends on the chair you are sitting in. An internal reviewer or an auditor granted archive access can confirm candidates and produce findings. An outside party holding no copy of the source can rank candidates but confirm none, so their strongest honest deliverable is a ranked hypothesis list with the confidence gap attached and no claim of leakage. Failing to say which of those you produced is the most common defect in an extraction write-up. ## Recall has no denominator The pipeline can tell you what it confirmed. It cannot tell you what fraction of the archive is recoverable, because it never enumerated the archive's presence in the corpus and never exhausted the generation space. A run that confirms forty spans confirms forty spans; it does not measure a leak rate, and a document with no confirmed hit has not been shown absent. Anyone reading the output as a coverage percentage is reading something the run never measured. ## The vantage can be taken away, at a price The reference comparison needs the target's own confidence on the candidate. If the endpoint stops returning per-token log-probabilities, that input disappears, and the adversary is left ranking on the reference alone — which ranks how intrinsically likely the text is, precisely the wrong quantity. The shortlist collapses back into unranked generated text and the verification stage becomes unaffordable. That is worth stating carefully, because it is a cost control rather than a boundary. Removing scores does not make anything unmemorized and does not make text unrecoverable; it removes one cheap discriminator and forces the adversary onto more expensive ones, which for a campaign that lives or dies on precision is a substantial bill. It is also not free to the model owner: log-probabilities are a legitimate product feature for evaluation and calibration work, so removing them is a trade against real users, not a costless hardening step. ## How to answer this in an interview State the inversion first — generation is free, discrimination is not — then name the two quantities that actually bind: the filter's false-positive rate at the chosen threshold, and verification effort per confirmed hit. Then add the two limits that shape what may be claimed: recall is unmeasured, and verification requires whoever holds the source. Someone who answers in terms of GPU hours or rate limits has not understood where the work is.