What a Model Memorizes

## The claim being rejected "It is a statistical model, so it generalizes rather than memorizes" is the standard reassurance, and it is half of a true statement used to deny the other half. A trained model does both. The two are not competing descriptions of one behaviour; they are two different routes to the same objective, and which route a particular string travelled was decided during training. ## Why storage is sometimes the only route Training pushes down a loss over the training text. For text with structure — ordinary prose, idiomatic code, a common import block — the efficient way to lower that loss across millions of similar examples is a general rule, because the rule pays off on every example that shares the pattern. The specific instance need not be retained, and usually is not: ask for it back and you get a fluent paraphrase, not the original. Now take a random-looking 32-character literal sitting in a config file. Nothing about language predicts it. There is no rule that generates it, no pattern it participates in, no neighbouring example it generalises from. If the trained model assigns that string an unusually low loss — that is, finds it unusually unsurprising — that low loss has only one possible source: the parameters hold something specific about that string. Structure gets a rule; structurelessness gets stored, or gets nothing. The same argument runs on the other side of the distribution. A span that appeared hundreds of times across the corpus — a licence header, a config block copied into every service, a fixture reused everywhere — is pulled toward exact reproduction simply because so much of the objective depends on that one sequence. ## What the adversary needs Nothing unusual. The vantage that recovers a memorised span is the vantage every user of the product already has: send a prefix, read the continuation. No weights, no gradients, no returned probabilities, no privileged endpoint. That is what makes this different from most attacks in the adversarial-ML space, where the interesting question is what access buys what capability. Here the access is ordinary and the interesting question is what the corpus contained. What the adversary cannot do is add anything to the recoverable set. They cannot make the model store a string it did not store. They choose which prefixes to spend queries on; the training run chose what there was to find. ## Two things to rule out before calling it memorization **Context echo.** If the string was in the prompt, in the file the user had open, or in a document a retrieval layer pasted in, the model repeating it is not memorization at all — it is the model doing exactly what was asked. This distinction matters operationally, because only the second case means the string travels with the checkpoint wherever the checkpoint goes. **Confabulation.** A model asked to continue a plausible-looking prefix will happily produce a plausible-looking secret that was never in any corpus. Deciding which candidates are real recall is a separate problem with its own machinery; for this question it is enough to know that an unverified candidate is not evidence of anything. ## Why "the model does not store the training data" is technically true and useless A checkpoint is parameters. There is no file table, no index, no copy of the corpus, and no way to look up whether a given string is in there — which is precisely why this is hard to answer to anyone who asks. But "there is no stored copy" does not imply "nothing can be reconstructed", and disclosure only needs reconstruction. Both statements are true simultaneously and only one of them is about risk. ## The operational reframing Because the recoverable set is determined by the corpus and the training run, it is fixed before deployment. Nothing done at the endpoint changes what is in it — filters and refusals change what an adversary can conveniently reach, not what the parameters hold. So the useful thing to know about a model is not a property of the model at all. It is: which classes of string went into the corpus, how often, and what would it cost us if one came back out. That is a data-governance question wearing a machine-learning costume, and it is answerable, which the model-internal version is not. ## The interview-grade summary Memorization and generalization coexist. Duplicated and high-entropy spans are the ones stored. Ordinary generation access is enough to retrieve them. The set was fixed at training. And a well-generalizing model with healthy held-out scores is not exempt, because a handful of stored literals barely moves an average.