Why does "the model never stores training records" fail to answer whether it reveals who was in its training set?

## What is actually being asked When someone asks whether a deployed model "reveals" its training data, engineers reach for the storage frame: the weights are numbers, no rows were copied into them, therefore nothing can come out. That frame answers a different question. The question on the table is whether an outsider can learn a **fact about a person** by interacting with the model, and one such fact is simply **that the person's record was in the training set**. This is called *membership inference*, and it is the smallest privacy attack there is: the adversary wants one bit. ## Why the imprint exists at all Training fits a model to a specific sample. On the examples it was fit on, the fit is tighter than on comparable examples it never saw, because that is what fitting means. Any model that generalizes imperfectly, which is every useful model, therefore behaves at least slightly differently on training members than on non-members. The behaviour is visible in whatever the deployment returns: a confidence figure, a ranking, or even just how stable the returned class is. Nothing was stored; a statistical trace was nevertheless left, and it is readable from the outside. ## Who the adversary is, and what they are limited to The adversary in this scenario is not exploring. They already have a named person's complete record: demographics, history, everything the model consumes. They do not want the record back; they have it. What they lack, and want, is confirmation that this person's record was part of the corpus this model was trained on. Their limit is unusually cheap. They need **one query per named person**, and on a deployment that returns only a risk band they need nothing more than the returned band. There is no reconstruction, no optimisation, no sample of auxiliary data required for the basic move. That cheapness is the whole reason this class of finding is taken seriously even when the measured edge is small. ## Why membership can be the entire harm A training set has a *definition*, and the definition can itself be the sensitive fact. If a model was trained on one specialty clinic's attending patients, then "this record was in the training set" means "this person attends that clinic", which is a statement about their health. If a model was trained on accounts a platform flagged for fraud review, membership means "this account was flagged". If it was trained on one insurer's claimants, membership means a claim exists. Contrast that with a model trained on a public image benchmark. An adversary who establishes with near-certainty that a given picture was in that benchmark has learned nothing about anybody. The mechanism is identical; the harm is not, and the difference lives entirely in what the corpus means, not in the model, the architecture, or the attack's accuracy. ## What the attack does not give the adversary Be precise about the shape of the leak, because overstating it is as much a mistake as denying it. - It is a **confirmation oracle over candidates the adversary can already name**. It does not enumerate the training set, and it does not hand anyone a roster. - It returns **presence**, not contents. Field values, notes and outcomes are not recovered by this move; recovering something about the contents is a different attack with different preconditions. - It needs **query access** to the deployment, by the adversary, with an input of their choosing. A model reachable only by authenticated clinicians faces a much smaller set of adversaries than one behind an open endpoint, and that is a real difference in exposure even though the model's imprint is unchanged. ## What to say instead of the storage sentence The defensible statement has three parts. First, describe the fact at risk: not "training data", but "whether a named individual attends this clinic". Second, describe the adversary concretely: someone who can query the deployment and already holds the candidate record. Third, say what you have measured about the edge such an adversary gets, and under which access assumption you measured it. "We do not store records" is not one of those three parts, because a model that stores nothing can still be interrogated one name at a time. ## The related trap The same storage frame produces a second wrong answer: that removing a person's row from the data store settles the matter. Deleting the row changes the store. The weights were already fit on it and are unchanged by the deletion, so the imprint that the attack reads is still there until the model is retrained without that record, or some other claim is made about the model itself. Distinguishing "gone from the database" from "gone from the model" is the whole point of taking membership seriously.