The Confidence Gap

## The claim, and the hole in it "The model does not store the training data" is true as stated and answers a different question than the one being asked. A trained classifier is a set of parameters plus an architecture; it is not a table of the complaint tickets it was fine-tuned on, and you cannot index into it and pull one out. What it *is*, though, is a function that was adjusted over and over specifically to reduce error on those tickets. Fitting is not storage — but it is not nothing. The parameters sit where they sit *because* those particular rows pulled them there, and each row's pull leaves a trace in how the finished function behaves when you show it that row again. ## The measurable consequence For essentially every trained model, error measured on the rows it was fit to is lower than error on fresh rows from the same distribution. The same asymmetry shows up in the numbers a product returns: on an input it was trained on, a classifier tends to put more probability on the answer it gives, and its per-example loss is lower. That difference is a property of the fit, and it is what the whole membership-inference family runs on. ## What an outsider can actually do with it The actor here is not the operator and does not own the model. Picture an internal complaint-triage classifier, fine-tuned on one employer's closed corpus of resolved tickets and exposed as a product feature that returns a handling track and one confidence number for that track. An adversary holds a candidate ticket — they wrote it, or they hold a copy of it — and wants to know whether that ticket was in the training corpus. Their vantage is ordinary product access. Their limit is one query per candidate record: no weights, no gradients, no full probability vector, no per-feature explanation. They send the ticket once and read the response. Records the model was fit to skew towards "right track, higher number"; records it never saw skew towards "less confident, more often wrong". Comparing that response against a threshold turns the skew into one bit about one record. In the literature this family is called membership inference. ## Why the storage answer does not touch it Not storing rows closes exactly one door: verbatim recovery of content. Membership inference never asks for content. It computes a statistic from the model's *behaviour* and reads presence off it. You can delete the training corpus the day training finishes, keep no logs, serve the model from a tensors-only weight file, and the gap is still there, because the gap is in the weights' behaviour and not in any retained copy of the data. A second common deflection — "the outputs are aggregates, aggregates are anonymous" — fails the same way. The response is not an aggregate over the corpus; it is the model's reaction to *this specific record*, and the reaction is conditioned on whether the model was fit to it. ## What bounds it The attack's average accuracy over a pool of candidates tracks how much better the model is on what it saw than on what it did not. That is a property of the training run, and the adversary does not control it: they did not choose the corpus, the capacity, the number of passes over the data, or anything else that sets it. They can only read the gap that exists. So the leak has a ceiling the model's owner sets, and a floor that is not zero for any model that overfits at all. One more thing follows from that, and it is the part candidates most often miss: the average is not the whole story. A model does not overfit uniformly. An unusual ticket — odd vocabulary, a rare category, a small subgroup — is often fit far more tightly than a typical one, so its response stands out much further from the non-member distribution. An attack that is near chance averaged over thousands of candidates can still be near-certain on a handful of them, and it is those records that make the finding a privacy problem. ## The bit itself The payoff is deliberately small: one bit about one record. No text of any ticket comes back, no field is reconstructed. The reason it matters is that presence is the fact being asked about, and presence was supposed to be unknowable to anyone outside the team. ## What a good answer sounds like "We do not store rows, but we fit them, and the fit is stronger on what we saw. That difference is readable from the confidence we return, at one query per record. The size of the average leak is set by how much our model overfits — which is our choice, not the attacker's — and the average understates what an attacker gets on the unusual records."