An attacker compares a classifier's returned confidence to a threshold — what caps that attack's accuracy?

## What is actually being thresholded A membership test at this vantage is a comparison, not a reconstruction. The adversary holds a candidate record — say one complaint ticket out of an internal triage corpus — and has ordinary product access to the classifier: send an input, get back the chosen handling track and one confidence number for that track. Nothing else. No probability vector over all tracks, no weights, no gradients, no per-feature explanation. From that response they can form a small statistic. Because they hold the record, they know (or can infer) what track it should get, so they can see both whether the model was right and how confident it was. Records the model was fit to skew towards low error and high confidence; records it never saw skew towards higher error and lower confidence. The test compares the observed statistic to a threshold and emits one bit. That is the entire mechanism, and its economy is the point: **one query per record**, no training access, no special vantage. ## Where the ceiling comes from The two distributions — responses on members, responses on non-members — overlap. How much they overlap is exactly how much the model generalizes. A model whose behaviour on fresh data is indistinguishable from its behaviour on training data presents two distributions that sit on top of each other, and no threshold placed on them beats a coin. A model that fits its corpus much more tightly than it fits the world presents two separated distributions, and a threshold placed between them is informative. So the average accuracy of the attack is bounded above by how much the model overfits. And here is the part that matters in an interview: **the adversary does not control any of that.** They did not choose the corpus size, the model capacity, the number of passes over the data, or the regularization. They are reading a property of somebody else's training run. Extra sophistication at inference can extract that property more efficiently; it cannot manufacture separation that the fit did not create. The same logic sets the floor. Because the separation is a consequence of imperfect generalization, and no useful model generalizes perfectly, the leak is not zero for any model that overfits at all. It gets small; it does not become impossible by ordinary means. ## Two ways the raw number misleads **The base rate.** "The attack was 70% accurate" is uninterpretable without the share of the candidate pool that were really members. If a pool is balanced 50/50 by construction — which is how such evaluations are usually built — then 70% is a 20-point advantage over guessing. In a realistic setting, where the attacker's candidate pool may be overwhelmingly non-members, the same raw accuracy could be worse than always answering "not a member". Membership inference is measured as advantage over the base rate, never as bare accuracy. **The average.** Overfitting is not spread evenly across a corpus. A ticket with common phrasing in a large, well-represented category sits in a region the model would have learned anyway, so its response barely differs from a non-member's. A ticket with rare vocabulary, an unusual category, or belonging to a small subgroup can be fit very tightly, and its response can be far outside anything a non-member produces. The consequence is that the *mean* accuracy of the attack understates the worst case badly. An attack near chance on average can still be nearly certain about a small set of records, and privacy harm is a per-record property. ## Where more attacker effort does and does not help More queries per record buy precision on the same underlying statistic — averaging away noise in the response, or probing to locate the record relative to a decision boundary. That is a genuine improvement in *reading* the signal, and it is why a stated query limit belongs in any threat model. What it cannot do is change the separation between the member and non-member distributions, because that was fixed when training ended. Likewise, a better-placed threshold — for instance one set per predicted class rather than one global cut — extracts more of the available signal without adding any. ## Where the defender's lever actually is Since the ceiling is the gap, the defender's only ordinary lever is shrinking the gap: fewer passes over the data, more data, less capacity relative to the corpus, anything that makes behaviour on training rows resemble behaviour on fresh rows. That reduces the *average* attack meaningfully — and it bounds nothing, because it does not stop an individual atypical record from being fit tightly. A bound, rather than a reduction, requires a training-time guarantee stated with its own parameters and accounting, which is a different subject entirely. ## A crisp answer "The threshold reads the difference between how the model behaves on rows it was fit to and rows it was not. The average edge is capped by that difference, which is set by the training run and not by the attacker. Report it as advantage over the pool's base rate, and do not let the average stand in for what happens to the unusual records."