A team says an attacker who never obtains their model's weight file cannot breach its confidentiality. Why is that wrong?

## The claim, and what is wrong with it "Nobody has our weights, so our model's confidentiality is fine" is the most common wrong answer in this part of a threat model, and a competent engineer gives it, because for almost every other system the reasoning holds. A database is confidential in the sense that its file and its connection are guarded; if neither was reached, nothing was disclosed. A trained model breaks that intuition, because the artefact is not the asset. ## Two confidential assets, neither of which is a file **The training data.** A model is a fitted summary of the data it was trained on. Fitting is never perfect and never uniform: the model's behaviour on examples it saw differs measurably from its behaviour on examples it did not, and that difference is visible from outside in the model's outputs. Whole classes of attack live in that gap: inferring whether a particular record was in the training set at better than the base rate; inferring a missing field of a record the adversary already mostly holds; recovering what the model considers a typical member of a class; eliciting spans the model reproduces close to verbatim. None of these requires the artefact. Some require only the returned decision. **The learned function.** Separately from the data, the model's input-output behaviour is often the thing the operator paid for. An adversary who can query it can build a stand-in that agrees with it well enough for their purpose. Again the file never moves. This is a confidentiality goal too, and one whose harm is commercial rather than personal. So "confidentiality" for a model splits: a **privacy** asset (facts about the people and records behind the training data) and a **proprietary** asset (the behaviour itself). The weight-file reading covers neither, because it protects the *encoding* of both while leaving the interface that exposes them wide open. ## Why query access is enough The short version is that a model is not a lookup table but it is not independent of its training set either — it sits somewhere in between, and where exactly is a property of how well it generalised. A model that generalises perfectly would treat a member of its training set exactly as it treats a fresh example from the same distribution, and there would be nothing to infer. Real models do not, and the residual is the leak. That is why this exposure is a *consequence of imperfect generalization* rather than an implementation bug: there is no patch that removes it, only training-time mechanisms that bound it and pay for the bound in accuracy. This also explains why the exposure does not care about the deployment shape. An on-premises model behind a private VPC leaks the same information through its predictions as a public endpoint does, at whatever rate the caller is allowed to query. ## What the weights-only reading actually buys It is worth being precise rather than dismissive, because guarding the artefact is genuinely useful: - Holding the file back removes the strongest adversary — one who can compute exactly, without paying per query — and forces everyone else to buy their information through the interface, which is meterable, loggable and rate-limitable. - It also protects the *cheapest* route to the learned function, which is simply having it. What it does not do is bound disclosure. "No one has our weights" answers the question "was the artefact exfiltrated?" and nothing else. Two different pieces of evidence are needed, and they come from different places: artefact custody from access logs and egress controls, output disclosure from an actual measurement against the query interface. ## Saying it correctly in an interview A good answer names the asset, then the route, then the measurement: 1. **Asset**: the training data (and separately the learned function), not the file. 2. **Route**: the query interface, available to anyone the product lets query it, including a paying customer. 3. **Measurement**: advantage over the base rate on a stated fact — not "a file was copied", and not an accuracy figure on its own. And then the honest caveat, which is what separates a strong answer from a scary one: the *harm* is set by what the inferred fact means about a person, not by the size of the number. Learning that a record was in a corpus of general product reviews is close to worthless; learning that the same record was in a corpus of people who sought a particular kind of treatment can be serious at the same measured advantage. The threat model has to say which corpus it is before anyone can rate the finding. ## The other direction, too The converse error is just as common and worth flagging: a verified, signed, correctly-custodied weight file tells you which bytes you have and who published them. It says nothing about how those weights behave. Custody and behaviour are different questions with different evidence, in both directions.