Does hiding confidence scores stop an outsider learning who was in a model's training set?

## The proposal and what it actually removes A common first response to a membership finding is to stop returning confidence figures: have the endpoint reply with a risk band or a class name and nothing else. It is worth being precise about what that buys, because the change is real but much smaller than it looks. What it removes is the **cheapest** family of membership signals: the ones that read a returned number and compare it against a threshold. Those need a single query per candidate and no cleverness. Removing the number removes that family outright, which is a genuine improvement in the adversary's bill. ## What it does not remove The imprint that makes membership readable is a property of the fitted model, not of the output format. A model that was fit on a record tends to place that record more decisively inside its class than it places a comparable record it never saw. That property is still observable when only a class name comes back, in two ways that need no numbers: - **Distance to the point where the answer changes.** A record the model was fit on typically sits further from the region where the returned class would flip. The adversary cannot read that distance directly, but they can bracket it by asking about variants of the record they already hold. - **Stability of the answer under small variations of the record.** Confidently fitted records keep their class under variation; borderline ones do not. Stability is itself a signal, and it is expressible entirely in returned class names. Neither of these needs the model's internals, an auxiliary corpus, or anything the adversary does not already possess. What they need is **more queries per candidate** rather than one. ## The right frame: price, not prevention The question to ask about any output-restriction control is *what does it multiply the adversary's cost by, and does that matter for the adversary I actually have?* Coarsening outputs is a **cost control**. It moves the attack from one query per name to some larger number, and it removes the laziest tooling. Whether that is worth anything depends entirely on the adversary's shape: | Adversary | Effect of hiding scores | | --- | --- | | Sweeping a long list of candidate names | Real: their total bill scales with the list, so a per-record multiplier bites | | Holding one named person and wanting one bit | Negligible: a handful of queries about one person is not a deterrent | The leaf's own scenario is squarely the second row. An adversary who wants to establish that one named individual attends a specialty clinic will spend a few dozen queries without noticing. Presenting score suppression to a regulator or a review board as though it closed the exposure would be a misstatement, and a technically literate reviewer will ask exactly the question above. ## Related controls in the same family, with the same character - **Quantising or banding the output.** The same trade: fewer distinguishable answers, so more queries needed to resolve the same distinction. Cost, not boundary. - **Rate limiting and per-caller auditing.** These change *who* can be an adversary and how visible their sweep is, which is often more valuable than coarsening outputs, and which does nothing about a single authenticated caller asking about one person. - **Refusing to answer near the boundary.** Reduces one signal and introduces another, since a refusal is also an observation. Anything the endpoint does differently for different inputs is a channel. None of these bounds an attack that has not been run. The only class of claim that does is a **training-time** guarantee, stated over an adversary you did not test, and it is bought with accuracy rather than with output formatting. ## Why this class of mistake recurs The underlying error is treating an output restriction as though it changed the model. It did not. The fitted function is the same function, with the same distinctive behaviour around the examples it was fit on; only the resolution of the channel through which someone reads it has been reduced. Whenever a defence acts on the interface rather than on the training, expect the honest statement to be about the adversary's price and the class of adversary it prices out, never about elimination. ## How to communicate it Say: "We removed the one-query version. The signal remains and costs more to read. Against an outsider sweeping thousands of names, that matters; against someone asking about one named patient, it does not. Here is what we did that changes who can query at all, and here is the residual we are accepting." That is a statement someone can act on, and it does not have to be retracted the first time an evaluator demonstrates a label-only result.