In federated training, why is 'the raw data never leaves the device' not a privacy guarantee against a server that sees only uploaded updates?

## The claim being challenged Collaborative or federated training is usually sold with one sentence: the data stays on the device. Each participant — a phone, a hospital, a smart meter in a household energy pilot — trains locally and uploads only a model update. A central server combines the updates into a new global model and sends it back. The sentence is true as a statement about file locations. It is not a statement about what an observer can learn. ## What actually leaves the device The uploaded object is computed **from** the local rows. In the simplest scheme it is the gradient of the loss with respect to the model parameters, evaluated on the client's current batch. In the more common scheme the client takes several local steps and uploads the difference between its finished weights and the weights it started from. Either way, the upload is a deterministic function of (a) the global model, which the server already knows, and (b) the client's examples, which are the secret. That framing is the whole leaf. Anything that is a function of a secret is a channel from that secret. The only question is how much of the secret survives the function — and for a single small batch, empirically, a great deal does. ## Why the function is invertible enough A curious aggregation server holds the model and receives an update it knows was produced by that model on unknown inputs. It can therefore ask: which inputs would have produced this update? Because the server can evaluate the same model on any candidate inputs it likes, it can compare the update those candidates produce against the one it received and keep adjusting the candidates until the two agree. When one update came from one example, or from a handful, the answer is heavily constrained and what comes back is not a vague statistical hint — it is a recognisable version of the training example. In an energy federation that is a household's consumption window: whether anyone was home, when the shower ran, when the oven came on. Two secondary facts make the leak larger than people expect. First, the structure of the last layer's update alone usually reveals which classes or targets were present in the batch, before any reconstruction is attempted. Second, dimensionality is not protection: an update for a modern model has far more numbers in it than a single training example does, so there is plenty of room for the example to be embedded in it. ## Who the adversary is, and what they are limited to The relevant adversaries are not remote attackers who breached anything. They are the aggregation server — honest-but-curious is the standard framing, meaning it follows the protocol and reads everything it legitimately receives — and, in designs where updates are visible to peers, a co-enrolled participant. Their limit is that they see the update, not the rows, and only for the rounds their vantage covers. That limit is real, and it is exactly why the design parameters below matter: they decide how much of the secret survives into the object the adversary is allowed to see. ## The three properties that do buy privacy 1. **Mixing.** The more examples fold into one uploaded update, and the more local steps are taken before uploading, the less any single example determines it. Reconstruction quality degrades as the local batch grows; with many local steps the upload is the endpoint of a trajectory through unknown intermediate weights, which is much harder to unwind. 2. **Aggregation.** If the server can only ever observe the sum of many clients' updates rather than any one client's, the easiest attack — pick a victim, invert their upload — is off the table, and the effective batch becomes the whole cohort's data. 3. **Calibrated noise with per-example clipping.** This is the only one of the three that produces a stated bound rather than an empirical difficulty. Bounding each example's contribution and then adding noise scaled to that bound is what turns 'we could not reconstruct it' into a quantified statement — and it costs accuracy, most of it on the unusual clients whose behaviour is rare in the population. ## What a candidate should say That locality answers the wrong question. The right question is what the shared object is a function of, and how many examples it was averaged over before anyone else saw it. Adjacent reassurances — encrypting the channel, hashing client identifiers, promising the server is trusted — address transport, attribution and policy. They do not change what the payload encodes.