Private training bounds what a membership adversary learns from one record — which examples pay the accuracy cost?
answer
- the cost is not spread evenly
- who needed individual influence to be learned?
- a million votes survive capping; two hundred may not
- the largest gradients belong to the odd examples
basics
~20 sThe rare ones. Capping each record's influence and adding noise removes exactly the individual influence that atypical examples depend on, so rare classes and small subgroups lose far more accuracy than the majority, whose pattern is carried by many records.
solid answer
~50 sPrivate training makes two changes: each example's own gradient is capped at a fixed bound, and noise sized to that bound is added before the weights move. Both are aimed at a record-level adversary — someone who reads the released model and tries to tell whether one person's row was in the training set. But the cost is not spread evenly. A pattern backed by a million records is reinforced a million times per epoch, so it survives capping and the noise averages out over the sum. A pattern backed by two hundred records survives only if those two hundred can each push hard, and pushing hard is precisely what the mechanism forbids. So the accuracy loss lands on rare classes, unusual inputs and small subgroups, while the headline average — dominated by the majority — barely moves.
go deeper
Be ready to say that private training costs accuracy on purpose, and that the loss falls hardest on rare classes and unusual inputs rather than being spread evenly.
Explain the two mechanisms behind it — a cap on each record's own contribution and noise sized to that cap — and why a pattern backed by few records cannot survive them.
Show you would never accept an aggregate utility delta as the cost. Say what you would insist sits beside it before the number means anything for a real deployment.
Own the question of who funds the guarantee. If the tail that pays is the population the product exists to serve, that is a decision to be made deliberately, not a line discovered in a dashboard.
## What the mechanism does Privately trained models are usually trained with per-example gradient clipping plus calibrated noise (the standard recipe is called DP-SGD in the literature). Two things change relative to ordinary training. First, **each individual example's gradient is capped at a fixed norm bound** — not the global norm of the whole batch, which is what ordinary gradient clipping bounds, but each record's own contribution separately. Second, **noise scaled to that cap is added to the summed, clipped gradient** at every step. Together they mean that adding or removing any single record can only shift the released weights by a bounded, noise-masked amount. That is what the formal guarantee is a statement about: an adversary who holds the released model and wants to decide whether a particular record was in the training set gains only a bounded amount from looking. ## Why that costs accuracy at all The model is no longer fitting the data; it is fitting a deliberately blurred version of it. Every step points in a slightly wrong direction, and no example is allowed to insist. Some accuracy loss is therefore intrinsic and not a tuning failure — it is the price of the bound, charged every day whether or not an adversary ever turns up. ## Why the cost lands on the tail Think of learning a pattern as accumulating votes. - A pattern present in a million records gets a million votes per epoch. Each vote is capped small, but the sum still points the same way, and the added noise — being zero-mean — largely cancels across many steps. The majority pattern survives comfortably. - A pattern present in two hundred records gets two hundred capped votes. The noise added per step does not shrink because the slice is small; it is sized to the clipping bound and the batch, not to how rare the pattern is. So the signal-to-noise ratio in the direction a rare pattern needs is far worse, and below some slice size the noise floor simply swamps it. Capping compounds this from the other side. The examples with the **largest** gradients are the ones the current model handles worst — the unusual pronunciation, the rare term, the atypical claim. Capping is a large proportional cut for them and close to a no-op for a typical example the model already predicts well. The mechanism therefore restrains most exactly the records that had the most to teach, because being influential is the thing it is designed to prevent. ## What the resulting number looks like Aggregate accuracy is a weighted average dominated by the head of the distribution. A privately trained model can show an aggregate error delta that reads like a rounding error while a rare-vocabulary slice or a small demographic slice carries most of the actual loss. That is why 'privacy cost us two points' is not a statement of cost until someone says two points *where*. ## Things this is not - **Not a bug or a bad hyperparameter.** More epochs do not fix it: the privacy budget is consumed by the steps you take, so training longer at the same guarantee means more noise per step, not less. - **Not the same thing as class imbalance**, though it interacts with it. The non-private model was already weaker on the tail; private training widens that gap, and the widening is the part attributable to the guarantee. - **Not evidence the guarantee is wrong.** The bound holds; the question is who funds it. If the deployment exists to serve the very population sitting in the tail, the cost has landed on the users the system was built for, and that is a fact somebody should have to state out loud rather than discover later.
- Why does the aggregate accuracy number stay so small even when a slice doubles its error?The aggregate is a weighted average and the weights are the population. A slice holding half a percent of the evaluation set can double its error and move the overall figure by a fraction of a point. The headline is therefore mostly a report on the majority class, which is the part the mechanism barely touches.
- Would training for more epochs recover the accuracy lost on rare classes?No. The privacy guarantee is consumed by the number of noisy steps taken, so holding the same guarantee while training longer means more noise per step. You can move accuracy between head and tail with tuning, but at a fixed guarantee you cannot train the loss away — the only clean way to help the tail is more data supporting it.
saying these in an interview costs you the question
- Says the accuracy loss is spread evenly across classes
- Quotes an aggregate delta as if it were the whole cost
- Thinks the added noise hurts every prediction equally
- Claims longer training recovers the tail at the same guarantee
- Treats the loss as a tuning mistake rather than the price of the bound