Privacy Attacks: Inference & Inversion

Storing rows and fitting them are different. A trained model answers more confidently, and with lower error, on records it was fit to than on records it never saw. One query reads that difference.

Storage is not the only channel. A model answers differently on records it was fit on, so an outsider holding a person's record can query the deployment and get an edge on whether it was in training.

Shadow models need data from the same population as the target's training set, not its actual rows. They learn what a model's response to a record it was trained on looks like in general, and that pattern transfers.

How much the model overfits. The test reads the difference between the model's behaviour on data it was fit to and data it was not, so the average edge is capped by a gap the attacker cannot enlarge.

A 60% figure establishes that a membership signal exists, and nothing about harm. Balanced-set accuracy is measured against a coin flip, not the cohort's real prevalence, and severity is set by what membership means.

A composite the model treats as typical of that class, not a training record. The search maximizes evidence pooled across every example the class contained, so the output resembles a class average rather than any individual input.

An embedding is a lossy but largely invertible encoding of its input. With the same encoder — usually public or purchasable — much of the original wording can be reconstructed, so the index carries the documents' sensitivity, not less.

Because the thing that does leave the device is computed from that data. An update is a function of the local rows and is often invertible enough to rebuild them. Privacy comes from batching, aggregation and calibrated noise, not from locality.

No. Dropping a column removes the field from the input, not from what the model's outputs encode, and the attacker reads outputs rather than the schema. Removal changes the attack from exact matching to estimation; a measurement settles which.

The one missing field. Holding the rest of the application, the attacker submits it once per candidate value and compares the returned premiums against the figure the real applicant was quoted; the match names the declared value.

A generative model invents well-formed text on demand, so a realistic-looking span proves only that it knows the format. A memorization claim needs verification against the genuine source, or a confidence gap against a model that never saw the data.

Generalization and memorization happen in the same model. Training drives loss down over the corpus, and for a rare structureless string there is no pattern to generalize to, so storing it is the only way loss falls.

A clean canary result proves only that a string of that shape, inserted that many times, resisted that extraction attempt against that model version. It is not evidence that the model does not memorize.

A canary is a known random string the owner inserts into the training corpus a counted number of times, then tries to pull back out of the trained model using only the access an outsider has.

Ranking candidates by the target model's own confidence selects intrinsically likely text such as boilerplate. Comparing its score with an uninvolved reference model cancels what is easy for everyone, leaving spans the target fits unusually well.

Neither operation bounds a single record. Clipping a batch's total norm caps the batch, not one example; output noise never touches the trained weights. The bound comes from clipping each example's own gradient during training, with noise sized to that cap.

Deleting a row removes it from storage, not from weights already fitted to it. Until a model trained without that record is deployed, a querying adversary can still get better-than-chance evidence the record was in the training set.

Much less than the headline number suggests. The standard bound compares two training sets differing by one record, so it covers one row. Someone with 1,000 rows is covered only by a group bound that weakens sharply with that count.

Epsilon caps how much one training record can change what comes out. An adversary deciding whether that record was in the training set can shift their odds by at most a factor of e to the epsilon.

Almost anything. The bound is multiplicative in e to the epsilon, so at 12 it lets an adversary's odds on whether a record was used move by a factor near 160,000. That excludes essentially nothing.