What does the epsilon in a differentially private training run bound?
answer
- two worlds, one record apart
- a cap on telling them apart
- the factor is exponential in it
- covers attacks nobody published yet
- not a radius around an input
basics
~20 sEpsilon caps how much one training record can change what comes out. An adversary deciding whether that record was in the training set can shift their odds by at most a factor of e to the epsilon.
solid answer
~50 sDifferential privacy compares two worlds: the model trained on a dataset with one record in it, and the model trained on the same dataset with that record removed. Epsilon bounds how different the released model can be between those two worlds — the probability of any particular output in one world is at most `e^epsilon` times its probability in the other, plus an additive slack called delta. As a threat statement, that caps a distinguishing adversary: someone who must say whether that record was used shifts their odds by at most that factor, whatever else they know and whichever attack they run. Because the bound is over every possible output rather than over a list of known attacks, it also covers attacks nobody has published. Two things it is not: it is not a perturbation radius, and it is not a promise that nothing leaks — only that one record's contribution is capped.
go deeper
Be ready to state the two-worlds comparison in one sentence and say which direction the knob runs: smaller epsilon, stronger promise. Do not confuse this epsilon with the radius of an allowed input perturbation.
Expect to explain that the bound is multiplicative and exponential in epsilon, that it covers every possible output rather than a list of known attacks, and that anything computed from the release stays inside it.
Show what a reported epsilon does not establish. It says nothing about the accuracy the run cost, nothing about how many releases it covers, and nothing at all when it arrives without its delta.
Own the framing that this is a dial you set and publish, not a checkbox you tick. Be ready to say who chose the number, what it is stated over, and who would be told if it changed.
## The two-worlds comparison Differential privacy is a property of a **randomised procedure** — here, the training run that turns a dataset into weights — and not of a dataset, a model file, or a person. The definition fixes two datasets that are identical except that one contains a particular record and the other does not. These are called *neighbouring* datasets. The procedure satisfies the guarantee if, for **every** possible outcome it could produce, the probability of producing that outcome on one dataset is close to the probability of producing it on the other. "Close" is where epsilon lives. The bound is multiplicative: the probability of any set of outputs under the with-the-record world is at most `e^epsilon` times its probability under the without-the-record world, plus an additive term delta. Epsilon is therefore a **log-scale knob on distinguishability**, not a percentage, not a probability, and not a distance in input space. ## What that means for an adversary Turn the definition around and it becomes a security statement, which is how an interviewer will want it. Suppose an adversary is trying to decide whether one specific record was in the training set. They may hold the released model, its architecture, the training code, the hyperparameters, and every other record in the dataset. The guarantee says their posterior odds on that one question can move away from whatever they believed beforehand by a factor of at most `e^epsilon` (with delta as slack). If they started at even odds, they end at worse than `e^epsilon` to one. Three consequences follow directly, and each one is a question in its own right: - **It is worst case over adversaries.** The bound quantifies over all outputs and all side information, so it constrains attacks that have not been invented. This is the property that makes it different in kind from every empirical robustness or privacy number in this field — those describe the attack that was run. - **It is a cap, not a measurement.** A model can satisfy a weak guarantee and still resist every attack you can build today, and a model can satisfy a strong guarantee and still be an ordinary, useful model. The number is what you can *promise*, not what an attacker actually achieved. - **It survives post-processing.** Anything you compute from the released model — serving it, distilling it, publishing metrics from it — is still covered by the same bound, because the adversary could have done that computation themselves. ## What it does not say The most common overclaim is that a differentially private model is "anonymised" or that "no information about anyone leaks". Neither is what the definition states. The definition constrains only the **marginal contribution of one record**: the release may reveal a great deal about the world, about the task, and about patterns that hold across many records, and the guarantee is untouched by that, because removing any single record would not change those patterns. That is exactly the design intent — a model that learned nothing general would be useless — but it means the guarantee is narrower than the words "private model" suggest. Equally, epsilon here is **not the epsilon of an adversarial-example threat model**. In the evasion literature the same letter names the radius of a ball of allowed input perturbations; a larger radius means a stronger attacker. In privacy the letter names a bound on a probability ratio; a larger value means a weaker promise. The two point in different directions and quoting one where the other is meant is a real interview failure. ## Reading a number Epsilon alone is half a statement. It must arrive with its delta, which is an additive failure probability, and with the accounting method and the set of releases the number covers. Rough intuition on the scale: `e^0.1` is about 1.1, so an epsilon near 0.1 barely lets an adversary move; `e^1` is about 2.7, a meaningful but real constraint; `e^8` is about 3,000 and `e^12` about 160,000, at which point the guarantee excludes essentially nothing. ## How it gets asked Junior loops ask for the two-worlds sentence and the direction of the knob. The follow-up is nearly always some version of "so does that mean the model can't leak training data?" — and the answer is that the guarantee bounds what any single record contributes, at a strength you chose and published, and that a number nobody stated is not a guarantee at all.
- Why does the bound hold against attacks that have not been published?Because it quantifies over every possible output of the training procedure and assumes an adversary with arbitrary side information, including every other record. Nothing in it refers to a particular attack, so a new attack is already inside the set of things being bounded. That is the opposite of an empirical robustness number, which describes only the attack that was actually run.
- A membership attack against the model reaches only 51 percent accuracy — does that mean epsilon is small?No. A measured attack is a lower bound on leakage, not a guarantee. A weak or badly tuned attack fails against models with a huge epsilon or none at all. The guarantee is worst case over all adversaries; the measurement is one adversary on one day. Report them as different kinds of claim.
- Does epsilon say anything about how much accuracy the training run lost?Nothing at all. It is a bound on distinguishability, not on utility. The accuracy cost has to be measured separately, and reported per slice rather than as an aggregate, because the cost of a private training run does not land evenly across the data.
Two nearly identical rooms differ only by whether one person is inside. Epsilon caps how differently the two rooms are allowed to sound to somebody listening at the door.
saying these in an interview costs you the question
- Says a differentially private model is anonymised
- Reads epsilon as a perturbation radius on inputs
- Claims the bound means no information leaks
- Treats differential privacy as something you have or lack
- Quotes epsilon with no delta beside it