skip to content

A model retrained daily reports a privacy epsilon of 1.9 per refresh — what is missing?

level: seniorimportance: should knowfreq 35%

answer

  1. the adversary sees every release
  2. one refresh is not the deployment
  3. budgets add over the window
  4. serving the model costs nothing more
  5. disjoint records would compose differently

basics

~20 s

The composed budget. Every refresh is a fresh release computed on records still inside the training window, so their costs add and an adversary sees all of them. A per-refresh number describes one release, not the deployed system.

solid answer

~50 s

Privacy budgets compose. Each refresh trained on a window that still contains a given record is another release about that record, and an adversary sitting outside can watch every one of them, so what bounds them is the **composed** epsilon across the releases the record appears in — not the per-refresh number. Two things sharpen that. Composition is over new computations on the data: serving the released model, distilling it, or publishing metrics derived from it costs nothing further, because the adversary could have done those themselves. And refreshes over genuinely disjoint sets of records compose in parallel, where the cost is the maximum rather than the sum — but a sliding retention window keeps the same records for months, so those releases are not disjoint. Ask for four things: the composed epsilon, the delta, the accounting method, and the number of releases the total covers.

code

text · 9 lines
text
privacy_report - interaction ranker, refresh 2026-06-14

  training rows (n)        4,100,000
  epsilon (this refresh)   1.9
  delta                    1e-9
  accountant               (not stated)
  releases covered         this refresh only
  ...
  deployment note: refreshed daily; interaction log retention 180 days

go deeper

for a junior

Know that a privacy budget is spent per release and that retraining on the same data again is another release, so one training run's number does not describe a system that retrains on a schedule.

for a middle

Explain sequential, parallel and post-processing composition and say which applies to a sliding retention window. Be clear that serving a released model is free while recomputing on the records is not.

for a senior

Given a report, name the missing columns out loud — composed total, delta against the record count, accounting method, releases covered — and refuse to sign off on a per-refresh figure presented as the deployment's guarantee.

for a principal

Own the reporting standard: what your organisation publishes is the composed pair over the retention lifetime with its accountant named, and any ledger reset has to be justified by records actually leaving the training set.

## One release is not one system The guarantee is a property of a procedure that produces a release. A model retrained on a schedule produces a stream of releases, and the adversary is not obliged to look at only one of them. If a record sits in the training window for a hundred and eighty days and the model is refreshed daily, that record has been the subject of a hundred and eighty releases, all visible. The quantity that bounds what an adversary can learn about it is the composed budget over those releases, and the per-refresh number in the report is not that quantity. This is the single most common way a privacy claim about a production system is wrong, and it is wrong in the flattering direction. ## How composition behaves Three rules cover almost every case a reviewer meets. **Sequential composition — costs add.** Independent releases computed on the same records combine into a bound whose epsilons add, in the simplest accounting. Tighter accounting exists and gives a smaller total than naive summation, but the total still grows with the number of releases; nothing makes it stop growing. **Post-processing — free.** Anything computed from a released model without touching the training data again is covered by the same bound. Serving the model to a billion requests, compressing it, publishing its evaluation curves, or building a product feature on top of it spends nothing, because an adversary holding the release could have computed all of it themselves. **Parallel composition — the maximum, not the sum.** Releases computed over disjoint sets of records cost the maximum of their epsilons rather than the sum, because each record appears in only one of them. This is the escape hatch teams reach for, and it usually does not apply: an overlapping sliding window means the same records are in many refreshes, so those releases are sequential for them. ## The accounting method belongs in the report The same sequence of training runs produces materially different totals depending on how the composition was accounted — naive summation, an advanced-composition bound, or a tighter moment-based accountant all yield different epsilons for identical runs. So two composed numbers are not the same unit unless the method is named. A report that gives a number and no method cannot be compared with anything, including its own previous quarter if the tooling changed. ## What the auditor actually asks for Given a per-refresh line, the reviewer's list is short and each item has a reason: - **The composed epsilon over the retention window.** This is the number that bounds an adversary watching the deployment. - **The delta, checked against the record count.** Deltas compose too, and an additive term that was negligible per release may not be negligible summed over hundreds. - **The accounting method.** Otherwise the number is not comparable. - **How many releases the total covers, and whether the ledger was ever reset.** A reset is only honest if the records covered by the spent budget are genuinely out of every future training set. ## The direction of the claim A per-refresh epsilon of 1.9 is a real fact about one training run; it is not a weaker version of the system's guarantee, it is a different quantity. Quoting it as the system's number overstates the guarantee by however many releases the record survived. Conversely, a large composed number does not mean the deployment is leaking — it means the promise you can make about it is weak, and the honest move is to report the composed figure with its scope rather than to report the flattering one with none.

  • Does serving the deployed model to millions of requests spend more of the budget?
    No. Post-processing is free: anything computed from a released model without touching the training data again is covered by the same bound, because an adversary holding the release could compute it themselves. Only new computations over the private records — retraining, fine-tuning, evaluating on them — add to the ledger.
  • When does retraining genuinely not add to the composed budget?
    When the releases are over disjoint sets of records. Then parallel composition applies and the cost is the maximum rather than the sum, because each record appears in one release only. A sliding retention window keeps records for months, so its refreshes overlap and their costs add for those records.
  • Why must the accounting method be named beside the number?
    Because identical training runs yield different totals under naive summation, an advanced-composition bound, or a tighter moment-based accountant. Two composed epsilons computed by different methods are not the same unit, so comparing them — across teams or across quarters — silently compares tooling instead of privacy.

saying these in an interview costs you the question

  • Quotes the per-run epsilon as the system's guarantee
  • Thinks serving the model spends more budget
  • Assumes the ledger resets each quarter
  • Compares epsilons computed by different accountants
  • Claims parallel composition on an overlapping window

context

resolving links…