skip to content

Why does fitting a generator to real customer records not make its output safe to share?

level: seniorimportance: should knowfreq 31%

answer

  1. Ask where the produced values came from
  2. The generator saw the real records
  3. Outliers have no crowd to hide in
  4. Rare attribute combinations survive the fit

basics

~20 s

Output from a generator fitted to protected records is derived from those records. It can re-emit a near-copy of a rare individual, and it reproduces the same rare attribute combinations, so it stays potentially identifying until an assessment says otherwise.

solid answer

~50 s

Fitting does not launder data. Two things carry through. First, **near-duplication**: a record with no close neighbours can be emitted almost as it stands — outliers are the most exposed, because there is no crowd for them to blend into. Second, **structure**: even with no verbatim copy, the output reproduces the joint frequencies of the source, so a combination of a few coarse attributes that singled out one person in the extract still singles them out in the produced data. So treat produced rows as derived from the protected source: compare every produced row against the extract for near-duplicates, coarsen or suppress rare categories before fitting, and keep the produced dataset inside the same handling boundary until a deliberate assessment moves it. In many privacy regimes, data derived from personal data stays in scope until it is demonstrably no longer identifying.

code

pseudocode · 16 lines
pseudocode
# 1. thin out what has no crowd, BEFORE fitting
for group in coarse_attribute_groups(source_extract):
    if count(group) < MIN_GROUP_SIZE:
        merge or suppress group in source_extract

generator = fit(source_extract)

# 2. compare every produced row with the source it came from
dropped = 0
for row in produced_rows:
    nearest, distance = nearest_record(row, source_extract)
    if distance < MIN_SAFE_DISTANCE:
        drop row; resample; dropped = dropped + 1

# 3. a high drop rate means the fit is memorising, not generalising
record memorisation_rate = dropped / count(produced_rows)

go deeper

for a junior

Know that data produced by a generator fitted to real customer records is derived from those records, and that newly generated values do not by themselves make a dataset anonymous.

for a middle

Explain the two mechanisms: a near-copy of a rare individual, and rare attribute combinations reproduced at the same frequency. Be able to say why an outlier is the most exposed row rather than the safest one.

for a senior

Show the checks you would actually run — thinning rare groups before fitting, comparing produced rows against the source for near-duplicates, and keeping the produced dataset inside the same handling boundary until assessed.

for a principal

Own the decision to move a produced dataset out of the protected boundary: who assesses it, on what evidence, what is recorded, and how the conclusion is revisited when the generator is refitted on newer data.

## "Manufactured" is not a privacy property It is tempting to reason that because every value in the produced dataset was newly generated, no real person is in it. That reasoning skips a step. The generator was fitted to real records; its behaviour is a function of them; the rows it emits are therefore **derived from protected data**, in exactly the way a summary statistic or an average is derived from it. Derivation reduces exposure — it does not by itself remove it. Whether the residual exposure is acceptable is a question somebody has to answer deliberately, with evidence, and formal privacy guarantees are a discipline of their own. ## Two ways identity survives a fit **1. Near-duplication.** A generator that has capacity to spare relative to the amount of data it saw will reproduce parts of that data rather than generalise from it. Even a modest fit does this for records with no neighbours: if one person in the extract is far from everyone else — a very large value, an unusual pairing of coarse attributes, a category with a single member — the fit has nothing to average them with, so the closest thing it can produce is very nearly that person. The inversion here is the part people find surprising and the part interviewers probe: **the most sensitive rows are the least protected**, because rarity is exactly what makes a record both identifying and hard to blend away. **2. Preserved structure.** Suppose no produced row matches any source row exactly. The fit still reproduces how often attribute combinations occur. A combination of a few coarse attributes — an area, a birth year, a job category, an unusual status flag — that appeared once in the source will tend to appear rarely in the produced data too. Anyone holding a second dataset with the same coarse attributes can match on that combination, and matching is what re-identification actually is. Nothing was copied, and someone is still identifiable. A third exposure is often forgotten: **the fitted generator itself is derived data**. Handing the fitted generator to another team, or publishing it, discloses something computed from protected records, and it is not covered by the argument that the dataset was checked. | Kind of source record | How the fit treats it | Residual exposure | |---|---|---| | Common, many near neighbours | Averaged into a dense region | Low; no individual is recoverable | | Moderately unusual | Partly smoothed | Combination may still be rare enough to match on | | Outlier with no neighbours | Reproduced nearly as-is | High; near-copies and unique combinations | | Single-member category | Emitted or dropped, both revealing | High; presence itself is information | ## What actually reduces the risk None of these is a guarantee on its own; together they turn a hopeful claim into a defensible one. 1. **Coarsen or suppress rare values before fitting.** Merge thin categories, bucket extreme values, and remove single-member groups from the source, so the fit never sees a person with no crowd around them. 2. **Check the output against the source.** For every produced row, find its nearest source record and compare; rows below a distance you chose in advance are dropped and resampled. Record how many were dropped, because a high rate is itself a signal that the fit is memorising. 3. **Check rare combinations in the output**, not just single fields. Count how many produced rows share each combination of coarse attributes and treat unique ones as suspect. 4. **Keep the fitted generator inside the boundary** as carefully as the data. It is derived from protected records and can be sampled from indefinitely by whoever holds it. 5. **Write down what was assessed, by whom, against which source version.** A refit on newer data invalidates the old conclusion, so the assessment has to be repeatable rather than a one-off sign-off. ## How to treat the produced dataset Default to treating the produced dataset as carrying the same handling rules as the extract it came from: the same access control, the same environments, the same retention discipline. Relax that only on the strength of an explicit assessment, and record what the relaxation was based on. In many privacy regimes data derived from personal data remains in scope until it is demonstrably no longer identifying, and the burden of demonstrating that sits with the team producing it — so the safe default costs little and the unsafe default is discovered late, usually by someone outside the team. The sentence worth being able to say is: **fitting moves data, it does not clean it — the output is protected until something specific shows it is not.**

  • Which records in the source are most likely to be closely reproduced?
    The ones with no near neighbours: an extreme value, a category with very few members, an unusual pairing of coarse attributes. A fit generalises by averaging over similar records, and those have none, so what it emits sits very close to the original. The most sensitive rows end up the least protected.
  • Does adding noise to the produced values settle the question?
    Not on its own. Noise chosen without reference to how identifying each field is either destroys the realism you fitted for, or leaves the identifying combination intact while blurring fields nobody could match on. What settles it is a deliberate assessment against the source, and formal guarantees are their own discipline.
  • The produced dataset passed a near-duplicate check. Can it leave the protected environment?
    Only on a decision somebody owns, not automatically. The check covers whole-row similarity; rare attribute combinations and single-member groups can survive it. Record what was assessed and against which version of the source, because the next refit invalidates that conclusion.

A crowd photograph blurred until no face is recognisable still gives away the one person standing a head taller than everyone else.

saying these in an interview costs you the question

  • Says generated data is anonymous because no real row was copied
  • Believes rare individuals are the safest because they are unusual
  • Ships the produced dataset outside the protected boundary without checks
  • Treats the fitted generator itself as free to share
  • Assumes one assessment still holds after the generator is refitted