How can a generator that memorises its training images still score an excellent FID?

## Why the metric cannot see it Start from what a distributional fidelity score is asking: *how far is the distribution of generated samples from the distribution of real samples, as measured in some feature space?* Now consider the degenerate generator that stores the training set and samples uniformly from it. Its output distribution **is** the training distribution. The distance to the real reference is as small as the estimator can report. This is not a bug in FID, and no amount of increasing the sample count fixes it. Matching the distribution and producing novel content are different properties, and only the first is being measured. The same holds for precision and recall for generative models: copied samples sit exactly on the real manifold, so precision is maximal, and a model that reproduces the whole training set also reaches all of the real support, so recall is maximal too. Partial memorisation is the realistic version. Models trained on small datasets, or on datasets with many duplicated images, reproduce some fraction of their training data closely while generating the rest. The distributional score improves as the copying increases, so the metric actively rewards the behaviour you want to prevent. ## Why it matters beyond metric hygiene A generator that emits near-copies of its training data carries the licensing, privacy and confidentiality properties of that data into every output. For medical, personal or licensed corpora that turns a modelling curiosity into a release blocker, which is why the audit belongs in the evaluation suite and not in a research appendix. ## The audit **Retrieve nearest neighbours in a feature space.** For each generated sample, find its closest training examples using embeddings rather than raw pixel distance. Pixel distance is defeated by a one-pixel shift, a small crop, a horizontal flip or a brightness change, all of which leave a copy perceptually identical while moving it far away in pixel space. **Calibrate against held-out real data.** This is the step people skip and the reason naive audits produce nonsense in both directions. Real datasets contain genuinely similar images, so some small nearest-neighbour distances are normal. Build the reference distribution by taking held-out real images — images the model never saw — and measuring *their* distances to their nearest training neighbours. Then compare the generated set's distance distribution against that reference. If generated samples are systematically closer to training images than a genuine unseen real image is, the model is copying. If the two distributions overlap, the retrieved "matches" are just the ordinary similarity present in the domain. **Deduplicate first.** A training set with many near-duplicates makes the model far likelier to reproduce those images and simultaneously makes the baseline distances tighter, so both sides of the comparison are distorted. Deduplicating in feature space before training and before the audit removes that confound. **Look at the tail with your own eyes.** Aggregate statistics tell you whether a problem exists; the worst hundred retrieved pairs tell you what it is. Side-by-side inspection of the most-suspicious generated samples and their nearest training neighbours is the step that turns a distance histogram into a decision. **Check the invariances that matter.** Retrieve under flips and crops if the training pipeline used them; otherwise an augmented copy passes the audit untouched. ## Reporting it A usable memorisation report is not a single number. It is: the distance distribution for generated samples overlaid on the held-out-real baseline, the fraction of generated samples below the baseline's low percentile, and the inspected worst cases. That gives a reviewer something they can argue with, and it gives you a threshold you can gate a release on. ## The framing that scores well in interviews The headline is that distribution matching and novelty are different questions, so no fidelity metric — however good — can answer both. Anyone who proposes "a lower FID" as the response to a memorisation concern has missed that the metric moves the wrong way. The correct answer names the separate audit, and the calibrated baseline is the detail that separates candidates who have actually run one from those who have read about it.