Sample Fidelity Metrics

## The problem FID is trying to solve An implicit generator — one you can sample from but whose density you cannot evaluate — gives you a bag of samples and nothing else. There is no held-out label to score against and no per-sample notion of correctness: a generated face is not "wrong", it is either plausible or not, and the set as a whole is either as varied as the real data or not. So the evaluation question has to be reframed from "is this sample right?" to "does the distribution of generated samples look like the distribution of real samples?" Comparing distributions directly in pixel space is hopeless. Two images of the same scene shifted by one pixel are far apart in pixel distance while being perceptually identical, and pixel statistics are dominated by low-level structure nobody cares about. FID's move is to compare the distributions somewhere more semantic. ## The recipe 1. Take N real images and N generated images. The conventional N is large — tens of thousands — for reasons covered below. 2. Push every image through a fixed, pretrained image classifier (conventionally an Inception network trained on ImageNet) and keep the activations of the pooling layer just before the classification head. Each image becomes a feature vector of a couple of thousand dimensions. These features encode object-ish, texture-ish content rather than raw pixels. 3. Summarise each of the two feature clouds by its sample mean vector `mu` and its sample covariance matrix `S`. This is the same as fitting one multivariate Gaussian to each cloud by moment matching. 4. Report the Fréchet distance (also called the 2-Wasserstein distance) between those two Gaussians, which has a closed form: `FID = ||mu_r - mu_g||^2 + Tr(S_r + S_g - 2*(S_r*S_g)^(1/2))` where the subscripts are real and generated, `Tr` is the matrix trace, and the square root is the matrix square root, not an elementwise one. ## Reading the two terms The squared-norm term measures how far apart the *average* feature vectors are. A generator whose samples systematically have the wrong colour balance, the wrong object mix, or a global artefact shifts the mean and this term grows. The trace term measures disagreement in the second moment: how spread out each cloud is and how its dimensions co-vary. A generator that produces beautiful but narrow output — say it has quietly stopped covering whole regions of the real data — has a feature covariance much smaller than the real one, and the trace term grows even if the means happen to line up. This is why FID reacts to lost variety while a metric that only checked realism per sample would not. ## What the number does and does not license Lower is better. FID is non-negative, and it is zero exactly when the two fitted Gaussians coincide. That last sentence is the whole caveat. FID only ever looks at the first two moments of the feature distribution. Two genuinely different distributions with matching means and covariances in that feature space score zero against each other. Nothing about the third moment, multimodality, or per-sample plausibility enters the formula. The Gaussian fit is a modelling assumption about deep feature activations that is convenient rather than true. A second limit: the score has no absolute meaning. "FID 14" is not a quality grade — it is a number relative to a particular real reference set, sample count, feature network and preprocessing pipeline. Useful FID work reports deltas within one frozen protocol, and often reports the floor obtained by scoring one half of the real data against the other half, which is greater than zero and tells you how much of your number is estimation noise. A third limit worth stating out loud: FID never inspects individual samples. It cannot tell you *which* images are bad, and it will happily reward a generator that reproduces its training data, because a copy of the training set matches the real distribution by construction. ## Practical framing for an interview Say what it compares (feature distributions, not images), what it assumes (one Gaussian per set, so only means and covariances), which direction is good (lower), and one honest limitation. Candidates who describe FID as "comparing each generated image to the closest real image" or as a pixel-space distance have not understood the object being measured, and that mistake propagates into every downstream decision about how many samples to score and what a difference of two points means.