Why can a GAN's generator and discriminator loss curves not be read as training progress?

## Why the curves lie Supervised training has a fixed target: the labels do not move, so the loss is a stable yardstick and a decrease is a real improvement. Adversarial training has no such yardstick. The discriminator's loss measures how well it separates real data from *the current generator's* output; the generator's loss measures how badly it fools *the current discriminator*. Both denominators move every single step. A number that goes down can mean either that the player improved or that its opponent got worse, and the two are indistinguishable from the curve alone. This also means the curves are not comparable across runs or across time within a run. There is no value of the generator loss that corresponds to good samples, and no threshold that signals convergence. ## The canonical trap A very common trace: the discriminator loss drops toward zero and pins there, the generator loss climbs steadily, and a reviewer looking only at the plots concludes the generator is diverging. Decode samples and they are visually unchanged for forty thousand steps. The correct reading is that the discriminator has *won*. It separates real from fake with near-certainty, so its output is saturated. The generator's rising loss is simply a report of how confident the discriminator has become, not a measurement of sample quality. Worse, a saturated discriminator returns a tiny gradient to the generator, so the generator has effectively stopped learning — the loss is climbing precisely because the mechanism that would fix it has switched off. Nothing about the sample distribution is changing, which is why the images stand still. The mirror image is equally misleading: a generator loss that suddenly collapses toward zero usually means the discriminator was beaten rather than that the samples became excellent, and that state is also unproductive. ## What the equilibrium is supposed to look like In the ideal game, the two players reach a point where the discriminator cannot tell real from fake and outputs an uninformative constant, and both losses settle at their game value. Real runs rarely converge to it; they orbit. So even the theoretically meaningful reading — losses hovering near the equilibrium value — is a weak signal in practice, and hovering is compatible with a fully collapsed generator that is fooling a discriminator which has not learned to check diversity. ## What to instrument instead - **A fixed-noise sample grid.** Draw a batch of noise vectors once, before training, and decode it at fixed step intervals. Because the input is held constant, everything you see is a change in the generator. Flipping through those grids in order is the most informative artefact an adversarial run produces. - **Discriminator accuracy on held-out data.** Score held-out real samples and fresh fakes and log the accuracy. Accuracy pinned at 100% is the alarm for the trap above: the game has stopped being a game. Accuracy near chance means either healthy equilibrium or a broken discriminator, which you separate by looking at the samples. - **The gradient magnitude reaching the generator.** If the norm of the gradient arriving at the generator's output has decayed to near nothing, the generator is not learning regardless of what its loss prints. - **Diversity statistics.** Per-feature variance across a generated batch relative to a real batch, or intra-batch nearest-neighbour distances, catch a collapse that no loss curve shows. - **A periodic quantitative sample metric.** Evaluating generated samples with a held-out quantitative score at checkpoints gives you the comparable number the losses refuse to provide. Choosing and interpreting those scores is its own topic; the point here is that it must be a *sample* measurement, not a loss. - **The critic estimate, if you switched objectives.** With a Wasserstein critic, the critic's value estimates a distance between distributions rather than a classification score against a moving opponent, and it does correlate with sample quality — which is a large part of why practitioners switch to it. ## Practical consequences Do not early-stop on loss, do not select the final checkpoint by loss, and do not compare two architectures by their generator loss. Checkpoint at fixed intervals and choose among checkpoints using samples and a sample-level metric. When someone reports that a GAN is 'converging nicely' and shows only loss curves, that report contains no evidence — ask for the fixed-noise grid. ## Interview framing Say the losses are relative scores against a moving opponent, give the near-zero-discriminator-loss reading as a worked example of a curve that looks alarming and means 'the game stalled', and finish with the two or three signals you would put on the dashboard instead.