What does switching a GAN to a Wasserstein critic with a gradient penalty actually fix?

## The problem being fixed A standard discriminator is a binary classifier. In high dimensions the real data and the generated data typically sit on thin, nearly disjoint regions, so a sufficiently expressive discriminator can separate them almost perfectly. When it does, its output saturates and the gradient it hands back to the generator shrinks toward zero. This is the vanishing-discriminator-gradient failure, and it is perverse: the generator learns least when its opponent is best. The practical consequence is a training regime built on deliberately hobbling the discriminator — train it too little, too weakly, or with too small a capacity, and hope the balance holds. That is fragile, it interacts badly with mode collapse, and it is why so many adversarial runs need per-dataset babysitting. ## What the Wasserstein formulation changes Replace the classifier with a **critic** that outputs an unbounded real number, and train it to maximize `E[f(x_real)] - E[f(x_fake)]`. If `f` is constrained to be **1-Lipschitz** — meaning its output cannot change faster than its input, so the norm of its gradient with respect to the input is at most 1 everywhere — then the supremum of that difference is the Wasserstein-1 (earth-mover) distance between the real and generated distributions. Two consequences follow, and they are the whole reason to switch. 1. **The signal survives disjoint support.** Earth-mover distance measures how far mass has to travel, so it is finite and decreases smoothly as fake samples move toward real ones, even when the two distributions do not overlap at all. The critic therefore keeps pointing the generator in a useful direction no matter how well it is currently doing. Training the critic to convergence becomes desirable rather than dangerous — the opposite of the classifier regime. 2. **The number means something.** Because the critic estimates a distance between distributions rather than a classification score against whatever the generator looks like today, its value tends to fall as sample quality improves. That gives back the one thing ordinary adversarial losses never provide: a curve you can compare across time within a run and use to rank checkpoints. ## Why the constraint cannot be skipped Without the Lipschitz constraint the critic can simply scale its outputs up without bound: push real scores to plus infinity and fake scores to minus infinity, and the objective improves forever without saying anything about the distributions. The constraint is what makes the value an estimate of a distance rather than an arbitrary number. ## Weight clipping and why it was replaced The original formulation enforced the constraint by clamping every critic weight into a small interval `[-c, c]`. This works, in the sense that a network with bounded weights is Lipschitz, but it is a crude proxy with several bad properties: - **Capacity waste.** The critic maximizes an objective that wants large outputs, so under clipping it drives most weights to the two extreme values. The result is a critic that behaves like a much simpler, often near-linear function than its architecture allows. - **Depth sensitivity.** The clip threshold has to trade off two failures across the network's depth: too small and gradients decay layer by layer, too large and they blow up. `c` becomes a delicate hyperparameter you retune per architecture. - **It constrains the wrong quantity.** Bounding weights is a sufficient condition for Lipschitzness, not a targeted one, so it over-restricts in some directions and under-restricts in others. ## The gradient penalty A differentiable function is 1-Lipschitz exactly when the norm of its input-gradient is at most 1 everywhere; the optimal critic in this formulation has gradient norm exactly 1 along the straight lines connecting real and generated pairs. That motivates a soft penalty added to the critic's loss: `penalty = lambda * (||grad_xhat f(xhat)||_2 - 1)^2` where `xhat` is drawn uniformly along the segment between a real sample and a generated sample. It targets the property the theory actually requires, it is applied where the theory says it binds, and it leaves the critic's capacity intact. Operational details worth knowing: - The penalty is computed **per sample**, so any normalization inside the critic that couples samples through batch statistics is inconsistent with it; a per-sample normalization such as layer normalization is used instead. - Computing the penalty differentiates through a gradient, so each critic step costs meaningfully more than an ordinary step. - Several critic steps are typically run per generator step, which is affordable precisely because a strong critic is now an asset. ## What it does not fix - **It does not guarantee coverage.** A Wasserstein critic with a gradient penalty can still mode-collapse; it removes a common cause of collapse, not the possibility. - **It does not remove tuning.** The penalty coefficient, the number of critic steps and the architecture still matter. - **The estimate is approximate.** The constraint is enforced softly, at sampled points, by a finite network. The critic value is a good relative signal inside one run, not an absolute distance you can quote or compare across architectures. ## Interview framing Lead with the failure — a winning classifier returns no gradient — then name the two things the switch buys (a signal that survives disjoint support, and a readable number), then explain the constraint and why the penalty targets it better than clipping. Finish by saying what it does not buy, because claiming it solves mode collapse is the standard overreach.