A spectrally normalized GAN trains stably at batch size 32 but destabilizes at 256 — how do you diagnose it?

## What spectral normalization does, and what it does not Spectral normalization divides each weight matrix by its largest singular value — its spectral norm — estimated cheaply with a power-iteration step carried across training steps. Each linear or convolutional layer then has operator norm 1, so with activations whose own Lipschitz constant is at most 1, the whole discriminator's Lipschitz constant is bounded by the product of layer constants. Practically, it caps how fast the discriminator's output can change as its input changes, which stops it from becoming an arbitrarily sharp separator and keeps the gradients it returns bounded. It costs almost nothing, needs no extra loss term and no interpolated samples. What it does **not** do is fix the *balance* of the game. It bounds one player's sharpness; it says nothing about how much each player improves per update, how many updates each gets, or how much noise is in those updates. Batch size moves all three. That mismatch is the whole diagnosis. ## Step one: name the failure 'Destabilizes' covers three different bugs, and they lead to different fixes. - **Collapse.** Sample diversity dies, held-out discriminator accuracy climbs toward 100%, the fixed-noise grid fills with near-duplicates. - **Divergence.** Critic or discriminator scores grow without bound, activations blow up, losses go to extreme values. - **Stall.** Everything is finite and steady but the samples stop changing. Decode the fixed-noise grid and log held-out discriminator accuracy on real and fresh fake data before touching anything. ## Step two: remove the update-count confound If the epoch budget was held fixed, raising the batch from 32 to 256 cut the number of generator updates by a factor of eight. Many 'large batch broke it' reports are really 'the generator got one eighth of the training'. Rerun at the larger batch with the number of generator updates held equal to the original run, same seed. If the instability disappears, the batch size was never the cause. ## Step three: the balance argument If it survives the control, the leading explanation is that the two players no longer improve at comparable rates. - **Estimate quality.** A larger batch gives the discriminator a much lower-variance estimate of the real distribution, so each of its updates is a bigger real improvement. The generator's updates also get less noisy, but the discriminator's task — separating two fixed-for-this-step distributions — benefits more directly. The effective strength ratio shifts even though no hyperparameter was 'changed'. - **Noise as a stabilizer.** In a two-player game, minibatch noise is not purely a nuisance. It perturbs the pair out of the cycling behaviour that produces mode hopping. Removing most of it can let the players settle into a tight oscillation or let the discriminator lock in a decisive win. - **Per-layer bound is loose.** The product-of-layer-norms bound is an upper bound. Skip connections, the output layer's scale, and the fact that the bound is rarely tight all mean the effective sharpness of the discriminator can still grow as it trains harder on cleaner gradients. ## Step four: check for batch-coupled layers Any layer in the discriminator that normalizes using statistics of the current batch behaves differently at 256 than at 32, by construction. Worse, if real and generated samples pass through in separate batches, their differing batch statistics leak information the discriminator can exploit in a way that does not correspond to any real-versus-fake distinction. This is a genuine batch-size-dependent failure mode and it is worth ruling out early because the fix is structural: use a per-sample normalization instead. ## Step five: instrument, then change one thing The log you want, per player, per step: gradient norm, held-out accuracy or critic estimate, and a diversity statistic across a generated batch. With that in place, change exactly one variable at a time. Useful moves once the diagnosis points somewhere: - If the discriminator is winning: fewer discriminator steps per generator step, or a stronger constraint on it such as adding a gradient penalty on top of the normalization. - If the loss of gradient noise is implicated: accumulate the large batch from several smaller micro-batches to separate 'noise scale' effects from 'batch-statistic' effects, or reintroduce controlled noise through input augmentation applied to both players' inputs. - If it is collapse specifically: bring in batch-level statistics on the discriminator side, which also happens to interact with batch size in your favour at 256. General step-size and schedule retuning for a larger batch is the neighbouring optimization topic, not the adversarial-specific part of this diagnosis — mention it as a known confound and control for it rather than making it the story. ## What to report The honest conclusion is almost never 'large batches are bad for adversarial training'. It is a specific statement: at this batch size, with this constraint, the discriminator outpaced the generator, and here is the metric that shows it. That is a claim someone can act on and re-test. ## Interview framing The interviewer is checking whether you reason about the two-player dynamics rather than reaching for a knob. Classify the failure, control the confound, then argue about balance and noise — and be explicit that a Lipschitz constraint bounds sharpness, not balance.