Autoencoders, GANs and Generative Models

Mode collapse is when a GAN generator maps many different noise vectors to a few nearly identical outputs, covering only part of the real data. You spot it by decoding a fixed batch of noise vectors and seeing the samples repeat.

An autoencoder is trained to copy its input, so a wide enough code can simply learn the identity map: near-zero loss, nothing learned. A code narrower than the input makes exact copying impossible and forces the encoder to keep only rebuildable structure.

Scoring against the clean original means copying the input no longer wins. The network has to use structure shared across the data to remove the corruption, so the code ends up describing that structure instead of the input itself.

Only the discriminator can create pressure to obey the label. If it judges images alone, any realistic image passes, so the generator's cheapest strategy is to ignore its label input and reproduce the overall data distribution.

Early on, the discriminator rejects generated samples confidently, and log(1 - D(G(z))) is flat in that regime, so the generator receives almost no gradient. The non-saturating form -log D(G(z)) is steepest exactly where samples are being rejected.

Each GAN player's loss is measured against an opponent that changes every step, so a falling or rising curve says who is currently ahead, not whether samples improved. Judge progress from samples and diversity, not from the losses.

FID embeds real and generated images with a fixed pretrained classifier, fits one Gaussian to each set of feature vectors, and reports the Fréchet distance between those two Gaussians. Lower means the feature distributions are closer.

The forward corruption is fixed and Gaussian, so any step is one shot: x_t = sqrt(abar_t)*x_0 + sqrt(1-abar_t)*eps, with abar_t the running product of (1-beta). Training draws a random t, builds x_t, and regresses eps.

An autoregressive model factorises the joint into conditionals over an ordering. Training scores all of them in one masked pass because the ground truth is already there; sampling must draw each value before the next, one pass per dimension.

DDIM reverses the diffusion with a non-Markovian, noise-free update that matches the same training marginals, so one trained network can be run on any subsequence of timesteps. Because the update is deterministic, dropping steps degrades gracefully instead of breaking.

The reparameterization trick moves randomness off the parameter path: draw fixed noise eps, then set z = mu + sigma * eps. That makes z a differentiable function of mu and sigma, so gradients reach the encoder.

A VAE's loss adds a reconstruction term, penalising how badly the decoder rebuilds the input from its latent code, to a KL term that pulls each input's encoded Gaussian toward a standard-normal prior. The two pull against each other.

Reconstruction loss constrains the decoder only at the codes the encoder actually produced for training data. Everywhere else the code space is unconstrained, so a randomly drawn point falls in an unallocated region and the decoder extrapolates into nonsense.

Pairs give a ground-truth target, and the L1 term pins the output to it — correct layout, colours and large-scale structure. The adversarial term then supplies the sharp local detail that a reconstruction loss alone averages into blur.

Precision is the share of generated samples falling inside the real data's feature manifold, which measures fidelity. Recall is the share of real samples falling inside the generated manifold, which measures coverage. Opposite failures cannot cancel.

A linear beta schedule destroys nearly all signal two-thirds of the way through the chain, so the final third of the reverse steps teaches the model almost nothing. A cosine schedule lets signal-to-noise fall gradually to the end.

Use Bernoulli cross-entropy when every output lies in [0,1] and the decoder emits it through a sigmoid, such as scaled pixels. Use squared error for unbounded real-valued outputs. The choice declares what output distribution you are assuming.

An activation penalty limits how many units may fire per example rather than how many units exist. The layer can be wider than the input without collapsing into a copy, and each unit specialises on one recurring pattern.

Squared error is the log-likelihood of a fixed-variance Gaussian output, and its minimiser is the mean over every image consistent with the code. Averaging misaligned edges and textures cancels fine detail, which is what blur is.

For a fixed generator, the optimal discriminator is D*(x) = p_data(x) / (p_data(x) + p_g(x)). Substituting it back turns the objective into a constant plus twice the Jensen-Shannon divergence between the two distributions, which is zero only when they are equal.

It fixes the vanishing signal from a discriminator that has already won. A Lipschitz-constrained critic estimates a distance between the real and generated distributions, so its gradients stay useful and its value tracks sample quality.

FID is only meaningful inside one fixed protocol. The number moves with sample count, the feature network's weights, the resizing applied before it, and which real split you score against. Change any of them and the comparison is void.

Because distributional scores only ask whether the generated distribution matches the real one, and a copy of the training set matches it exactly. Catching memorisation needs a separate nearest-neighbour audit against the training data, calibrated against a held-out baseline.

One network learns both conditional and unconditional prediction, because the condition is replaced by a learned null token on roughly ten percent of training examples. Sampling evaluates it twice per step and extrapolates along the difference between the two predictions.

The forward jump is closed-form because it conditions on the clean sample. Backwards, many clean samples explain one noisy one, so the reverse distribution is multimodal; a single pass returns only its blurry average. Small steps keep each reverse move Gaussian.

A flow's exact likelihood comes from the change-of-variables formula, which needs the inverse map and the log-determinant of its Jacobian. A general determinant costs cubic time, so flow layers are built to have a triangular Jacobian.

At the noisiest steps the input is almost pure noise, so predicting the added noise is nearly trivial while small errors explode when converted back to an image. v-prediction blends the noise and image targets, staying informative at both ends.

Train the autoencoder on normal data only, score each window by its reconstruction error, and threshold at a high quantile of the error measured on held-out normal data. Keep the code narrow enough that the model cannot rebuild behaviour it never saw.

Gumbel-Softmax adds independent Gumbel noise to the logits and replaces the argmax with a temperature-scaled softmax. The noise is parameter-free, so gradients flow to the logits; the price is a soft sample and a biased gradient.