Autoencoders, GANs and Generative Models

Both estimators are unbiased, but the pathwise one differentiates the objective at the sample, so each draw carries directional information. The score-function estimator only multiplies a scalar value by a score vector, which is far noisier and needs a baseline.

Posterior collapse: a decoder strong enough to model the sentence alone gains nothing from the code, so the encoder drives every posterior onto the prior. The KL cost vanishes and the decoder degenerates into an unconditional language model.

Decide on three things: whether one direction of the mapping can be simulated to fabricate pairs cheaply, whether the task needs geometry changed or only appearance, and what a plausible-but-wrong output costs. Cycle consistency constrains the mapping; it does not guarantee it is meaningful.

Beta scales the KL term, so it picks an operating point on a rate-distortion curve, not a quality level. Fix the acceptance test first — a reconstruction budget plus a probe on known factors — then sweep and take the knee.

Not exactly. At its global optimum a linear autoencoder with a k-unit code spans the same subspace as the top k principal components, but its axes inside that subspace are an arbitrary rotation and rescaling — not orthonormal, not ordered by variance, not unique across runs.

Log-variance is unconstrained, so any real number a linear output emits is legal, and exponentiating it always yields a positive variance. It also keeps tiny variances representable and drops straight into the Gaussian KL formula, which already contains a log.

An auxiliary-classifier head rewards the generator for samples that are easy to classify, which pulls each class toward a few prototypical examples. A projection discriminator folds the label into the single adversarial score instead, so no such reward exists.

Solving the inner maximization at every generator step is prohibitively expensive, and on a finite training set a converged discriminator overfits and saturates, leaving the generator with almost no gradient. Training instead alternates a few discriminator steps with one generator step.

Spectral normalization bounds how sharp the discriminator can be, not how the two players are balanced, and batch size changes that balance. Control for generator update count first, then log per-player gradient norms and held-out discriminator accuracy.

A contractive penalty is analytic: it shrinks the encoder's derivatives with respect to the input, in every direction at once. Input corruption chases the same robustness stochastically, over a finite radius, and for the whole encode-decode function.

You build the yardstick yourself: train a domain encoder on real data and measure distribution distance in its features, back that with train-on-synthetic-test-on-real utility and physically meaningful summary statistics, and accept that the numbers are comparable only inside your own project.

Gaussian corruption assumes continuous, comparably scaled features. It suits smooth vector data such as robot action trajectories, and suits discrete tokens, hard-constrained quantities and heavy-tailed features badly - and every sample costs many network evaluations.

Likelihood and sample quality measure different things. Maximum likelihood minimises a mode-covering divergence that punishes missing data but not wasted mass, so a better likelihood is evidence of density fit, not of better-looking samples.

Cut steps and change solver first - both are free and reversible. Distillation earns its cost only when a hard latency floor sits below what any training-free sampler reaches, and you accept narrower diversity plus a retraining stage per model version.

Pick a relaxation when the code may be soft during training and the class count is modest; pick a quantized codebook when the forward pass must be discrete and a downstream model consumes the codes. Both are biased.