Bottleneck and Reconstruction

## What an autoencoder is actually optimising An autoencoder is two networks trained together. The **encoder** maps an input `x` of dimension `d` to a **code** (also called the latent vector or bottleneck representation) `z` of dimension `k`. The **decoder** maps `z` back to a reconstruction `x_hat` of dimension `d`. Training minimises a reconstruction loss — for instance the mean squared difference between `x` and `x_hat` — over unlabelled data. There are no labels: the target is the input itself. Read that objective literally and the difficulty appears immediately. The loss says *reproduce the input*. It says nothing about discovering structure, disentangling factors, or compressing. If the architecture permits the network to pass information through unchanged, then the identity map is a perfect solution: zero loss, zero learning. ## Undercomplete: the constraint does the work A code is **undercomplete** when `k < d`. This is the classic bottleneck. Because `z` holds fewer numbers than `x`, exact reconstruction of arbitrary inputs is impossible — the encoder is a many-to-one map and information must be thrown away. Training then becomes a competition for scarce capacity: which `k` numbers, when handed to the decoder, produce the lowest average reconstruction error over the training distribution? Concretely, take 784-pixel handwritten digit images pushed through a 32-unit code. Thirty-two numbers cannot express 784 independent pixel values, but real digit images are nowhere near arbitrary — they occupy a tiny, highly structured region of pixel space. The lowest-error use of 32 numbers is therefore to describe *what varies across digits*: which digit shape, how thick the stroke, how slanted, how large. Pixel-level noise and background, which vary little or unpredictably, get dropped because encoding them buys almost no error reduction. The compression is not the goal the loss states; it is what the loss produces once the capacity is restricted. ## Overcomplete: the failure this predicts A code is **overcomplete** when `k >= d`. Give a network 2000 code units over 784-dimensional inputs and nothing in the objective stops it from learning an approximate identity: the encoder scatters the input across the wide code, the decoder gathers it back. Reconstruction loss falls close to zero on training *and* held-out data — this is not overfitting in the usual sense, because copying generalises perfectly to any input at all. The model trains beautifully and encodes nothing. This is the single most useful diagnostic to carry into an interview: **reconstruction loss is a training signal, not a quality metric for the representation**. A model that scores 0.001 with a 2000-unit code has learned strictly less than a model that scores 0.02 with a 32-unit code. If you select code width by picking the width with the lowest reconstruction error, you will always select the widest one available, which is precisely the useless end of the range. Depth does not rescue an overcomplete code either. A ten-layer encoder feeding a 2000-unit code still has an unobstructed path to the identity. Capacity along the width of the bottleneck is what matters, not the number of layers around it. ## What the bottleneck actually buys, and what it costs Narrowing the code buys three things. First, forced selection: only reconstructable structure survives, which is the sense in which the code is a learned summary of the data. Second, a smoother error signal about the data itself — the reconstruction error of a *narrow* model on a new input tells you something about how well that input matches the training distribution, because the model has no spare capacity to rebuild anything it has not learned. Third, a fixed, small representation size, which is what makes the code cheap to store and index. It also costs something. Too narrow and reconstruction error is dominated by capacity starvation: the decoder outputs an over-smoothed average and the code cannot separate cases you care about. So the width is a genuine hyperparameter, chosen against the downstream requirement and held-out behaviour, not against training loss. ## A narrow layer is not the only possible constraint The general principle is that *something* must make exact copying unprofitable. Width is the simplest such constraint and the one meant by the word 'bottleneck'. Other families of constraint exist, imposed on the code or the training signal rather than on the layer size, and they let a wider code still learn something; those are separate methods with their own tradeoffs. But an unconstrained overcomplete autoencoder trained on plain reconstruction has no reason to learn anything at all, and usually does not. ## How to check a trained model Compare held-out reconstruction against a trivial baseline such as predicting the per-dimension training mean; a model that barely beats that has not learned enough. Perturb one code unit and see whether the reconstruction changes in a coherent way, which tells you the unit carries signal rather than a routed copy of a pixel. And measure the code on the job you actually want it for. Reconstruction error alone will not tell you.