After masked-reconstruction pretraining, why do you throw away the reconstruction decoder?

## Scaffolding versus the thing you are building In a masked-reconstruction setup there are two parts. The **encoder** consumes the visible portion of a corrupted input and produces a representation. The **decoder** takes that representation and predicts the content that was hidden, so that a reconstruction error can be computed against the ground truth you already have — the original, unmasked input. That error is the entire training signal, and it is the reason no labels are needed. When pretraining ends, only one of those two parts has value. The decoder implements a mapping from representation back to raw input space. Nothing downstream wants that: a classifier wants class scores, a detector wants boxes, a regressor wants a number. So the decoder is dropped, a fresh randomly-initialised head appropriate to the real task is attached to the encoder, and the whole thing is fine-tuned on whatever labelled data exists. Reconstruction was a device for extracting supervision from unlabelled data, and once the supervision has done its work the device is discarded. A useful concrete case: MAE-style pretraining on a large archive of unlabelled Sentinel-2 satellite tiles, where three quarters of each tile's patches are hidden. Nobody ever ships the pixel predictor. What ships is the encoder, later fine-tuned on a few thousand labelled tiles for land-cover classification. ## Why the decoder is usually small on purpose Because it is thrown away, every parameter and every step of compute spent on the decoder is compute not spent on the part you keep. That alone argues for keeping it lightweight. But there is a second, more interesting reason: **the division of labour between encoder and decoder is a design choice with consequences for what the encoder learns.** If the decoder is very powerful, it can do much of the inference work itself — reconstructing plausible content from a fairly shallow summary — and the encoder is under less pressure to build a rich representation. If the decoder is very weak, the encoder is pushed to keep whatever the decoder cannot recover, and some of that is low-level detail such as exact texture and noise, which is not what you want transferred either. The usual compromise is a decoder that is clearly smaller and shallower than the encoder: enough to render the prediction, not enough to substitute for representation. ## Reconstruction quality is not representation quality The most common misconception here is that the run with the lowest reconstruction error produced the best encoder. It frequently produced the worst one. A network that has learned to fill masked regions by continuing the surrounding texture reaches a low error while having learned essentially a smoothing operator, and this is exactly the failure that makes low masking ratios useless. Reconstruction error rewards pixel-level fidelity; transfer rewards abstraction. The two objectives agree only up to a point and then diverge. The practical rule that follows: use the pretext loss to confirm training is progressing and not diverging, and use downstream performance on the real labelled task to decide which pretrained encoder to keep. Never select a checkpoint on reconstruction error alone. ## What actually carries over When people say a masked-reconstruction pretext 'worked', they mean the encoder's weights place the downstream optimisation in a better region than random initialisation does — features that respond to structure, texture, layout and object-like regularities in the domain, learned without labels. That is the deliverable. Everything else in the pretraining pipeline — the masking schedule, the reconstruction target, the decoder, the loss — is machinery that produced it and then retires. One related detail worth having straight: masking is a training-time corruption, not a property of the model. At fine-tuning and inference time the input is not masked. The encoder therefore sees complete inputs downstream when it was trained on partial ones, which is a real distribution shift and one of the reasons fine-tuning matters rather than just reading off frozen features. Candidates who claim you must keep masking the input downstream have misunderstood which part of the setup was the scaffolding.