Why does U-Net concatenate encoder feature maps into its decoder instead of upsampling alone?

## What downsampling costs Each stride-2 stage halves the spatial resolution and doubles the physical area each feature position summarises. After five such stages the feature grid is 32 times smaller: one position covers a 32x32 patch of the image, and the *location* of anything inside that patch is known only to within about half a stride. In exchange, that position has a large receptive field and can encode abstract, class-level information. This is the central trade of a segmentation encoder: **resolution buys precision, downsampling buys context**, and you cannot have both from one feature map. A decoder that only upsamples is therefore doing something impossible. Bilinear interpolation, or a learned transposed convolution, can make the grid bigger, but it can only smooth between values that already exist. The high-frequency information — the exact pixel column where one region stops and another begins — was destroyed on the way down. The visible symptom is characteristic: masks look approximately right at object scale, but edges are rounded, thin structures vanish, and two objects separated by a thin gap merge. ## The skip connection U-Net's answer is to route information sideways. The architecture is symmetric: for every encoder stage at a given resolution there is a decoder stage at the same resolution. When the decoder upsamples to, say, 128x128, it **concatenates along the channel axis** the encoder's 128x128 feature map from the corresponding depth, then applies convolutions to the merged stack. What each side contributes is different in kind: - The **encoder feature** at that resolution is spatially precise. It was computed before the later downsampling stages, so its edges are still where the image's edges are. But it is shallow, so it encodes texture and gradient rather than class identity. - The **decoder feature** is spatially coarse but semantically rich. It knows this region is a cell interior, not background, because it was computed from a large receptive field. The convolution after the concatenation is what actually combines them. It is free to learn, per channel, how much to trust the sharp-but-dumb signal against the smooth-but-informed one, and that weighting differs across the image and across classes. Nothing hand-tunes this; it is learned along with everything else. ## The concrete failure it prevents Take 512x512 microscopy images where cells touch. The class labels are cell and background, and the entire difficulty is the one-to-two-pixel membrane between adjacent cells. A plain encoder-decoder learns the cell texture easily and produces a confident mask, but the membrane sits well below the resolution its deepest features retain, so the mask covers two cells as one region. With skip concatenations, the 512x512 and 256x256 encoder features still carry the intensity discontinuity at the membrane, and the final decoder convolutions can cut the mask there. The difference is not a small metric gain; it is whether downstream counting of cells works at all. ## Concatenation, and its costs Concatenation is not free. It multiplies the channel count entering each decoder convolution, so both parameters and activation memory grow, and the encoder activations must be **kept alive** through the whole forward pass instead of being released stage by stage. On high-resolution inputs this, not the encoder, is often what determines the largest batch that fits. The alternative merge is summation, which requires the two feature maps to have the same channel count and commits in advance to weighting them equally. It is cheaper and sometimes adequate, but it fixes the combination before any learning happens, whereas concatenation defers the decision to the following convolution. ## Related effects Skips also give gradients a short route from the loss back to early encoder layers, which helps a deep encoder-decoder optimise. That is a real benefit, but it is secondary here — the reason the architecture exists is the information path, not the gradient path. A practical detail worth knowing: skips only work if the encoder and decoder tensors at a stage have matching spatial sizes. Odd input dimensions, or padding choices that round differently on the way down and the way up, produce a mismatch at the concatenation, which is a shape-arithmetic problem rather than a design problem. ## What weak answers get wrong - Claiming skips exist to fight vanishing gradients. That is a side effect. The architecture is motivated by recovering spatial detail. - Claiming they copy the input image forward. They copy learned features from a specific encoder depth, not raw pixels. - Assuming more skips are always better. Skipping from a very shallow layer injects mostly noise and raw texture and can make boundaries jittery rather than sharp.