Why does an inverted residual block expand the channel count with a 1x1 before projecting back?

## Two ways to build a bottleneck A classic residual bottleneck block, as used in the deeper ResNet variants, is wide at its ends. It takes a wide tensor, uses a 1x1 convolution to *reduce* the channel count, performs the spatial convolution in that cheap narrow middle, then uses a second 1x1 to *expand* back to the wide width so the residual can be added. The design assumes the expensive operator is the spatial convolution, so the way to save is to run it on few channels. An inverted residual block turns that inside out. Its inputs and outputs are the *narrow* tensors. The first 1x1 convolution *expands* the channel count by a small factor, the spatial work happens in the wide middle, and the second 1x1 *projects* back down to the narrow width. The residual connection joins the narrow input to the narrow output. That reversal - narrow ends, wide middle - is what the word "inverted" refers to. ## Why expanding is affordable The design only makes sense when the middle spatial operator is one whose cost scales linearly with the number of channels rather than with the square of it, because each channel is processed independently rather than mixed with all the others. Under that condition, doubling or sextupling the middle width multiplies the middle operator's cost by the same small factor, not by its square. All the quadratic channel-mixing cost is concentrated in the two 1x1 convolutions, and those are attached to a narrow side, so their cost is (narrow width) x (expanded width) rather than (wide) x (wide). So the block spends its width where width is cheap and stays thin where width is expensive. That is the whole economic argument. ## Why the nonlinearity wants a wide space There is a representational argument on top of the cost argument. A rectifier-style activation zeroes every negative coordinate. In a high-dimensional space that is a mild operation: the information carried by a zeroed coordinate is typically still recoverable from the many other coordinates the representation lives in, because the useful signal occupies a lower-dimensional subspace than the tensor it sits in. Squeeze the representation down until it has almost no redundant directions, then apply the same rectifier, and the zeroing is no longer recoverable - each destroyed coordinate was carrying signal nothing else duplicates. This gives the block its second rule: put the activations in the expanded middle, where they have room, and leave the final projection **linear** - no activation after the down-projection. This is the linear bottleneck. It looks counterintuitive to anyone who has learned "more nonlinearity is better", but the narrow tensor is the block's carrier of information between blocks, and clipping it costs more than the extra nonlinearity gains. ## Where the residual attaches, and why The skip runs between narrow tensors. Two benefits follow. First, the tensor that must be kept alive from the start of the block to its end - and therefore held in memory during the forward pass, and during backpropagation - is the small one, not the expanded one. In a network built from many such blocks, peak activation memory is dominated by what crosses block boundaries, and here that is thin. Second, the added tensors are the ones the linear bottleneck has kept information-preserving, so the residual is adding two clean, uncompressed-by-a-rectifier representations. The skip is only used when it can be: the block must have stride 1 and matching input and output channel counts. A block that downsamples spatially or changes width omits the residual. ## The expansion factor as a dial The expansion factor - how many times wider the middle is than the ends - is the block's main capacity knob. A larger factor gives the nonlinearity more room and typically raises accuracy, but it multiplies the parameter and arithmetic cost of both 1x1 convolutions and inflates the intermediate activation tensor, which is the block's peak-memory term. Small integer factors are the usual choice, and the factor can reasonably differ between early blocks (where spatial resolution is high, making the wide intermediate expensive in memory) and late ones. ## The common misreads One is thinking "inverted" describes the skip connection running in some unusual direction; it does not, it describes the wide/narrow profile of the block. Another is adding an activation after the projection because it seems free - it is not free, it is the specific thing the design removes. A third is treating the expansion as a cost-free way to add capacity: the wide intermediate tensor is exactly where the block's activation memory peaks, and at high input resolution that can bind before parameters do.