Why does VGG stack two 3x3 convolutions instead of using one 5x5 layer?

## The two designs being compared A convolution layer with kernel size `k`, `C` input channels and `C` output channels holds `k*k*C*C` weights plus `C` biases, and applying it to an `H x W` feature map costs roughly `k*k*C*C*H*W` multiply-adds. VGG's design rule is that `k` is always 3 with stride 1 and padding 1, so the spatial size never changes inside a block and downsampling happens only at the pooling layers between blocks. Earlier convolutional designs mixed kernel sizes freely: LeNet-5, working on 32x32 grayscale handwritten-digit images, used 5x5 kernels in both of its convolution layers. ## Receptive field arithmetic The receptive field is the region of the input that can influence a single output unit. For stride-1 layers it grows additively: `r <- r + (k - 1)`. Starting from 1, one 3x3 layer gives 3, a second gives 5, a third gives 7. So two stacked 3x3 layers see exactly the 5x5 input patch a single 5x5 layer sees, and three of them reproduce a 7x7 field. Nothing about coverage is given up. ## Parameter and compute arithmetic At `C -> C` channels the comparison is: - one 5x5 layer: `25C^2` weights; two 3x3 layers: `18C^2` — about 28% fewer. - one 7x7 layer: `49C^2` weights; three 3x3 layers: `27C^2` — about 45% fewer. Because every layer in both stacks runs at the same spatial resolution, the multiply-add ratio equals the parameter ratio. The gap widens with kernel size for a simple reason: parameters grow as `k^2` while reach grows as `k`, so a big kernel pays quadratically for a linear amount of coverage. ## The nonlinearity argument The 5x5 layer applies one linear map over the window followed by one nonlinearity. The 3x3 pair applies a linear map, a nonlinearity, a second linear map and a second nonlinearity. Same reach, twice the nonlinear stages. A function that a single linear-plus-nonlinearity stage over the 5x5 window cannot represent may be easy for the two-stage composition. This is the half of the argument candidates most often forget, and it matters more than the parameter saving. ## The honest caveat about expressiveness Composing two 3x3 linear convolutions does yield a 5x5 kernel, but not an arbitrary one. `18C^2` free parameters cannot span the `25C^2`-dimensional space of 5x5 kernels, so the factorization is a restriction of the function class, not a generalization of it. The empirical claim VGG makes is that the restriction is benign and the extra nonlinearity more than pays for it. Saying this out loud in an interview shows you understand that matched receptive field is not the same as matched expressive power. ## What the swap does not buy Training memory. The pair produces two activation maps that must both be retained for the backward pass instead of one. At high resolution, activations, not weights, dominate training memory, so the two-layer version usually costs more memory during training even though the saved model is smaller. Latency. Two layers are two sequential, dependent steps. At high spatial size and modest channel counts each one is bound by memory traffic rather than arithmetic, so measured wall-clock time can be worse than the multiply-add ratio suggests. ## Where the argument stops paying Two limits. First, a stride-1 3x3 stack grows the receptive field only linearly — two pixels per layer — so covering a large fraction of the image purely by stacking takes many layers, whereas downsampling grows the field geometrically for free. Second, the depth you can actually optimize is bounded: as plain stacks got deeper, training difficulty, not capacity, became the binding constraint, and later architectural work was aimed squarely at that. The 3x3-stacking rule is an excellent default for the layers inside a stage; it is not a licence to keep adding layers indefinitely.