Why is a depthwise-separable convolution roughly nine times cheaper than a dense 3x3 layer?

## The dense baseline A convolution with kernel `k x k`, `C_in` input channels and `C_out` output channels has one `k x k` plane of weights for every (input channel, output channel) pair: ``` params_dense = k * k * C_in * C_out macs_dense = H_out * W_out * params_dense ``` Every filter does two jobs at once: it aggregates a `k x k` spatial neighbourhood *and* it combines all `C_in` channels. Depthwise-separable convolution is the hypothesis that those two jobs can be done sequentially by two cheaper layers with little loss. ## The factorisation **Stage 1, depthwise.** One `k x k` filter per input channel, each applied to its own channel only. This is a grouped convolution with the group count equal to `C_in`. It produces `C_in` output channels (with a depth multiplier of 1) and does no channel mixing at all: ``` params_depthwise = k * k * C_in ``` **Stage 2, pointwise.** A dense 1x1 convolution mapping `C_in` channels to `C_out`, which does no spatial aggregation and all of the channel mixing: ``` params_pointwise = C_in * C_out ``` A normalization and a nonlinearity are normally applied after each stage, so the block is not a linear collapse of the two. ## The ratio ``` params_sep / params_dense = (k*k*C_in + C_in*C_out) / (k*k*C_in*C_out) = 1/C_out + 1/(k*k) ``` Two terms, each with a clean reading. `1/(k*k)` is what you save by not repeating the spatial kernel for every output channel. `1/C_out` is what you save by not repeating the channel mixing at every spatial offset. With a 3x3 kernel and any width beyond a few dozen channels, `1/C_out` is small and the ratio sits just above `1/9`. Worked concretely for `k = 3`, `C_in = 128`, `C_out = 256`: ``` dense = 9 * 128 * 256 = 294,912 depthwise = 9 * 128 = 1,152 pointwise = 128 * 256 = 32,768 separable = 33,920 ratio = 33,920 / 294,912 = 0.115 -> 8.69x fewer ``` Because both stages operate at the same output resolution, the multiply-accumulate ratio is identical: multiply either parameter count by `H_out * W_out`. The kernel size drives the answer. At `k = 1` the factorisation saves nothing (there is no spatial kernel to factor out). At `k = 5` the ratio is `1/25 + 1/C_out`, roughly 23x for a wide layer. Larger kernels make separability more attractive, which is why compact designs pair depthwise stages with kernels they would never afford densely. ## Where the parameters end up Note the split in the worked example: the depthwise stage holds 1,152 weights and the pointwise holds 32,768 — over 96 percent of the block. The stage people talk about is the cheap one; the stage that actually costs is the 1x1. That matters when reasoning about what to shrink next, and it explains why designs that further attack cost tend to attack the pointwise stage. ## What you give up The factorisation is a structural constraint, not a free lunch. A dense filter can learn a spatial pattern that exists only in a particular *combination* of input channels. A separable block cannot: the spatial filtering happens per channel, before any mixing, so the spatial pattern each channel is scanned for is fixed independently of how channels will later be combined. In practice the loss is modest at typical widths, and the saved budget usually buys back more than it costs — more channels, more layers, or higher input resolution — but on small or heavily structured problems the dense layer can genuinely win. A second consequence is optimisation behaviour: a depthwise filter receives gradient from a single channel's activations, so a dead or badly scaled channel has nowhere to hide. Depthwise stages are more sensitive to weight decay and to aggressive regularisation than dense ones, because there are so few weights per channel that any shrinkage is felt directly. ## Stride and shape When the block downsamples, the stride is applied in the depthwise stage and the pointwise stays at stride 1. The usual output-size arithmetic still applies to the depthwise stage: `out = floor((n + 2p - k)/s) + 1`. The pointwise stage never changes spatial size; it only sets the output depth. ## The one-line answer Separating a convolution into a per-channel spatial filter followed by a 1x1 channel mixer replaces a `k*k*C_in*C_out` cost with `k*k*C_in + C_in*C_out`, a ratio of `1/C_out + 1/(k*k)`, which for a 3x3 layer of ordinary width is about one ninth.