Structured Channel Pruning

## The unit of removal is the whole point Pruning means removing capacity from a network that has already been trained. *Structured* pruning fixes the unit of removal at something the tensor shape can see: an entire output filter of a convolution (equivalently, one output channel of its feature map), an entire attention head, or an entire residual block. Nothing is left behind holding a zero. The layer that comes out has a genuinely smaller weight tensor — a convolution that produced 256 channels now produces 192, and its weight tensor went from `256 x C_in x k x k` to `192 x C_in x k x k`. When you delete output filter *j* of a layer, several things travel with it: that filter's bias entry, the per-channel scale, shift and running statistics of the normalization layer that follows it, and — crucially — the *j*-th input slice of every kernel in every layer that consumes this feature map. Structured pruning is a graph edit, not a mask. ## Why the speed-up materialises A convolution or a linear layer executes as a dense matrix multiply, tiled to fit the machine's registers and caches. Making one of the matrix dimensions smaller keeps the operation in exactly that regime: same kernel, same tiling, fewer tiles. There is no index array to decode, no gather of scattered values, no metadata to carry alongside the weights, and no dependence on hardware support for a particular sparsity pattern. The smaller model is also literally smaller on disk and in memory, which matters when weight traffic — not arithmetic — is what the layer is waiting on. ## The arithmetic For a convolution, parameters are about `C_out * C_in * k * k` and multiply-accumulates about `C_out * C_in * k * k * H_out * W_out`, where `H_out` and `W_out` are the output spatial dimensions. Pruning channels changes `C_out` and `C_in`; it does not change `k`, `H_out` or `W_out`. Now notice that one cut lands twice. If you keep a fraction `(1 - p)` of layer L's output channels, layer L's own cost falls by that factor — and layer L+1's input dimension falls by the same factor, so if L+1 is also pruned on its outputs, its cost falls by about `(1 - p)^2`. Across the interior of a network where every layer is pruned at both ends, a 30% channel cut is closer to a 50% cost cut than a 30% one. This quadratic effect is the reason moderate-looking channel ratios produce large compute reductions. ## Where the win under-delivers Fewer operations is not the same as less time, and a candidate who cannot say why is missing the practical half of the topic. - **Bandwidth-bound layers.** A layer whose runtime is dominated by reading activations and weights rather than by multiplying gains far less than its operation count suggests. Early high-resolution layers and pointwise operations often sit here. - **Fixed overheads.** Per-layer launch cost, normalization, activation functions, memory copies and the input pipeline do not shrink with width at all. If they are a third of your frame time, the best a width cut can do is shrink the other two-thirds. - **Alignment.** Accelerators run best at channel counts that are multiples of the hardware's preferred tile — often 8, 16 or 32. Pruning to 183 channels can be *slower* than pruning to 176, because the last partial tile is paid for in full. Round pruned widths to the machine's granularity. - **Batch size.** At batch size one, which is what a live stream gives you, layers are far more likely to be latency- and overhead-bound than compute-bound, so measured gains are smaller than an operation-count spreadsheet promises. ## Width is not the only structural unit Removing whole residual blocks — depth pruning — deletes entire sequential stages along with their normalization, activation and launch overhead. Because it removes work that width pruning cannot touch, an equal nominal reduction taken from depth frequently converts into wall-clock better than one taken from width: dropping several blocks from an image backbone can deliver a measured 1.6x end-to-end speed-up. The trade is that each block is a coarse, capacity-destroying cut, and depth is what lets features compose. ## The accuracy bill The unit that makes structured pruning fast also makes it blunt. Inside a filter you are removing, some weights were doing nothing and some were doing real work; you take them all. So per parameter removed, a structured cut costs more accuracy than a finer-grained one. The standard remedy is recovery: fine-tune the pruned network on the training data so the surviving channels re-adapt. Expect to prune and recover, not to prune and ship. ## What to actually do Decide the target on the deployment device, not on paper. Measure end-to-end latency at the real batch size and input resolution before and after; treat operation counts only as a search signal. Round pruned widths to hardware-friendly multiples. And always report the accuracy after recovery training, never the accuracy immediately after the cut — the raw post-cut number is meaningless.