When would you concatenate skip features as DenseNet does rather than add them as ResNet does?

## Two ways to let a later layer see an earlier one Both families answer the same question — how do features from early layers reach late ones without being destroyed in between — and they answer it differently. **Addition (residual).** `y = F(x) + x`. The shortcut and the branch output are summed elementwise. Width is unchanged, so the block is a repeatable unit and the stack can be arbitrarily deep at constant per-block cost. The merge is destructive in the information-theoretic sense: given `y` you cannot recover `x` and `F(x)` separately. In practice this is fine, because the branch learns a small correction and the sum is dominated by the carried signal. **Concatenation (dense).** Each layer's output is appended to everything produced before it inside the block, and each layer reads the whole stack. Nothing is overwritten: a feature computed at layer 2 is still verbatim available at layer 20, and the layer that consumes it decides how much weight to give it. The cost is that width grows. ## The arithmetic of growth With a **growth rate** `k` — the number of channels each layer contributes — the input width to layer `l` inside a dense block is `k0 + k*(l-1)`, where `k0` is the block's input width. Growth is linear in depth, not exponential, because each layer adds only `k` channels. But linear growth still means later layers read very wide inputs, so dense blocks put a 1x1 convolution in front of each 3x3 to squeeze the stack back down before spatial convolution, and place **transition layers** between blocks — a 1x1 convolution with a compression factor plus average pooling — to stop the width from running away across the network. The striking empirical result is parameter efficiency: because features are never recomputed, each layer only has to produce something *new*, and `k` can be small. Dense networks match residual accuracy with substantially fewer parameters. ## Where it costs you - **Activation memory.** Every layer's output must remain resident for the rest of the block because later layers concatenate it. Residual addition can release the summands as soon as the sum exists. Naive implementations quadruple memory versus a comparable residual model; shared-buffer strategies that recompute concatenations on the backward pass recover much of it, at the cost of extra compute. - **Bandwidth, not arithmetic.** Repeatedly reading a growing stack is memory-bandwidth work, and modern accelerators are far better at dense arithmetic than at moving tensors. Fewer parameters and fewer multiply-accumulates therefore do not translate into faster training or inference. - **Irregular shapes.** Every layer inside a block has a different input width, so the block is not a uniform repeated unit. That complicates surgery, pruning and hardware scheduling in ways a residual stack does not. - **Ecosystem gravity.** Residual backbones are the ones with the widest set of pretrained weights and downstream recipes. For most transfer-learning work that alone decides it. ## How to choose Ask what the constraint really is. - **Parameter count or model file size is the binding constraint**, depth is moderate, and you can afford training memory: concatenation is attractive. - **Throughput, latency, or very large depth is the constraint**: addition. Constant width per block is what lets a backbone go to a hundred-plus layers with predictable cost, and it is why the additive pattern became the default merge in deep stacks generally. - **You need features at multiple semantic levels preserved exactly** — a decoder that must recover fine detail, a head that consumes several scales: concatenation is the honest choice, because addition would force incompatible representations into one tensor. - **Small data, feature reuse matters, training memory is available**: dense connectivity acts as a mild regularizer through reuse and often does well. A reasonable hybrid exists and is common: use addition as the repeated in-block merge for depth, and reserve concatenation for the few places where you deliberately want to preserve distinct sources rather than blend them. ## The interview answer Say that addition holds width fixed and merges destructively while concatenation preserves everything and grows width linearly with depth; that concatenation buys feature reuse and parameter efficiency and pays in activation memory and bandwidth; and that the choice follows from which resource is actually scarce. Then note that parameter count is not a cost measurement — the dense model with fewer weights may well be the slower one. That last point is what separates a candidate who has trained these from one who has read about them.