In a 200-block residual network, why zero-initialize each block's final scale parameter?

## What goes wrong at 200 blocks A residual block adds a computed correction to its input: `out = x + branch(x)`. The input path is an exact copy, so the block's output is at least as large as its input. If the branch is randomly initialized and active, it contributes a roughly independent term, and independent contributions add in variance: ``` Var(out) = Var(x) + Var(branch(x)) ``` Stack that 200 times and the residual stream's variance grows monotonically with depth -- linearly if each branch emits an order-one signal regardless of its input, geometrically if each branch's output scales with its input. Either way the network's output at step zero is nothing like the scale the loss expects, the first steps are spent undoing the accumulated growth rather than learning anything, and the very deep stack is fragile precisely where it was supposed to be strong. Note what is *not* the fix: shrinking the per-layer initialization inside the branch. The problem is not one branch being too big; it is 200 correct-sized branches summing. You want the depth-dependent term controlled, and per-layer fan-in rules know nothing about depth. ## The zeroed scale Most residual branches end in a multiplicative learnable parameter -- a per-channel scale, or a single learnable scalar placed there for the purpose. Initialize *that one parameter* to zero and the branch's output is identically zero for every input. Then: - `out = x + 0 = x`, so every block is exactly the identity; - the 200-block stack passes its input straight through to the head; - the network's output scale at step zero equals the scale set by the stem, independent of depth. The network begins life as a shallow network -- a stem and a head -- and grows deeper as training proceeds. That is a much better-behaved starting point than a 200-deep random function. ## Why the branch is not permanently dead This is the objection worth being ready for, and the answer is a one-line derivative. Write the branch output as `g * u(x)` where `g` is the zeroed scale and `u` is everything before it. The gradient with respect to `g` is the incoming gradient dotted with `u(x)`, and `u` is *randomly* initialized, so `u(x)` is nonzero and generically not orthogonal to the incoming gradient. `g` therefore moves off zero on the very first step. The gradient with respect to the inner weights is proportional to `g`, so those weights are frozen at step zero and start moving as soon as `g` becomes nonzero. The dynamics that follows is the point of the trick: a block only invests in its inner weights once its own gate has decided the block is worth something. Blocks turn on in rough order of usefulness rather than all shouting at once. This also explains why zeroing the *whole* branch is a mistake rather than a stronger version of the same idea. If `u`'s weights are zero too, `u(x)` is zero, the gradient with respect to `g` is zero, and the branch really is dead -- and you have reintroduced the symmetry problem inside the branch on top of it. Exactly one parameter is zeroed; everything before it stays random. ## Alternatives when there is no scale parameter to zero If the branch has no trailing multiplicative parameter, the same effect is available by construction: - **Insert a learnable scalar** at the end of each branch, initialized at zero. Cheap: one parameter per block. - **Divide each branch's output by a fixed `sqrt(L)`**, where `L` is the number of blocks. Summing `L` independent contributions each scaled by `1/sqrt(L)` keeps the total added variance order one no matter how deep the stack. This does not give an exact identity start, but it makes the stream's scale depth-independent, which was the underlying goal. - **Downscale the branch's own last weight matrix** by a depth-dependent factor. Same idea, less explicit, harder to audit later. ## When it does not apply The trick works *only* because an identity path carries the signal past the zeroed branch. Zero a layer's output scale in a plain stack with no skip connection and you have zeroed the forward signal entirely: everything downstream sees zero, the gradients arriving from below are zero, and nothing recovers. The skip is what makes a switched-off block harmless rather than fatal, so "zero the last scale" is a residual-architecture technique and not a general initialization rule. ## How you verify it One check, before training: run a batch through the untrained network and compare the tensor entering the first block with the tensor leaving the last. Under a correct zeroed-scale initialization they are identical to numerical precision. If they are not, some branch has a live path you did not account for -- an unzeroed scale, a bias after the scale, or a branch whose output is not gated by the parameter you zeroed. That check takes a minute and catches the whole class of mistakes here.