Why does He initialization use a weight variance of 2/fan_in for ReLU layers?

## The quantity being preserved Initialization scale is not a matter of taste; it is the solution of a small variance equation. Take one layer: `y = W x + b`, with `fan_in` incoming connections per output unit, weights drawn independently with mean zero and variance `Var(W)`, and biases at zero. `fan_in` is the number of inputs each output unit sums over; `fan_out` is the number of outputs each input feeds. Because the weights are zero-mean and independent of `x`, the pre-activation of one output unit is a sum of `fan_in` independent zero-mean terms: ``` Var(y) = fan_in * Var(W) * E[x^2] ``` Read `fan_in * Var(W)` as the **per-layer multiplier**. If it is greater than one, the signal grows geometrically with depth; if less than one, it shrinks geometrically; if exactly one, it holds. ## Glorot's answer for a near-identity activation A tanh unit is approximately the identity for small inputs, so if the signal stays small the activation barely changes the second moment: `E[x_next^2] is about Var(y)`. Setting the multiplier to one gives `Var(W) = 1/fan_in`. The backward direction gives a different answer. The signal travelling back through the same layer is summed over `fan_out` terms, so preserving *its* scale wants `Var(W) = 1/fan_out`. Unless the layer is square, no single number does both. Glorot's compromise splits the difference: ``` Var(W) = 2 / (fan_in + fan_out) ``` which for a square layer collapses back to `1/fan_in`. The uniform version draws from `-sqrt(6/(fan_in+fan_out))` to `+sqrt(6/(fan_in+fan_out))`, since a uniform distribution on `[-a, a]` has variance `a^2/3`. ## Where the factor of two comes from ReLU is not near-identity. It passes positive inputs unchanged and maps negatives to zero. If the pre-activation `y` is symmetric around zero -- which it is, since the weights are symmetric and zero-mean -- then exactly half its probability mass is discarded and the surviving half contributes its usual share: ``` E[relu(y)^2] = 0.5 * E[y^2] = 0.5 * Var(y) ``` So the recursion across a ReLU layer is `Var(y_next) = fan_in * Var(W) * 0.5 * Var(y)`. Setting that multiplier to one: ``` fan_in * Var(W) / 2 = 1 -> Var(W) = 2 / fan_in ``` That is the whole derivation, and the factor of two is nothing more exotic than "ReLU throws away half the second moment, so put twice as much in". Note it is a *forward* rule keyed to `fan_in`; the backward variant keyed to `fan_out` exists and, in practice, either works, because the discrepancy is a factor of the width ratio, not of depth. ## What getting it wrong costs, quantitatively Take a 20-layer ReLU stack for seismic-waveform regression, width 256, initialized with the tanh-style `Var(W) = 1/fan_in`. The per-layer multiplier is `256 * (1/256) * 0.5 = 0.5`. Each layer halves the second moment, so at layer 20 the variance is `2^-20` of the input's and the typical activation magnitude is about `2^-10`, near one thousandth. The last layers are computing on numerical dust, the output is nearly constant across examples, and the run looks "slow" rather than broken. Swap to `2/fan_in` and the multiplier is `256 * (2/256) * 0.5 = 1`: the scale is flat all the way down. The symmetric failure is a fixed standard deviation. Consider a 30-layer plain tanh MLP over 60-channel machine-vibration sensor windows, hidden width 5,000, every weight drawn at a fixed 0.01 standard deviation because that "looks small". The multiplier is `5000 * (0.01)^2 = 0.5` -- activation variance halves at every layer, and by layer 30 the signal is gone. Widen the layer to 20,000 with the same 0.01 and the multiplier becomes 2 and the signal explodes into tanh saturation instead. A fixed standard deviation is not a scale rule at all: it silently means something different at every width, which is exactly why the rules are stated in terms of fan-in. ## The assumptions, and where they leak The derivation assumes zero-mean independent weights, independence between weights and inputs, and a symmetric pre-activation distribution. These are true at step zero and progressively less true afterwards -- correlations build up as training proceeds. That is fine: the rule is a rule for *step zero*, whose job is to get the run onto a reasonable trajectory, not to hold for all time. One more consequence worth internalising: the rules are about **variance**, so the choice of Gaussian versus uniform is essentially cosmetic. What matters is the second moment and the fact that the draw is random across the width. ## Why the choice feels less critical in modern networks Architectures that rescale each layer's output absorb a constant multiplicative error in the incoming weights, so a mis-scaled initialization that would have killed a plain stack often merely costs some early progress. The scale still matters where such layers are absent or infrequent, in very deep stacks where per-layer errors compound before the first rescaling, and because the weight magnitude interacts with how large a *relative* step each update makes. Treating an explicit fan-in-based rule as free insurance is the right default -- it costs one line and removes an entire failure class.