Why does a bias vector broadcast across a batch get a gradient summed over the batch axis?

## What broadcasting actually is Suppose a layer produces activations `Z` of shape (256, D) -- a batch of 256 rows, each of width D -- and a bias vector `b` of shape (D,) is added: `Y = Z + b`. The shapes do not match, so the smaller operand is *broadcast*: conceptually `b` is replicated into 256 identical rows, and then the addition is elementwise. Implementations do not physically copy anything, but the mathematics is exactly the copy. That word *copy* is the whole answer. Broadcasting is a `copy` op wearing an `add` op's clothes, and the local gradient of a copy is what makes the bias gradient look different from the activation gradient. ## The chain rule over a reused variable Call the upstream gradient `G = dL/dY`, of shape (256, D). Entry-by-entry, the forward is ``` Y[i][j] = Z[i][j] + b[j] ``` The variable `b[j]` appears in row 0, row 1, ..., row 255 -- 256 separate places in the graph. The multivariate chain rule says the derivative with respect to a variable used in several places is the *sum* of the derivatives through each use: ``` dL/db[j] = sum over i of G[i][j] * (dY[i][j] / db[j]) = sum over i of G[i][j] ``` because the local derivative of an addition with respect to either operand is 1. So `dL/db` is `G` summed down the batch axis, of shape (D,). And ``` dL/dZ[i][j] = G[i][j] ``` unchanged, because `Z` is used in exactly one place and addition scales nothing. Notice the asymmetry: the same op hands one operand the gradient untouched and the other a reduction of it. That is entirely down to which operand was broadcast. ## The rule to remember **A broadcast in the forward pass is a sum over the broadcast axes in the backward pass, and a sum in the forward pass is a broadcast in the backward pass.** The two operations are adjoints of each other. Once you hold that pairing, you never have to re-derive a broadcast gradient; you just ask which axes were expanded and sum over exactly those. The procedure generalises to any broadcast, not just a bias: 1. Find every axis where the operand was expanded -- an axis it did not have at all, or one it had with length 1 while the other operand had length n. 2. Sum the upstream gradient over those axes. 3. For axes that existed with length 1, keep them as length 1 so the result matches the operand's original shape; for axes the operand never had, drop them entirely. A (1, D) bias and a (D,) bias behave identically in the forward pass and produce gradients of different shapes in the backward pass -- (1, D) versus (D,) -- which is a genuine source of confusion when hand-writing a backward. ## What breaks when you forget The failure mode is worth rehearsing, because it is the classic hand-written-backward bug. - **The loud version.** You compute `dL/db = G`, hand a (256, D) gradient to a (D,) parameter, and the update step raises a shape mismatch. Annoying, but self-diagnosing. - **The quiet version.** The update `b <- b - lr * grad` itself broadcasts. Now `b` -- shape (D,) -- minus a (256, D) gradient produces a (256, D) *parameter*. Nothing errors. The bias silently stops being a bias, memory grows, and the model's behaviour becomes batch-order dependent. This is why shape assertions on gradients are worth writing even when the framework would eventually complain. - **The subtle version.** You sum over the wrong axis -- down the feature axis instead of the batch axis -- and get a (256,) gradient. If D happens to equal the batch size, even the shape check passes, and you get a model that trains, badly, for reasons nothing will report. ## Scale, not just shape One more property follows from the sum: the raw magnitude of `dL/db` grows with the number of rows summed, because it accumulates N terms. Whether that is a problem depends on how the loss aggregates over the batch -- a loss that averages over the batch already divides those N contributions back down, while one that adds them does not. The bias gradient is simply the most visible place where that batch-aggregation choice shows up, because it is where the summation over the batch is explicit rather than hidden. ## Interview framing If you are asked to hand-derive a small network's backward pass, the bias line is where interviewers watch for the sum. Saying "the bias gradient is the upstream gradient" is the giveaway that a candidate has memorised a per-example derivation and never thought about batching; saying "summed over the batch, because the bias is shared across rows" shows you understand that shared parameters accumulate.