Deriving Backprop By Hand

## The network you are deriving on Use a concrete one so nothing is ambiguous: a 3-layer MLP over 12-feature clinical triage records, predicting one of 3 acuity classes. ``` z1 = W1 x + b1 W1: 64x12 z1, h1: 64x1 h1 = f(z1) z2 = W2 h1 + b2 W2: 32x64 z2, h2: 32x1 h2 = f(z2) z3 = W3 h2 + b3 W3: 3x32 z3: 3x1 L = loss(z3, y) ``` Here `x` is one record as a column vector, `f` is an elementwise nonlinearity (any of them — the derivation does not care which), and "3-layer" means three weight matrices. ## Cut the network at the pre-activations The single decision that makes hand derivation easy is *where you define your intermediate variable*. Define it at the **pre-activation**, not at the weights and not after the nonlinearity: ``` delta_k = dL/dz_k (same shape as z_k) ``` This is the quantity people say out loud as "the delta of layer k". Everything else on the layer is one short step away from it, and the recursion between layers is a single line. The backward sweep starts at the top: the loss hands you `delta3 = dL/dz3`, whatever form that takes for your particular output head. From there each layer does the same three-part move. ## Step 1 — the weight gradient is an outer product Write `z2` entrywise: ``` z2_i = sum_j W2_ij * h1_j + b2_i ``` A single weight `W2_ij` appears in exactly one output entry, `z2_i`, with coefficient `h1_j`. So `dz2_i/dW2_ij = h1_j`, and multiplying by the delta that already sits on `z2_i`: ``` dL/dW2_ij = delta2_i * h1_j ``` Every entry of the weight gradient is (an output-side delta) times (an input-side activation). Stacked over all i and j, that is the **outer product**: ``` dL/dW2 = delta2 h1^T (32x1)(1x64) = 32x64 ``` It is an outer product and not an ordinary matmul of two big matrices because for one example, one index runs over the layer's outputs and the other over the layer's inputs, and nothing is being summed at all. The bias falls out of the same expansion: `b2_i` sits in `z2_i` with a coefficient of one, so ``` dL/db2 = delta2 (32x1) ``` ## Step 2 — push the delta back through the transpose Now ask what the *hidden activation* h1 receives. Unlike a single weight, `h1_j` feeds **every** entry of `z2`, so the chain rule sums over the output index: ``` dL/dh1_j = sum_i delta2_i * W2_ij ``` That sum contracts the first index of W2, which is exactly what the transpose does: ``` dL/dh1 = W2^T delta2 (64x32)(32x1) = 64x1 ``` This is the whole reason the transpose appears in backprop. Forward, `W2` maps a 64-vector to a 32-vector; backward, gradients travel the opposite way, from a 32-vector to a 64-vector, and `W2^T` is the only orientation with those dimensions. ## Step 3 — cross the nonlinearity elementwise `h1 = f(z1)` acts entry by entry, so its derivative is a diagonal Jacobian and the product degenerates to an elementwise multiply: ``` delta1 = (W2^T delta2) * f'(z1) elementwise, 64x1 ``` Two details are worth saying out loud, because both are common interview slips. First, `f'` is evaluated at the **pre-activation** `z1`, not at `h1` — even when a convenient identity lets you rewrite it in terms of the output value, the argument is still z1 conceptually. Second, the derivative belongs to the layer you are *entering*, layer 1, not layer 2; an off-by-one here is the classic hand-derivation bug. Then the recursion repeats: `dL/dW1 = delta1 x^T` (64x12), `dL/db1 = delta1`. ## The shape ledger Before writing any arithmetic, write the shapes. Two invariants hold at every layer and catch most mistakes instantly: - a parameter gradient has the **same shape as the parameter** — `dL/dW2` is 32x64 like W2, `dL/db2` is 32x1 like b2; - an activation gradient has the **same shape as the activation** — `dL/dh1` is 64x1 like h1. So when you are unsure whether to write `W2^T delta2` or `delta2^T W2`, you do not need to re-derive: only one of them produces a 64x1 result. ## The five-minute whiteboard version Asked to derive the second-to-last layer's update under time pressure, say this in order: define delta at the pre-activation; the weight gradient is delta times the layer's input transposed; the bias gradient is the delta; the gradient sent further back is W transposed times delta; then multiply by the earlier layer's activation derivative. State each shape as you go. That sequence is the whole of dense backprop, and every layer repeats it unchanged.