Backpropagation and Autodiff

A max passes the incoming gradient straight to whichever input won and gives exactly zero to the loser. For ReLU that is a 0/1 mask: positive inputs pass the upstream gradient through unchanged, non-positive ones cut it to zero.

Max pooling sends a window's entire upstream gradient to the one input that was the maximum and exactly zero to the others. Average pooling spreads it evenly, so each input in a k-by-k window receives one k-squared-th of it.

A reused tensor appears in several terms of the chain rule, so its gradient is the sum of one contribution per consumer. The reverse sweep adds each contribution into a single buffer as it reaches that consumer.

The forward pass records a directed acyclic graph whose nodes are ops and whose edges are the tensors between them. The backward sweep replays it in reverse topological order: a node is visited only after every op that consumed its output.

Perturb one parameter by plus and minus a small h, recompute the scalar loss each time, and compare (L_plus - L_minus)/(2h) with the analytic gradient for that coordinate. Score the agreement by relative error, not by raw difference.

Global-norm clipping rescales the whole gradient by a single factor once its norm passes a threshold, so the update direction is unchanged. Elementwise value clipping clamps each component on its own, which rotates the update toward the diagonal.

Start at layer 2's pre-activation with delta2 = dL/dz2. Then dL/dW2 is the outer product delta2 h1^T, dL/db2 is delta2 itself, and the hidden layer receives W2^T delta2, which you multiply elementwise by f'(z1).

With G as the upstream gradient of the loss with respect to C, dL/dA = G B^T and dL/dB = A^T G. Matmul has two separate local rules, one per operand, and each carries exactly one transpose.

A convolution's input gradient is itself a convolution: the upstream gradient against the kernel rotated 180 degrees. The kernel gradient correlates the layer input with the upstream gradient, summed over every position the kernel visited.

Reverse mode costs one sweep per output; forward mode costs one per input. Training has one scalar loss and a hundred million inputs, so a single reverse sweep gets every gradient, while forward mode would need a hundred million.

Backpropagation multiplies one Jacobian per layer, so the gradient reaching layer 1 is a product of 50 factors. If those factors average below one the gradient decays geometrically with depth; above one it grows the same way.

The bias gradient is a 128-length vector, the same shape as the bias itself. All 64 records contribute their own delta to the same shared bias, so the batch axis is summed away rather than kept.

Automatic differentiation applies exact per-operation derivative rules to numeric values as the program runs, giving machine-precision derivatives at one input point. Symbolic differentiation manipulates formulas and can blow up in size; numerical differentiation perturbs inputs and carries approximation error.

It is the classic signature of exploding gradients: one update was large enough to push weights into a range where activations overflow to infinity, and infinity arithmetic then produces NaN. Once NaN reaches the weights, the run never recovers.

Two errors pull in opposite directions. A central difference's truncation error falls like h squared, but subtracting two nearly identical losses and dividing by 2h amplifies round-off like 1/h. In double precision the total is smallest near h = 1e-5.

Broadcasting copies the same bias into every row of the batch, so every row contributes a separate term to the same parameter. The backward of a copy is a sum, so those contributions add up into one bias-shaped gradient.

The sum of the upstream gradient vectors from every occurrence. The lookup gathers rows, so the backward scatter-adds into them. Overwriting instead of accumulating would apply only the last occurrence and silently discard the rest.

A hard argmax is flat between its jumps, so its derivative is zero almost everywhere and undefined at ties. Reverse mode multiplies by that zero, and any parameter whose only route to the loss crosses it gets an exactly-zero gradient.

Each op stashes only what its own backward step reads: a linear op keeps its input, tanh keeps its own output, ReLU needs nothing more than the sign of its input, and a plain addition keeps no tensor at all.

The loss is piecewise linear in that parameter with a corner between the two probe points, so the numerical secant blends both slopes while the backward pass reports one. The check fails though the code is correct.

Measure before you choose. Record the global gradient norm for the first several hundred steps with the clip effectively off, look at the distribution, and set the threshold near its upper tail — around the 90th percentile — so ordinary steps pass and only outliers get rescaled.

Shape checks only prove conformability. A transpose flipped between equal dimensions, an activation derivative dropped or taken at the wrong layer, and a sum where the loss averaged all leave every shape valid while corrupting the gradient.

Every gradient is multiplied by the batch size N. A mean reduction hands each element a local gradient of 1/N; a plain sum hands it 1. The descent direction is identical, the step is N times larger.

Seven orders of magnitude across the depth means the gradient is decaying geometrically layer by layer. Only the top few layers learn; the rest still hold their initial weights, acting as a fixed random feature map. Effective depth is about four.

Whenever the huge gradient has a findable cause. Clipping bounds the step but leaves the cause running, and because the run now survives, nobody investigates. It is a seatbelt for rare tail events, not a substitute for finding what produces them.

Because the batch mean and variance are functions of every sample, not constants. Differentiating through them puts batch-wide sums into each sample's gradient, so the same input gets a different gradient depending on its batch-mates.

Forward mode. One forward sweep produces one Jacobian column, so six sweeps cover the six parameters. Reverse mode produces one row per sweep and would need thousands, one per residual.

A stop-gradient passes its input forward unchanged and returns zero gradient, so the optimizer treats that value as a constant. Put one wherever a model-computed quantity should act as data, not as something the loss can lower by changing it.

Wherever a derivative was written by a human rather than composed from already-trusted primitives: a custom layer, a custom loss, a hand-optimised operation. Keep the test tiny, deterministic and in double precision so it runs in seconds and never flakes.