Which hand-derived backprop mistakes survive a shape check and still break training?

## Why the shape check is not enough Writing the shape ledger before the arithmetic is the right habit, and it catches a lot. But a shape check answers only one question — *is this product legal, and does the result match the thing being updated* — and there are whole families of derivation errors it cannot see. Knowing which ones they are is what separates someone who has actually hand-derived a backward pass in anger from someone who has only read the formulas. ## The shape-silent family **1. A flipped transpose between equal dimensions.** The transpose in `W^T delta` is forced by the shapes only when the layer's input and output widths differ. A layer of 128 units feeding another 128 units makes `W` square, and then `W delta` is just as conformable as `W^T delta` while being the wrong gradient. Gradients now flow through the forward map instead of its adjoint. This is the single most common shape-silent bug, and the reason it hides is that debugging networks are so often built with uniform widths. **2. The outer product written backwards.** `dL/dW = delta h^T` becomes `h delta^T` — the transpose of the correct gradient. On a square layer both are legal shapes, so the update applies happily and moves the weights in a systematically wrong direction. **3. A dropped elementwise activation derivative.** Multiplying elementwise by `f'(z)` does not change any shape, so omitting it changes nothing a shape check can see. What it does is remove the nonlinearity's contribution from every delta below that point, so all lower layers train on a gradient that ignores which units were in a saturated or inactive regime. **4. The activation derivative at the wrong layer.** Using `f'(z2)` where `f'(z1)` belongs is an off-by-one in the recursion. When the two layers have the same width, the shapes line up perfectly, and every lower layer is corrupted while the top layer looks fine. **5. Sum where the loss took a mean.** If the loss is defined as an average over the batch but the derivation accumulates a plain sum, every gradient is larger by the batch factor. Shapes are untouched. This one does not point the update in a wrong direction — it just rescales it — which is why it is the most insidious of the five: training still works, just at an effective step size that silently tracks the batch size. **6. Pairing the delta with the wrong activation.** `dL/dW2 = delta2 h1^T` uses the *input* of layer 2. Reaching for `h2` instead — the output of the same layer — is a natural slip, and when the widths coincide it is shape-clean. ## Making the bugs loud The strongest defence is structural rather than diagnostic: **derive and test on a network with all-distinct layer widths**. Take input 7, hidden 13, hidden 5, output 3. Now a square matrix does not exist anywhere in the network, so every flipped transpose, every reversed outer product and every mispaired activation becomes an immediate conformability failure instead of a slow training mystery. Uniform widths are convenient for real models and terrible for debugging derivations; use ugly widths deliberately for the derivation, then move to real ones. A second structural move: run the backward pass with batch size 1 and again with batch size 2 on duplicated rows. Everything that treats the batch axis correctly gives the same parameter gradients when the loss averages, or exactly doubled ones when it sums. A batch-axis bug shows up as neither. ## Separating scale errors from direction errors When training is bad, the first split to make is *is the gradient pointing the wrong way, or merely scaled wrong*: - A **scale** error (the mean/sum factor) keeps the direction correct. The loss still falls, the run is unusually sensitive to the step size, and rescaling the step size by the batch factor makes the pathology disappear entirely. Nothing else about the run looks strange. - A **direction** error does not respond to step-size tuning in that way. Shrinking the step size just slows down whatever wrong thing is happening; the loss stalls, wanders, or diverges across a wide range of step sizes. ## Localising by layer One more piece of leverage: the layers are not symmetric in how much of the derivation they exercise. The top layer's parameter gradient uses only the delta the loss handed back and that layer's input — it does not use the recursion at all. Every layer below it additionally uses the transpose step and the elementwise activation step. So if the top layer's gradients behave and everything below is broken, the fault is almost certainly in the recursion — the transpose orientation or the activation-derivative factor — and not in the loss or in the outer-product rule. If even the top layer is wrong, look at the delta the loss produces or at the outer product itself. That single split usually cuts the search space in half in one step. ## What to say in the interview Name two or three concrete shape-silent bugs, then immediately give the structural defence: distinct layer widths so the shape check regains its teeth, batch-size-1-versus-2 to expose batch-axis handling, and the scale-versus-direction symptom split to classify what you are looking at. The interviewer is checking whether you know that "the shapes match" is a statement about conformability and not about correctness.