Per-layer gradient norms logged every 50 steps put layer 1 four orders below layer 12 — what do you check?

## The number on the dashboard is not yet a diagnosis A per-layer gradient-norm log is one of the highest-value pieces of training telemetry, and also one of the easiest to over-read. "Layer 1 is four orders of magnitude below layer 12" is a statement about two numbers, and there are three separate questions to answer before it becomes a statement about the network: is the comparison fair, does the difference have a consequence, and where exactly does the profile break. ## Step 1: make the comparison fair The usual quantity logged is the Euclidean norm of a layer's gradient, which sums squared partial derivatives over every parameter in that layer. That means the norm grows with parameter count. A wide layer with a million parameters and a narrow one with ten thousand can have identical typical per-parameter gradients and still differ by more than an order of magnitude in norm. Depth-wise comparison across layers of different shapes is therefore meaningless until you normalize. Two normalizations are worth logging: - **Per-parameter root-mean-square** — the norm divided by the square root of the parameter count. This makes layers of different widths comparable on the same axis. - **The update-to-weight ratio** — the norm of the step the optimizer is about to take, divided by the norm of the layer's current weights. This is dimensionless and is the most useful single health number: a widely used rule of thumb is that a healthily training layer sits somewhere around a thousandth. Several orders of magnitude below that means the layer is effectively frozen; several above means it is being rewritten every step. Often the alarming four-orders-of-magnitude spread shrinks to something unremarkable once one of these is applied, and there is nothing to fix. ## Step 2: ask whether it has a consequence A small gradient is not automatically a small update. An adaptive optimizer divides each parameter's step by a running estimate of that parameter's squared-gradient magnitude, so a layer whose gradients are uniformly tiny can still take steps of ordinary size — the scaling largely cancels. This is why the direct measurement matters more than the gradient statistic: log the norm of the change in each layer's weights between checkpoints, or over an epoch. If layer 1's weights are moving by a normal relative amount, the tiny gradient norm is an artifact of scale and not a pathology, whatever the dashboard suggests. The converse check is also worth making. If layer 1's weights are genuinely static while layers 10 to 12 are churning, you have a real capacity problem: you are training a shallow model with a fixed random front end, and its early features will never adapt to the task. ## Step 3: find where the profile breaks A smooth, gentle decay from output to input is normal in many architectures. What is diagnostic is a **cliff** — a place in the stack where the normalized statistic drops by orders of magnitude between two adjacent layers. Read the profile top-down and locate that boundary, then look at the activation statistics of the layer at and just above it. Two patterns show up repeatedly: - Units in that block are saturated: their outputs are pinned against a bound, so the local slope multiplying everything below is near zero. - Units in that block are dead: they emit a constant, so nothing downstream depends on what came before them through that path. That is the payoff of logging activation and gradient statistics together — one localises the break, the other names it. ## Step 4: confirm the instrumentation Before acting, verify you are reading what you think you are reading: - **Clipping.** If norms are logged after a global clip, the recorded values are compressed toward the clip threshold and the true profile is hidden. Log the pre-clip norm. - **Accumulation.** With gradients accumulated over micro-batches, a norm sampled mid-accumulation is a fraction of the real one, and layers logged at different points are incomparable. - **Reduced precision.** Very small gradient values can fall below the representable range of a low-precision format and be flushed to zero, which produces an exact-zero norm rather than a merely small one. An exact zero is a different signal from a small number and deserves separate treatment. - **Scale coupling.** If the loss is a sum rather than a mean over the batch, or if a loss term is weighted, every norm shifts by that factor uniformly — which changes absolute readings but not the shape of the profile. ## What to do with the answer If the normalized profile really does collapse and early weights really are frozen, the fix is a change to the run — the step size, the initialization scale, where normalization sits, whether there are skip paths — rather than more logging. But reach that conclusion in the order above. The single most common outcome of this investigation is that the raw norms were incomparable and the run was healthy, and the second most common is that the cliff points at one specific block whose units had already gone inert.