Why do gradients vanish across time steps in a simple RNN trained with BPTT?

## The setup A simple ("vanilla") recurrent network carries a hidden state across time with one reused weight matrix: ``` a_t = W_h h_(t-1) + W_x x_t + b h_t = tanh(a_t) ``` `W_h` is the recurrent matrix, `W_x` maps the input at step t, and `tanh` is applied elementwise. The crucial word is *reused*: the same `W_h` appears at step 1 and at step 200. ## What backprop through time actually multiplies Suppose the loss is computed at the final step T. The chain rule gives the sensitivity of that loss to a hidden state far in the past, at step k: ``` dL_T/dh_k = (dL_T/dh_T) * prod_{t=k+1..T} (dh_t/dh_(t-1)) ``` and each factor in that product is ``` dh_t/dh_(t-1) = diag(tanh'(a_t)) * W_h ``` where `tanh'(a) = 1 - tanh(a)^2`. So the long-range gradient is a product of (T - k) matrices that are all the *same* `W_h` scaled by a diagonal of activation derivatives. A product of many near-identical matrices is essentially a matrix power, and matrix powers behave exponentially. ## The two shrinking factors **The activation derivative.** `1 - tanh^2` peaks at exactly 1, and only when the pre-activation is exactly zero. Anywhere else it is strictly less than 1, and once a unit sits in the flat part of the curve it is close to 0. So this factor can only ever hold the product steady or shrink it — it can never amplify. **The recurrent matrix.** If the largest singular value of `W_h` is below 1, then combined with a derivative bounded by 1 every backward step is a strict contraction, and the product decays geometrically in the lag. This is the sufficient condition for vanishing. Concretely, an effective per-step factor of 0.9 over 200 steps is 0.9^200 ~ 7e-10; the gradient reaching step 1 is nine orders of magnitude smaller than the one reaching step 199. ## Exploding is the same equation If the recurrent spectrum is large the product runs the other way: 1.1^200 ~ 1.9e8. A spectral radius above 1 is a *necessary* condition for exploding gradients, not a sufficient one — saturating `tanh` derivatives can still contract a matrix whose radius exceeds 1. The practical point is that vanishing and exploding are not two problems; they are one mechanism, repeated multiplication, seen at the two signs of the exponent. The knife-edge where the product neither grows nor shrinks is a measure-zero regime that training does not stay in. ## What does *not* vanish A common confusion is to conclude that the gradient on `W_h` is zero. It is not. Because `W_h` is shared, its gradient is a **sum** of contributions over every (loss step, source step) pair: ``` dL/dW_h = sum_t sum_(k <= t) (dL_t/dh_t) (dh_t/dh_k) (dh_k/dW_h) ``` The lag-1, lag-2, lag-5 terms are perfectly healthy. Only the long-lag terms are annihilated. The sum is therefore dominated by recent history, and training proceeds happily — it just optimises a short-memory model. This is why the failure is described as *long-range credit assignment dying first*: the model's effective memory horizon is roughly where the per-step factor has driven the contribution below the noise floor of the other terms, and everything beyond that horizon is invisible to the optimiser. ## Why it is hard to notice Exploding gradients announce themselves: a loss spike, an overflow, a step that destroys the parameters. Vanishing gradients produce no error at all. The loss curve looks normal, the gradient norm looks normal (the short-lag terms supply it), and the only symptom is that a dependency the task genuinely requires never gets learned. Someone who has multiplied `tanh`'s peak derivative of 1 by itself 200 times and watched a healthy-looking loss curve at the same time understands why the diagnosis has to be done deliberately rather than read off the training log. ## The interview framing Say the mechanism (repeated multiplication by one Jacobian), name the two factors (`tanh` derivative bounded by 1, and the recurrent spectrum), give the exponential arithmetic, and note that vanishing and exploding are the same expression. Then add the subtlety that the total weight gradient survives while the long-lag component does not — that last point is what separates a memorised answer from an understood one.