Why does an LSTM's additive cell-state update keep gradients alive over long sequences?

## Two routes backwards Training a recurrent model means propagating an error at step `T` back to a step far earlier. Whether that arrives with usable magnitude depends entirely on what sits between the two states on the backward path. In a plain recurrent cell the only route is through the state itself, and the state is recomputed each step as `h_t = tanh(W h_(t-1) + U x_t + b)`. Differentiating one step gives two factors stacked in the way: the recurrent weight matrix, and the derivative of the squashing nonlinearity. Both are applied at **every** step, so going back `k` steps means `k` copies of both. Each copy typically shrinks the signal a little, and a little compounded a few hundred times is nothing at all. A gated cell adds a second route. The cell state is updated as ``` c_t = f_t * c_(t-1) + i_t * g_t ``` and along this route the local derivative is simply `f_t`, elementwise. No weight matrix. No activation derivative. Just a per-dimension number in (0, 1). ## Why addition is the whole trick The reason the derivative is that clean is that `c_(t-1)` enters the new state **additively**, not through a transformation. In a sum, each term's derivative is untouched by the other term — the `i_t * g_t` half contributes nothing to how `c_t` depends on `c_(t-1)`. The old memory is scaled and passed along, which makes the operation close to the identity whenever the scale is close to one. Contrast a state that is *recomputed*: there the old value is fed through a matrix and a nonlinearity, and whatever those do to the forward value, their derivatives do to the backward one. Over `k` steps the direct cell-state route contributes ``` product of f over the k steps ``` for each dimension. If a dimension holds its forget gate at 0.999 the product over 400 steps is about 0.67 — the gradient arrives at two thirds strength. At 0.99 it is about 0.018, already a two-percent signal. At 0.9 it is roughly 5e-19, which is nothing. So the additive design buys a route whose decay is set by a *learned* quantity that can be pushed arbitrarily close to one, instead of by fixed weights and a fixed nonlinearity. ## The honest caveats Three things separate this from a magic fix, and interviewers listen for them. **It is conditional, not automatic.** The highway is open only where the model has learned `f` near 1. Nothing forces that. This is the reason the forget-gate bias is often initialised positive: it starts every dimension biased toward keeping, so gradients can reach far back early in training, before the gates have learned anything. **The cell-state route is not the only route.** The gates themselves are computed from `h_(t-1)`, so there are additional backward paths that do run through weight matrices and squashing functions and do shrink the way a plain recurrence shrinks. The additive path is the one that survives; the analysis above is about that dominant path, not about every term in the exact gradient. **Forgetting is a feature.** A cell that learns a small forget value has decided that dimension's history is irrelevant, and killing the gradient along it is correct behaviour, not a bug. What the architecture removes is the *obligation* to shrink; it does not remove the ability. ## Exploding is a separate story An additive route helps against decay but does not by itself bound growth. Because forget values sit at or below one, the cell-state path cannot amplify along its own route, but the paths through the gate weights can still blow up, and a cell state that keeps accumulating without forgetting can grow large in the forward pass too. Gradient clipping is still standard practice with recurrent models for exactly this reason. ## How to say it One sentence carries the whole answer: *the cell state enters the next cell state additively, so the local derivative along that route is the forget gate alone, and a forget gate near one is a near-identity that lets gradient travel hundreds of steps.* Then add the caveat that the route is learned rather than guaranteed, and you have covered both what the design achieves and what it does not.