Why can't a GRU learn a dependency 3,000 steps back even with well-behaved gates?

## What gating actually fixed In an ungated recurrent cell, backpropagating a gradient from step `T` to step `t` multiplies together `T - t` Jacobians. If their typical magnitude is below 1, the product decays geometrically and long-range gradients vanish; above 1, it explodes. Gates change this because the state update becomes an interpolation, `h_t = (1 - z_t) * h_(t-1) + z_t * cand_t`. With the update gate near 0 for a unit, the derivative of `h_t` with respect to `h_(t-1)` along that unit is close to 1, so the gradient can travel many steps roughly undamped. That is the whole of what gating buys, and it is real. It is not the same as being able to *use* information from 3,000 steps back. Four separate obstacles remain. ## Obstacle 1 — fixed-size state, unbounded content The hidden state is a fixed vector of `H` numbers. Everything the model needs from the entire past — the antecedent, the current clause, the syntactic state, whatever else — must be encoded in those `H` numbers simultaneously. Capacity does not grow with distance or with sequence length. A long novel does not get a bigger memory than a short paragraph. So carrying a specific rare fact for thousands of steps means dedicating part of a scarce, shared resource to it for the whole span, and every intervening write competes for the same room. ## Obstacle 2 — the gate must stay shut for the entire span For a unit holding the antecedent, the update gate must remain near 0 at every one of the 3,000 steps. If it opens even briefly — because some intervening token looked relevant — the stored value is blended away and cannot be recovered. And "near 0" is not the same as 0: the carried coefficient is a product of `(1 - z_t)` terms. At 0.999 per step, the value after 3,000 steps is scaled by roughly `0.999^3000`, which is about 0.05. Gating slows the decay by orders of magnitude; it does not abolish it. ## Obstacle 3 — no content-based retrieval A gated recurrent cell reads only its own previous state. There is no operation that says "go find the position that best matches this query and bring it back." Anything the model wants at step 3,000 must have been continuously carried forward from step 0 through every intermediate state. That is fundamentally different from a mechanism that can address earlier positions by content, and it is the deepest of the four limits: it is architectural, not an optimisation problem. ## Obstacle 4 — the training signal is truncated and rare Two practical facts finish the job. **Truncation.** Backpropagation through time is almost always truncated to a window of tens to a few hundred steps, because storing activations for the full unroll is expensive. A dependency spanning 3,000 steps sits entirely outside that window: the gradient linking the pronoun's error to the weights that stored the antecedent is never computed. The model is not failing to learn the pattern — it is never shown a learning signal for it. **Sparsity of the pattern.** Even with a long enough window, dependencies at that range are rare in text. A handful of examples per corpus produce a tiny gradient contribution against millions of local-context examples, and the optimiser very reasonably spends capacity on the frequent pattern. ## How to diagnose it in practice If you suspect a long-range failure rather than a general capacity problem, the useful checks are: measure error as a function of the distance between the cue and the target and see whether it degrades with distance specifically; compare against a model given the cue at a short distance, which isolates retrieval from the task itself; and check the truncation length actually used in training against the distances you care about, because that number alone often explains everything. ## The honest conclusion Widening the hidden state helps a little — more room to store things — and lengthening the truncation window helps a little, at a memory cost. Neither turns a gated recurrent cell into something that can reliably retrieve an arbitrary fact from thousands of steps back. The right answer in an interview is to say which of the four limits you think binds for the case in front of you, and to be clear that gating addressed the gradient path only.