Why does a sentiment RNN learn the final clause of a review but ignore the opening sentence?

## The symptom A recurrent sentiment classifier trains cleanly: the loss falls, the gradient norm looks ordinary, validation accuracy is respectable. But inspection shows it is essentially reading the tail of the review. Reviews that end '...but the ending ruined it' are classified negative even when the first four sentences are glowing, and reviews whose verdict is stated up front and then qualified are classified badly. Nothing in the training log says anything is wrong. ## Why the early steps get no signal The gradient of the loss with respect to a hidden state k steps in the past is a product of k per-step Jacobians, each of which is the shared recurrent matrix scaled by a diagonal of activation derivatives. Multiplying that many near-identical factors makes the magnitude exponential in the lag. With an effective per-step factor below 1, the contribution from a token 200 steps back is many orders of magnitude below the contribution from a token 5 steps back. Gradient descent can only strengthen a pathway it receives signal about. A dependency whose gradient contribution is 1e-9 relative to its neighbours is, for optimisation purposes, not there. So the parameters that would make the opening sentence matter never move in a coordinated direction — not because the model decided the opening was uninformative, but because it never got told. ## Why the loss looks healthy anyway Two things conspire to hide it. **The gradient is a sum over lags.** Because one recurrent matrix is shared across all steps, its gradient aggregates contributions from every (loss, source-step) pair. The lag-1 to lag-30 terms are full size. The total gradient is therefore normal in magnitude and points somewhere sensible — it just contains no information about long-lag structure. Monitoring the global gradient norm will never surface this. **Short-range cues carry a lot of the label.** Natural review text is redundant: sentiment words cluster, and the last clause is often the summary judgement. A short-memory model can get most of the way on that alone. So the loss curve and the headline metric both reward the shortcut, and the model converges to it. The failure only shows up on the slice of examples where the shortcut is wrong. This asymmetry is worth stating explicitly in an interview: exploding gradients are a loud failure — a loss spike, an overflow, a wrecked parameter vector — while vanishing gradients are silent. The training log cannot distinguish 'learned the dependency' from 'never received a gradient about it'. ## How to confirm it Do not diagnose from the loss curve. Three checks that produce evidence: 1. **Position-bucketed evaluation.** Partition the evaluation set by where the decisive cue sits — reviews whose sentiment is determined in the first sentence versus in the last. If accuracy collapses on the early-cue bucket and is fine on the late-cue bucket, the model's memory horizon is the story. 2. **Sensitivity as a function of lag.** Measure the norm of the loss gradient with respect to the input representation at step k, and plot it against the lag from the loss. A clean exponential decay is a direct measurement of the effective horizon: the lag at which the curve hits the noise floor is roughly how far back the model can see. 3. **Occlusion.** Blank out the opening sentence at inference and see whether predictions change at all. If they are numerically identical, the model was never using it. ## What the horizon actually is The effective memory horizon is where a step's gradient contribution falls below the other terms in the sum. It is set by the per-step contraction factor, not by the sequence length you feed in. Feeding longer sequences does not extend it; it just adds steps whose gradient contributions are already negligible. That is the trap in 'we'll fix it by using the whole document' — the extra tokens cost compute and change nothing about what the optimiser can learn. It is also worth being precise that this is a *training* pathology, not an inference one. The forward pass really does carry information from step 1 into step 200 — imperfectly, but it is there. What is missing is the learning signal that would have taught the network to preserve the *useful* part of it. The network is capable of long memory in principle; it was never trained into it. ## The interview framing Name the mechanism (exponential decay of the per-lag gradient), explain why the aggregate loss and gradient norm hide it (the sum over lags is dominated by recent steps and short-range cues fit most of the data), and then propose a measurement rather than an opinion. Candidates who jump straight to 'the dataset is biased toward endings' without checking gradient reach are guessing.