A third stacked recurrent layer barely improves your tagger - what do you check?

## What stacking actually does Stacking recurrent layers means the sequence is processed by layer 1 to produce an output at every time step, and that whole output *sequence* becomes the input sequence of layer 2, which produces its own output sequence, and so on. Each layer has its own weights and its own hidden state, and each carries state along the time axis independently. The motivation is representational hierarchy along the vertical axis, analogous to what depth buys anywhere else: the lower layer learns local, surface regularities of the input, the upper layer operates on a stream of already-abstracted features. On a part-of-speech tagger you can see the effect empirically - going from one layer to two usually gives a clear gain, two to four typically gives very little, and the curve flattens. The single most important mechanical fact, and the one interviews probe: **the upper layer reads the lower layer at every step, not just at the end.** If you wire the lower layer to expose only its final state and feed that upward, you have destroyed the sequence - the upper layer now sees a length-one input and the whole stack degenerates. This mistake produces exactly the symptom in the question: adding depth changes nothing. ## A diagnosis order ### 1. Wiring Confirm the lower layer emits a full output sequence and the upper layer consumes it step-aligned. Confirm the directions are consistent - a bidirectional layer under a forward-only layer is legal but means the upper layer's causal-looking input is not causal at all. Confirm the layer was actually added: parameter counts that did not change are the fastest tell. ### 2. Optimisation A third layer that is present but not learning looks identical from the metric. Check that gradients reaching layer 1 have not collapsed, and that the top layer's activations have not saturated. Deep recurrent stacks are harder to optimise than shallow ones because gradients now travel a long path in two directions - backwards through time *and* down through layers. Standard remedies are skip connections that let a layer's input bypass it and be added to its output, and normalisation applied per layer. If the third layer only pays once you add skip connections, that is a genuine finding, not a workaround. ### 3. Error analysis This is what separates a senior answer. Sample the remaining errors and classify them: - **Rare or unseen tokens.** No amount of depth invents evidence that is not in the data. This calls for better input representations or more or broader data. - **Genuine ambiguity** that even a human annotator resolves inconsistently. Check annotator agreement; you may already be at the ceiling of the label quality. - **Long-range dependence** where the disambiguating evidence sits far away in the sequence. Depth does **not** extend how far back a recurrent model can reach - that reach is a property of the recurrence along the time axis, not of the number of layers stacked on top. If this is the failing class, more layers is the wrong knob entirely. - **Class imbalance in the tag set,** where the errors concentrate in a handful of rare tags and the aggregate metric hides it. If the errors are dominated by the first three, capacity is not your constraint and the third layer was never going to help. ## The cost side of the ledger Even when depth does buy a little accuracy, it is not free in a way that a feed-forward stack is not. - **Latency scales with depth.** Recurrence is already sequential over time; stacking adds another full pass over all T steps. Two layers is roughly two passes. Because the time axis cannot be parallelised inside a layer, you cannot amortise this away. - **Widening the hidden state is often cheaper than deepening it** for the same parameter budget, because a larger matrix multiply per step is work that hardware parallelises well, whereas an extra layer is more sequential work. - **Memory during training** grows with depth times sequence length, since every layer's states across all steps are retained for the backward pass. So the real decision is not "does layer three help?" but "does the fraction of a point it buys survive the serving latency budget and the per-request cost target?" Frequently the answer is no, and reporting that clearly is the right outcome. ## What a strong answer sounds like "Before adding a fourth layer I would confirm the stack is wired step-wise rather than through the final state, check that the third layer's gradients are alive and try skip connections between layers, then sample two hundred errors. If they are rare words and annotation noise, depth is not my problem and I would report that the tagger is data-limited at two layers - which also happens to be the cheapest model to serve."