Why doesn't widening a seq2seq context vector from 1,000 to 4,000 dimensions fix long inputs?

## Separate capacity from access The fixed-vector bottleneck is usually described as a capacity problem, which invites the obvious remedy: make the vector bigger. Working through why that disappoints is a good test of whether someone actually understands the failure. Three distinct things limit a single fixed context vector: 1. **Capacity** - how much can be represented at all. 2. **Access** - the decoder gets one static summary, identical at every output step, and cannot ask for the part of the source it needs now. 3. **Commitment order** - the summary is written when the encoder finishes reading, before any target token exists, so the allocation of that capacity is made blind. Widening addresses only the first. Whatever the width, the decoder still receives the same numbers at step 1 and step 40, and the encoder still had to guess what would matter. Those are properties of the wiring, not of the size, so no width fixes them. ## What widening actually buys It is not nothing. A larger budget does push out the point where the budget starts binding, so the flat part of the length-quality curve extends and the cliff moves right. If your real workload is 35-word sentences and the breakpoint sits at 30, buying a shift can be a perfectly rational short-term move. But the *shape* does not change. You still get flat-then-falling, just with the knee somewhere else, because the mechanism producing the knee is untouched. And the return diminishes: representing twice as much source detail does not obviously require only twice as many numbers, while the cost of those numbers grows faster than linearly. ## The cost side In a recurrent layer the recurrent weight matrix is hidden-width by hidden-width, so parameters in those matrices grow with the square of the width. Going from 1,000 to 4,000 is a factor of four in width and therefore roughly sixteen in recurrent parameters, and the same factor in the matrix multiply performed at every time step of every sequence. An LSTM layer holds several such matrices, one set per gate, so the constant is worse than for a plain RNN. Memory for activations rises linearly with width and multiplies by sequence length and batch size. Then there is generalization. Sixteen times the parameters on the same corpus is a large change in capacity relative to data. Without extra regularization you should expect the train-validation gap to widen, and it is entirely possible to widen the model and see held-out quality *fall* while the bottleneck symptom you were chasing is still there. ## How to decide, in practice Before spending anything, establish that the bottleneck is what is binding. Bucket held-out data by source length and check for the flat-then-cliff signature; if every length bucket is equally bad, widening is the wrong lever entirely and you should be looking at data or optimization. Then ask where your traffic actually sits. If the length distribution is concentrated below the current knee, the bottleneck is not costing you much and the money belongs elsewhere. If the bottleneck is binding and your inputs genuinely run long, rank the options by whether they change the level or the shape: - **Widen the state.** Changes the level and the knee position. Quadratic parameter cost. A stopgap. - **Feed the same context at every decoder step, or add encoder depth or a backward pass.** Helps the summary survive a long decode and enriches what goes into it, but it is still one vector written once. - **Split the input into segments and process each.** Sometimes the right pragmatic answer for very long documents, at the price of losing cross-segment context and needing a way to stitch results. - **Let the decoder read the source at every output step.** The only option that removes the single-summary constraint rather than relaxing it, which is why it is what the field adopted; the scoring mechanics belong to alignment-based attention. ## The interview signal A weak answer says widening does not work because the model would overfit. That is a real cost but a secondary one - it is a statement about data, not about the architecture. The strong answer separates capacity from access, notes that only capacity responds to width, prices the quadratic cost, and says explicitly that the fix that changes the curve's shape is a per-step read of the source rather than a bigger one-shot summary.