At generation time, what does a recurrent model's fixed-size state buy over a position-parallel model?

## Two cost profiles, side by side Take hidden size d and a sequence of length T. **Recurrent.** One step performs a handful of matrix multiplies against fixed-size weight matrices, so its cost is on the order of d-squared and does not depend on t. Memory carried between steps is a single vector of size d (plus a cell state for a gated cell). Generating T tokens costs about T times d-squared, and peak state stays constant. Crucially, the cost is the same whether the answer depends on the token just emitted or on one from a thousand steps ago — reaching further back costs *nothing extra per step*, because everything the model retained is already in the state. **Position-parallel.** There is no carried summary. Producing the output at position t requires combining that position against the representations of positions 1 through t-1, so per-step work grows with t and the retained representations grow with t as well. Summed over a full generation, total work grows faster than linearly in T and peak memory grows with T. So the profile flips. For a recurrent model a longer sequence means *more steps but not more work per step*. Without recurrence a longer sequence means more steps *and* more work in each of them. ## Why this does not undo the training win The key asymmetry is between training and generation. During training on a known sequence, every target position is available in advance, so a position-parallel layer can compute all of them at once — that is the whole wall-clock argument. A recurrent layer cannot, because its own outputs are its inputs. During autoregressive generation, neither design can escape being serial in tokens: you must emit token t before you can condition on it. Removing recurrence buys *nothing* in that dimension. What differs is only what one step costs. That is why a model can be dramatically cheaper to train and simultaneously more expensive per emitted token than the recurrent design it replaced — and why 'it removes the sequential bottleneck' is only true of training. ## The quality side of the same trade A fixed-size state is a lossy compression of the past. Everything the model wants to remember competes for the same d numbers, and anything overwritten is irrecoverable, which is what makes very long dependencies hard for a recurrent model however well it is gated. Keeping every position's representation is the opposite bargain: exact, nothing forgotten, and unbounded. So the choice is not merely cheap-versus-expensive; it is bounded-and-lossy versus exact-and-growing. A candidate who names only the compute axis has seen half of it. ## Operating consequences a senior engineer should raise - **Memory becomes a function of session length.** With growing per-position state, a long conversation or a long document is not just slower, it can exhaust memory mid-stream. You need a cap, an eviction policy, or a hard session limit — decisions the constant-memory design never forces on you. - **Latency drifts within a single request.** Early tokens are fast and late tokens are slower, so a p99 measured on short outputs does not predict long ones. Benchmark at the output length you actually serve. - **Batching economics change.** When retained state per sequence grows with length, the number of concurrent sequences you can hold falls as they progress, which complicates throughput planning. - **Streaming and small targets favour bounded state.** For an indefinite stream where quality demands are modest and the memory ceiling is hard, constant per-step cost and flat memory remain a real argument for recurrence. ## How to answer State the two profiles with their scaling, name the flip (more steps versus more steps *and* more work per step), then make the senior point: the serialisation did not disappear, it moved from training — where it was pure waste, because the sequence was already known — to generation, where token-by-token serialisation is unavoidable regardless of architecture. Close with the memory and latency consequences you would plan for in a deployment, and with the lossy-versus-exact framing, which shows you see the compute trade and the modelling trade as the same decision.