What does a sequence model lose about token order when recurrence is removed?

## Order as a by-product of the computation A recurrent layer consumes a sequence one position at a time: `h_1` from `x_1`, `h_2` from `h_1` and `x_2`, and so on. Because the update is applied in sequence, position is baked into the arithmetic. Feed in the tokens of *dog bites man*, then feed in the same three tokens as *man bites dog*, and the two runs pass through different intermediate states and end at different final states. Nobody added a 'position feature'. The order of operations *is* the position information. This is why a recurrent model can learn that a word two tokens back matters, or that a sentence's first token is a question word. Distance is implicit as well: the state at position t has been transformed t-k times since it absorbed the token at position k, so 'how long ago' is encoded in how much processing has intervened. ## What a position-agnostic layer does instead Now consider a layer that produces the output at each position from a weighted combination of *all* positions' representations, where the weights depend on the content of those representations and not on where they sit. Written as a function over a set of vectors, the operation contains no index arithmetic at all. Such a layer is **permutation-equivariant**. Formally: if P is a permutation of the positions, then layer(P(x)) = P(layer(x)). Apply any reshuffling to the inputs and the outputs are the very same vectors, reordered to match. The layer has literally no way to tell an input from its shuffle. Two consequences follow immediately. 1. **Absolute position is gone.** There is no notion of 'the first token' or 'the third token'. Anything the task needs about position — sentence-initial capitalisation, the fact that a subject precedes a verb — is unavailable. 2. **Relative distance is gone.** The layer cannot express 'two tokens to the left'. Every position is equidistant from every other in the operation itself. If the model then pools across positions (sums or averages the outputs), permutation-*equivariance* becomes permutation-*invariance*: shuffled inputs produce a bit-for-bit identical result. That is exactly a bag-of-tokens model, and it cannot separate *dog bites man* from *man bites dog* at all. ## Giving the order back The standard remedy is to make position part of the data rather than part of the computation: attach a position-dependent signal to each token's representation before the mixing layer runs. Once each vector carries a marker of where it came from, the layer's weighted combination can condition on it, and the permutation symmetry is broken — the inputs are no longer interchangeable, because their representations now differ by position even when their tokens do not. How that signal is best constructed is a design question in its own right; the point for this topic is that it must exist, and that it is an explicit cost you took on the moment you removed the recurrence. ## Why the trade was still worth it Order came free with recurrence, but it came bundled with a strictly sequential unroll that cannot be spread over hardware. Adding an explicit position signal is cheap — a per-position vector, a handful of extra numbers per token, no serial dependency introduced. Trading an implicit property for an explicit input in exchange for making the whole sequence computable at once is a very good deal, and understanding *what* was traded is the substance of this question. ## The common misconceptions The first is to say 'the model will learn order from the data'. It cannot: permutation equivariance is a property of the function class, not of the fit. No setting of the weights breaks a symmetry that is built into the operation. The second is to assume token embeddings already encode position — they encode *identity*; the same word maps to the same vector wherever it appears, which is precisely why a separate position signal is needed. The third is to say a position-blind layer's outputs are unchanged by shuffling — they are the same *values*, but at permuted positions; the outputs become genuinely identical only after a pooling step that discards position. ## How to answer under pressure One sentence for the mechanism (recurrence encodes order in the order of operations), one sentence for the loss (a position-agnostic layer is permutation-equivariant, so absolute position and relative distance both vanish), one for the fix (position must be supplied as an input signal), and one for the framing (this was one of two free properties given up in exchange for parallel training).