In a sliding-window model, why do attention sinks keep an endless stream coherent?

## The setup: why anyone slides a window Attending over the full history gets more expensive the longer the history gets, and the per-token state you must keep grows without bound. For a session with no natural end — tailing production logs, a monitoring assistant that has been running for days, a live transcription — full attention is not an option, because there is no length at which it stops growing. A sliding window caps the span: each token attends only to the last N tokens. Cost per step becomes constant and memory becomes bounded. This is an extension strategy in the practical sense — it lets a model process an arbitrarily long stream — even though it does not increase how much the model can actually *use* at once. ## The failure that surprised everyone The naive sliding window degrades catastrophically at a very specific moment: not gradually as the session lengthens, but sharply, the instant the window slides past the beginning of the sequence. Perplexity spikes and the output turns to noise, even though the model still has a full window of perfectly good recent context. That timing is the clue. Nothing about the recent content changed. What changed is that the first tokens left the window. ## Why the first tokens matter out of all proportion Softmax normalises attention weights to sum to one. A head therefore *cannot* decide to attend to nothing — if none of the visible tokens is relevant to what it is looking for, its mass still has to land somewhere. Empirically, models learn to park that surplus on the earliest tokens of the sequence. These tokens work as sinks for structural reasons, not semantic ones. They are visible to every subsequent position, and because they come first, they are the one anchor every head can rely on being there. Their actual content barely matters — the effect persists when they are unremarkable filler. So the first few tokens end up carrying a large, semantically meaningless share of attention across many heads and layers. Evict them and that mass does not vanish; softmax redistributes it onto real content tokens, inflating their influence far beyond what the head intended. Every representation built on top is distorted, and the distortion compounds through the layers. ## The fix StreamingLLM's remedy is almost trivially simple: keep the first few tokens resident permanently and slide the window over everything after them. The attended set becomes "a handful of pinned initial tokens, plus the most recent N". The sinks stay available, heads keep their dumping ground, and perplexity stays flat over streams far longer than the model's nominal window. A cleaner variant, used when you control training, is a **dedicated sink token** — a learned placeholder prepended to every sequence whose entire job is to absorb surplus attention. Because it is trained for the role, one such token can replace several natural ones, and it removes the dependence on whatever text happened to start the sequence. ## The caveat that separates a good answer from a shallow one Sinks restore *stability*, not *memory*. The model is not remembering the pinned tokens' content in any useful sense — they are numerically load-bearing, not informative. Anything that scrolled out of the sliding window is genuinely unavailable, and no amount of sink pinning brings it back. So for a never-ending log-tailing session, this technique means the assistant keeps producing coherent commentary indefinitely instead of degenerating after an hour. It does not mean it can answer "what was the first error you saw this morning". If you need that, you need something else entirely: persist the important observations outside the model and re-supply them, or use a model whose usable span actually covers the horizon you care about. Stating that distinction unprompted is the mark of someone who has actually deployed this rather than read the abstract. ## Where this sits among extension strategies It is worth being clear about the taxonomy, because interviewers probe it. Position rescaling plus long-sequence training extends how much the model can genuinely reason over — it costs training compute and it degrades short-prompt quality. Sliding windows with sinks extend how long a session can *run* at bounded cost, while leaving the amount usable at any instant fixed. They answer different questions, and a system that streams forever but must also recall the whole stream needs both a bounded-cost mechanism and an external memory. Architectures that interleave a small number of full-attention layers among many local-attention layers sit between the two: most layers pay the bounded local cost, while the occasional global layer preserves genuine long-range routing. That interleaving is a design-time decision, not something you retrofit onto a shipped checkpoint.