What does the causal mask do in a decoder-only transformer, and what breaks without it?

## What the mask is Self-attention as defined lets every token attend to every token. For a model that generates text left to right, that is wrong: predicting token 5 while looking at token 5 is not a prediction. The causal mask (also called the autoregressive or look-ahead mask) is the restriction that turns unrestricted attention into next-token prediction. Mechanically it is applied to the score matrix, after the scaled dot product and before the softmax. For a sequence of length n, build a lower-triangular pattern: entry (i, j) is allowed if j <= i and forbidden otherwise. Forbidden entries are set to negative infinity. The softmax exponentiates, e to the minus infinity is zero, so those positions receive exactly zero weight and contribute nothing to the weighted sum of values. Each output row is a mixture over the current token and its predecessors only. Note it is applied to the scores, not to the weights after the softmax. Zeroing weights after normalization would leave the remaining weights summing to less than one; masking before the softmax renormalizes over the allowed positions automatically. ## Why it makes parallel training possible This is the part worth saying explicitly in an interview, because it is the real payoff. Given a training sequence of n tokens, the model is asked to predict token 2 from token 1, token 3 from tokens 1-2, and so on — n-1 prediction problems. Without masking you would have to run n-1 separate forward passes over progressively longer prefixes. With masking, one forward pass over the full sequence computes all of them at once, because row i of the attention output was computed as if only the first i tokens existed. The loss is then averaged over all positions. That single trick is a large part of why transformers train efficiently at scale. ## What breaks if you drop it The failure is diagnostic and easy to recognize. Training loss falls implausibly fast, often approaching zero within a small fraction of an epoch, because the task has become trivial: position i can attend to position i+1 and read off the answer it is being scored on. Perplexity on the training data looks superb. Then generation produces incoherent output, because at inference time there is no token i+1 to copy — the model is being asked to do a task it never actually learned. A held-out evaluation that also runs teacher-forced will *not* catch this, since it leaks the same way; the tell is the gap between teacher-forced metrics and free-running generation. A subtler version of the same bug appears when the mask is off by one — allowing position i to see i+1 — or when it is applied in some layers and not others. The symptom is the same shape: too-good loss, bad generation. ## Causal mask versus padding mask These are different masks that are often combined into one tensor, and conflating them is a common interview stumble. The causal mask depends only on position and is identical for every sequence in the batch. A padding mask depends on the data: when sequences of different lengths are batched together and short ones are padded, the padding positions must be masked out so real tokens do not attend to filler. In practice the two boolean patterns are combined and applied together, but they exist for unrelated reasons — one enforces the direction of information flow, the other cleans up a batching artifact. ## Where causal masking does not apply Encoder stacks used for embedding or classification are bidirectional by design: to represent a sentence for retrieval or sentiment, every token should see the whole sentence, so no causal mask is applied. The same architecture with the mask removed is a fundamentally different objective. This is one of the clearest ways to state the encoder/decoder distinction in terms of mechanism rather than diagram. ## Consequences at inference Because the mask enforces that a token's representation depends only on itself and its predecessors, appending a new token never changes the representations already computed for earlier ones. Generation is therefore a strictly left-to-right process, and the training condition matches the deployment condition exactly. That property is also what a well-behaved implementation relies on when it processes a long prompt and then emits tokens one at a time. ## The one-line version "Set the upper triangle of the score matrix to negative infinity before the softmax. Position i then attends only to positions 1 through i, which is what lets one forward pass train every next-token prediction in the sequence — and without it the model just copies the answer and produces garbage when generating."