Why does a transformer layer use many attention heads instead of one wide head?

## The constraint that motivates multiple heads Attention computes, for each token, one probability distribution over positions and one weighted average of value vectors. That is a strong restriction. A distribution that sums to one is a scarce resource: putting 0.9 on the referent of a pronoun means having 0.1 left for everything else in the sentence. So a single head expresses essentially one relation per token. Sentences contain many relations at once. In "the trophy didn't fit in the suitcase because it was too big", the token "it" simultaneously participates in a coreference relation (to "trophy"), a subject relation to "was", and a position in a clause introduced by "because". A model that can encode only one of these per layer is handicapped. Making the head wider does not lift the constraint. Increasing d_k gives richer queries and keys and increasing d_v gives a richer payload, but the head still produces one distribution over positions. The bottleneck is the softmax, not the vector width. ## What multi-head actually does The layer splits its width. With model dimension d_model and h heads, each head typically projects to d_k = d_model / h. Every head has its own W_Q, W_K and W_V, runs the same scaled-dot-product computation independently on the same input, and produces its own (n, d_v) output. Those h outputs are concatenated back to width d_model and passed through a final output projection W_O before being added to the residual stream. Two consequences follow from the d_model / h convention. First, parameter count is roughly unchanged relative to a single head of full width — this is a re-partitioning of capacity, not an increase. Second, there is a real tension: at fixed model width, more heads means narrower heads. Each head gets a lower-dimensional subspace to work in, and past some point individual heads become too thin to represent a useful matching function. Head count is a genuine hyperparameter with a sweet spot, not something to maximize. The output projection deserves a mention because it is often skipped. Concatenation alone would leave each head's contribution confined to its own slice of the output vector. W_O mixes across slices, letting downstream computation combine what different heads found. ## What individual heads specialize in Interpretability work on trained models consistently finds recognizable, human-nameable behaviours in some heads. Common examples include heads that attend almost entirely to the immediately preceding token, heads that attend to the start-of-sequence token as a kind of no-op sink when nothing is relevant, heads that track matching delimiters or brackets, and heads that carry syntactic dependencies such as a verb attending to its subject. A coreference-flavoured head in the trophy sentence would put most of "it"'s attention mass on "trophy". Three honest caveats belong with that story. Specialization is emergent, not designed — nothing in the architecture assigns roles, and a different training run produces a different assignment. Many heads have no clean interpretation at all, and pruning studies have repeatedly shown that a substantial fraction of heads can be removed from a trained model with modest loss in quality, which suggests real redundancy. And attention weights alone are an incomplete account of a head's function: what the head *writes* through its value and output projections matters as much as where it looks. ## How to talk about this in an interview The strong answer has three beats. First, the constraint: one head, one distribution, one relation. Second, the mechanism: split the width into independent subspaces, run attention in each, concatenate and project. Third, the tradeoff: head count trades the number of relations a layer can attend to against how expressive each one can be, at roughly constant parameter cost. A weak answer says "more heads means the model can look at more things" and stops there. It is not wrong, but it does not identify *why* one head cannot look at more things, which is the actual question being asked. ## A practical framing If you are choosing head count for a model you control, the useful mental model is: total attention capacity per layer is roughly fixed by d_model; heads decide how you slice it. Many narrow heads suit tasks with many simple positional or syntactic relations; fewer wide heads suit tasks where each match needs a richer notion of similarity. In practice the field converged on head dimensions in the 64-128 range and derives the head count from the model width, which is a reasonable default to state if asked.