Why must a softmax classification head have exactly one output unit per class?

## What the head actually is A classification network ends with a **head**: a linear layer that takes the last hidden representation `h` — a vector of length `d` produced by whatever trunk sits underneath — and produces one raw score per class. Concretely the head holds a weight matrix of shape `K x d` and a bias vector of length `K`, where `K` is the number of classes. Class `k` owns row `k`: ``` s_k = w_k . h + b_k for k = 1..K ``` Those `K` numbers are the **logits**. Softmax then maps them to a probability distribution: ``` p_k = exp(s_k) / (exp(s_1) + ... + exp(s_K)) ``` Every `p_k` is strictly positive and the `K` of them sum to `1`. ## Why one unit per class, and not fewer Each class needs its **own** weight vector — its own direction in feature space that says "this input looks like class `k`". That is the whole content of the head. If a class has no row, no input can ever score highest for it, so the class is not merely unlikely, it is **unrepresentable**: every example of it must be filed under one of the classes you did keep. This is the single most important practical consequence of head width. Two tempting shortcuts both fail: - **One unit predicting the class index.** Regressing to the integer id imposes an ordering and a metric that the label set does not have: it makes class 3 sit "between" class 2 and class 4, so an error toward a neighbouring id looks cheaper than an error toward a distant one, and there is no way to express "probably 7, possibly 2". - **Binary-encoding `K` classes into about `log2(K)` units.** The head would emit a code rather than a distribution: nothing constrains it to a valid code, and flipping one output bit lands on an arbitrary unrelated class rather than a near miss. The `K`-unit design avoids both. It is redundant in a precise sense — the distribution is unchanged if you add the same constant to every logit — but that redundancy costs one extra weight vector and buys a clean, symmetric parameterization. ## Why not extra units Spare units are not free. They are never the true class, so training pushes their logits down, but never to minus infinity; they keep taking a slice of the probability budget, and on an unusual input a stale, undertrained row can score highest and win the argmax. They also cost `d` weights each. Width is a modelling decision determined by the taxonomy, not a hyperparameter to sweep. Note that head width has nothing to do with the width of the layer feeding it. A 7-class head fed by a 512-unit hidden layer is `7 x 512` weights plus 7 biases: `d` sets how many weights each class gets, `K` sets how many classes there are. ## Targets: one-hot rows and integer ids are the same thing The head's output is a distribution over `K` classes, so the target is naturally a distribution too: a **one-hot** row of length `K` with a `1` at the true class and `0` everywhere else. In practice labels are usually stored as a single **integer class id** instead. These are not two different setups — the id is simply the index of the `1` in the one-hot row. The training objective reads off the predicted probability of the true class either way, so the model, the head width and the gradients are identical; ids are just far cheaper to store and pass around when `K` is large. If an interviewer asks "are your labels one-hot or integers?", the correct answer is that it is a storage question, not a modelling one. ## The fixed budget: classes compete Because the `K` outputs must sum to `1`, they share a fixed budget. A class's probability can only rise if others fall. That coupling is the point: it encodes the assumption that **exactly one** of the `K` classes is correct for any input. A head over a label space where two labels can be simultaneously true is the wrong design, because you cannot raise both without robbing the rest. The budget also explains a behaviour beginners misread as a bug. Feed a 40-way spoken-language head a two-second clip that genuinely could be Portuguese or Spanish and you may see roughly `0.45` and `0.44` on those two units with the remaining `0.11` spread thinly over 38 others. Nothing has failed. The head is reporting that it cannot separate the two candidates, and the only way it could report `0.9` for one of them is to take that mass from the other. ## At inference Prediction is the `argmax` over the `K` units — the index of the largest score. The probabilities are what you look at when you need to know *how* strongly the head prefers that class, or how the runner-up compares.