In knowledge distillation, what does dividing teacher and student logits by a temperature above 1 accomplish?

## The knob The temperature-scaled softmax is `q_i = exp(z_i / T) / sum_j exp(z_j / T)`, where `z` are logits and T is a positive scalar. T = 1 is the ordinary softmax. T > 1 divides every logit down before exponentiating, which shrinks the *gaps* between them and produces a higher-entropy, flatter distribution. T < 1 does the reverse, sharpening toward one-hot. In the limit of very large T every class approaches 1/C. Note what T does *not* do: it never changes the ordering of the classes, because dividing all logits by a positive constant is monotone. Argmax is unaffected. Only the relative magnitudes move. ## Why distillation needs it A well-trained teacher on a training example it has effectively memorised is close to one-hot: 0.9995 on the true class, and the remaining mass spread as values around 1e-6 or smaller. The ranking inside that tail is precisely the dark knowledge you want to transfer, but at those magnitudes it barely moves the loss — the student can match the teacher's top class, drive the rest to zero, and the objective is essentially satisfied. The similarity structure is present in the numbers and absent from the gradient. Raising T rescales the tail into a range the loss responds to. Sweeping a 100-class head illustrates the tradeoff directly: - **T = 1** — near one-hot; the tail is numerically present but contributes almost nothing. - **T = 2** — the top few classes separate from the rest; the confusable neighbours become visible. - **T = 5** — the similarity ranking is clearly expressed across a broad tail; typically the useful range. - **T = 20** — everything is drifting toward the uniform 0.01, the true class is barely distinguished, and the teacher's genuine confidence has been discarded along with the ranking. So T is not monotonically better. Too low and you transfer nothing beyond the label you already had; too high and you transfer near-uniform mush, amplifying whatever arbitrary noise sits in the teacher's smallest logits. It is a hyperparameter, swept on validation, and it interacts with class count — heads with many classes generally tolerate and need less flattening than a small head, where even a modest T pushes you close to uniform. ## Applied where, exactly Three details candidates routinely get wrong: 1. **Both networks, same T.** The teacher's targets and the student's predictions must be softened by the same temperature for the distillation term, or you are asking the student to match a distribution of a different shape. 2. **Only the distillation term.** The hard-label cross-entropy uses the student's ordinary T = 1 outputs against the true label. Softening that term as well would degrade the true-label signal for no reason. 3. **Inference uses T = 1.** Temperature is a training-time device. The deployed student computes its ordinary softmax, and since T does not change the ordering, argmax predictions would be identical anyway — but calibrated probabilities would not be, so leaving T > 1 in the serving path silently under-confidences every output. ## The T-squared rescaling With temperature-softened distributions, the gradient of the soft-target cross-entropy with respect to a student logit takes the form `(1 / T) * (q_i - p_i)`, where `q` is the softened student probability and `p` the softened teacher probability. There is a 1/T out front, and the probability difference itself shrinks as the distributions flatten — so the magnitude of the soft-target gradient falls roughly as 1/T-squared. The consequence is practical: change T and the effective weight of the soft term relative to the hard-label term changes with it, so a blend weight tuned at T = 2 is wrong at T = 5. The standard fix is to multiply the soft-target loss by T-squared, restoring comparable gradient magnitude and making the blend weight roughly temperature-independent. Skipping the rescale is a classic silent bug: raising T appears to stop helping, when in fact you have quietly turned the distillation term down. ## The high-temperature limit If T is made large relative to the logit magnitudes and the logits are zero-mean per example, the softened cross-entropy gradient reduces to a term proportional to the difference between student and teacher logits. In that regime distillation is essentially matching logits directly rather than probabilities. This is a useful thing to know because it clarifies what the middle range buys you: at moderate T the loss still weights the classes the teacher considers plausible more heavily than the ones it has rejected outright, whereas logit matching treats a very negative logit as just as worth reproducing as a competitive one. Very negative logits are typically noisy and constrained by little training signal, so forcing the student to reproduce them wastes capacity. ## Choosing T in practice Start with a small sweep — 2, 4, 8 — on a validation set, jointly with the blend weight, since the two interact. Larger students and larger class counts tend to tolerate higher T. If your teacher was trained with a technique that already produces a high-entropy output, less flattening is needed. And always confirm the T-squared factor is present before concluding a temperature does not help.