In distillation, how do you weight the teacher-KL term against hard-label cross-entropy when labels are noisy?

## The two terms and what each one is for The distillation objective is a blend: `L = alpha * T^2 * KL(teacher_softened || student_softened) + (1 - alpha) * CE(hard_label, student)`. The first term pulls the student toward the teacher's softened distribution; the second anchors it to the recorded ground truth. Alpha is the dial. The key structural observation for noisy data: **the hard-label term is the only place a wrong label enters the objective.** The KL term never sees the label at all — it sees the teacher's output on that input. So `1 - alpha` is, quite literally, your exposure to label noise. That reframing is what an interviewer is looking for; everything else follows from it. A useful default before noise even enters the picture is that distillation setups typically put the *majority* of the weight on the soft term, with the hard-label term contributing a minority share. Starting from there, noisy labels push you further in the same direction rather than into unfamiliar territory. ## The prior question: what did the teacher see? Raising alpha only helps if the teacher's targets are cleaner than the labels. Three cases, and you must know which you are in: 1. **Teacher trained on a cleaner or much larger corpus.** Best case. Where the recorded label is wrong, the teacher usually puts substantial mass on the true class, having generalised from many correctly-labelled similar examples. Raising alpha is a real denoiser, and alpha close to 1 can beat any blend that respects the labels. 2. **Teacher trained on the same corrupted set, but large and regularised, or stopped before convergence.** Networks fit the clean, consistent structure of a dataset before they memorise its inconsistent examples, so an early-stopped teacher often *generalises past* the noise: its prediction on a mislabelled example reflects what similar correctly-labelled examples say, not the wrong label. Raising alpha usually still helps, though less dramatically. 3. **Teacher trained on the same corrupted set to full convergence with enough capacity to memorise.** Its output on a mislabelled training example may be confidently the wrong class. Now the soft targets carry the same errors dressed up as confident predictions, and shifting weight toward the teacher launders noise rather than removing it. This is the trap. A cheap diagnostic separates case 3 from the others: take a small set of examples you *know* are mislabelled, and look at the teacher's distribution on them. Confident agreement with the wrong label is memorisation; mass on the plausible true class is generalisation. ## How to actually pick alpha - **Hand-clean a small holdout.** A few hundred to a couple of thousand carefully re-labelled examples is enough. This is the single highest-value step, because every downstream decision is measured against it. - **Never tune on a validation split drawn from the same noisy pipeline.** A noisy validation set scores a student higher for reproducing the noise, so it will push you toward exactly the wrong alpha — the failure is self-concealing, since your metrics look fine. - **Sweep coarsely**: alpha in roughly 0.3, 0.5, 0.7, 0.9, 1.0, jointly with the temperature, since the two interact — a high temperature with a high alpha spreads mass broadly and can dilute the true class as well as the wrong one. - **Keep the T-squared factor in place** while sweeping, or alpha and temperature will be confounded and the sweep will be uninterpretable. - **Consider alpha = 1 honestly.** Teacher-only training caps the student at the teacher's fidelity, so you forfeit any signal the labels carry that the teacher lacks — new classes, recently corrected annotations, distribution shift the teacher predates. If the labels are only a few percent wrong, that forfeit is often larger than the noise you avoided. ## Per-example refinement A global alpha treats every example as equally suspect, and they are not. Where the teacher confidently disagrees with the recorded label, that example is a strong candidate for being mislabelled. Two productive responses: downweight or drop the hard-label term for exactly those examples, keeping the full hard-label weight everywhere else; or route them to human review and fix the labels. The second is usually the better investment — teacher-student disagreement is one of the cheapest mislabelled-example detectors available, and unlike a loss-weighting trick it improves every model you train afterwards. The honest framing to offer an interviewer: tuning alpha is damage control, not a fix. If the noise is systematic — a whole annotator, a class pair the guidelines never disambiguated, a broken ingestion path — the right move is to repair the labelling process. Blend weights compensate for random noise reasonably well and for systematic noise badly, because a systematic error is likely present in the teacher too. ## What good looks like You should be able to say: which term carries the noise, what the teacher's training history implies about whether it can outvote the labels, that the blend is tuned against clean data only, and that teacher-student disagreement is a signal worth mining rather than a nuisance to smooth over.