Why can a much larger teacher distill worse into a small student than a mid-sized teacher does?

## The observation The intuitive expectation is monotone: a better teacher should produce a better student. Empirically it is not. Hold the student architecture fixed, distil it from a sequence of progressively larger and more accurate teachers, and student accuracy typically rises, peaks, and then **falls**, even as teacher accuracy keeps climbing. The distance between teacher and student capacity, not the teacher's own quality, is what governs transfer. This is usually called the **capacity gap**. ## Why it happens Three mechanisms, and a good answer distinguishes them. **1. Representational capacity.** The student's hypothesis space simply may not contain a function close to the large teacher's. Asking it to reproduce that function means asking for something it cannot express, and the best it can do is a compromise that may be worse than what it would have found by fitting the task more directly. **2. Optimisation, not just expressiveness.** The sharper and less obvious point: even when the student *could* in principle represent something close, training frequently fails to get there. Students distilled from very large teachers often cannot match those teachers on the **training set** - the objective is not being minimised, quite apart from generalisation. That reframes the gap as partly an optimisation failure, which is why remedies that make the target easier to chase (a smaller intermediate teacher, a less converged teacher) work at all. **3. Target sharpness.** A very large, thoroughly converged network is also a very confident one. Its per-example targets sit close to the extremes, which supplies less usable structure than a moderately confident teacher's targets and reduces the objective towards something close to plain label fitting - at which point you have paid for a teacher and received little. ## Remedies **Teacher assistant chain.** Insert an intermediate-size network: distil the assistant from the large teacher, then distil the small student from the assistant. Each hop spans a gap that the receiving model can actually close. The chain costs an extra training run and adds a size choice to make, and it stops paying once the individual hops are already small. **Use an earlier teacher checkpoint.** A teacher stopped before full convergence is less confident and less contorted, and often distils better than the same teacher at its own accuracy peak. This is cheap - the checkpoints already exist - and it is the first thing to try. **Down-weight the teacher term.** If the teacher's target is partly unusable, give it less influence relative to the task objective and let the student spend its capacity on the task. **Ensemble mid-sized teachers.** Averaging several moderate teachers gives a smoother, better-calibrated target without the extreme sharpness of one giant model. Training cost goes up, but only at training time. **Change what you ask for.** Instead of asking the student to reproduce the teacher's outputs, ask for a weaker correspondence - matching coarse internal summaries, or ranking rather than exact values. A weaker constraint is one a small student can satisfy. ## The selection rule All of this collapses into one operating principle: **choose the teacher by measured student accuracy, not by teacher accuracy.** Run short distillation trials with two or three candidate teachers of different sizes and compare the students on your own validation set. This costs far less than a full training cycle and routinely overturns the choice you would have made from the teachers' own numbers. Record the choice, because it is student-specific - re-shrinking the student later can flip which teacher is best. ## Diagnosing it in your own run The distinguishing measurement is **training-set agreement between student and teacher**. If the student is close to the teacher on training data but poor on validation data, you have a generalisation problem and the capacity gap is not your issue. If the student cannot match the teacher even on training data, you are in the capacity-gap regime, and the remedies above apply. A second useful check is whether the student distilled from a deliberately smaller teacher does better - a two-run experiment that either confirms the diagnosis or eliminates it. ## Common mistakes The most common is treating teacher accuracy as a proxy for transfer quality and reaching for the largest available model on principle. The second is concluding from a single failed distillation that distillation does not work for this student, when the actual finding is that *this teacher* does not work for this student. The third is inserting an assistant chain reflexively: it is a fix for a diagnosed large gap, and when the gap is already modest it just adds a training run and a hyperparameter for no gain.