In born-again self-distillation the student copies the teacher's architecture exactly, so why does it improve?

## The setup **Self-distillation** is distillation where the student is not smaller than the teacher. The cleanest version is the *born-again* procedure: train a network normally; freeze it; then train a **second network with the identical architecture**, from a fresh random initialisation, using the first network's outputs as its training target. The result that makes this interesting is that the second network frequently ends up *more accurate on held-out data than the network that taught it*. Repeat, using generation two as the teacher for generation three, and accuracy often improves again for a generation or two before flattening. Averaging the predictions of several generations helps further, though that is an ensemble at inference cost. ## Why the result matters The usual story told about distillation is a compression story: a small student cannot learn the task from labels alone, so it learns from a big teacher that can. Born-again training breaks that story cleanly, because **the student has exactly the same capacity as the teacher**. Whatever is happening, it is not the student being rescued from a capacity shortfall. This is the single most useful thing to say about the result in an interview: distillation is a way of *changing the training signal*, and compression is one application of it rather than its definition. ## What actually changes The only thing that changed is the target each example is trained against. 1. **The target is no longer a hard one-hot vector.** A trained network almost never outputs a degenerate distribution, so the student is fit to a smoother objective. Smoother targets reduce the pressure to drive the correct output to an extreme and generally reduce overfitting to individual examples. 2. **The target is input-dependent.** This distinguishes it from a fixed smoothing scheme applied identically to every example. The teacher's confidence varies from example to example, so easy, unambiguous examples and hard, ambiguous ones are effectively weighted differently in the loss. An example the teacher finds obvious contributes a sharper target and a stronger pull; an example it finds genuinely ambiguous contributes a softer one. This per-example reweighting is a real, measurable difference from any input-independent regulariser. 3. **Optimisation is easier.** The gradient signal from a smooth, self-consistent target has lower variance than the signal from one-hot labels, particularly late in training when a fit to hard labels keeps pushing outputs further apart with little accuracy left to gain. Note what is *not* in this list: the student is not inheriting a better architecture, more parameters, more data, or more supervision than the teacher had. It sees the same inputs; only the labels it is asked to reproduce changed. ## The fidelity result A sharp piece of evidence sits alongside this. If you measure how often a distilled student's *prediction* agrees with its teacher's prediction - fidelity, as distinct from accuracy - the agreement is often surprisingly low, and it stays low even when the student's own test accuracy has improved. Equal-capacity students frequently fail to match their teacher's function even on the training set. Two conclusions follow. First, distillation in practice is not solved function-matching; it is an optimisation problem that is only partly solved, and the residual is not simply a capacity shortfall. Second, accuracy gains and fidelity gains are different objectives - if you actually need the student to *behave like* the teacher (say, because downstream systems were tuned against the teacher's mistakes), you must measure agreement directly, because accuracy will not tell you. ## Related variants The same idea appears in other shapes. One is a within-network form, where deeper parts of a single network supply targets to shallower exits during a single training run, so no second run is needed. Another is generational training on unlabelled or weakly labelled data, where each generation labels more data for the next. All share the property that the teacher is not larger than the student, so none of them can be explained by compression. ## Costs and honest caveats - **Each generation is a full training run.** The accuracy gain is real but modest; on a fixed compute budget you should ask whether another generation beats spending the same compute on data, augmentation, or a longer single run. - **Gains saturate.** Beyond two or three generations, improvements typically stop, and the sequence can drift if the teacher's errors are systematic - each generation inherits and can amplify the previous one's biases, especially on rare classes. - **Attribution requires a control.** Because the mechanism is partly generic regularisation, a claimed born-again gain should be compared against the same architecture trained with a comparable simple regulariser and a matched compute budget before it is credited to the teacher specifically. ## The interview answer State the setup precisely (same architecture, fresh initialisation, trained on the teacher's outputs), state that identical capacity rules out compression as the explanation, and name the mechanism as input-dependent target smoothing that regularises and reweights. Then add the fidelity nuance - the student does not actually reproduce the teacher's function - and the cost caveat. That combination shows you have read past the headline.