Feature Distillation and Limits

## What feature distillation adds Standard distillation puts a loss only on the network's output: the student is trained to reproduce the teacher's predicted distribution. That is a single supervision point at the very top of a network that may be dozens of layers deep. **Feature distillation** (also called hint-based training, after the FitNets work) adds one or more losses on *intermediate* activations, so the student's internal representation is supervised too. Terminology: the teacher layer you read from is the **hint layer**; the student layer you attach the loss to is the **guided layer**. The loss is typically a squared error between the two activation tensors, added to the usual objective with a weight. ## The shape problem and the projection A convolutional feature map has shape channels x height x width. A teacher is usually **wider** - more channels per stage - so the student's guided activations and the teacher's hint activations do not live in the same space and cannot be subtracted. The standard fix is a **learned projection**: a `1x1` convolution, which is a per-position linear map across channels, lifting the student's `C_s` channels to the teacher's `C_t`. It is trained jointly with the student and thrown away after training, so it costs nothing at deployment. Spatial mismatch is not fixed by resizing but by *choosing which layers to pair*. ## Pairing layers when depths differ With a 24-layer teacher and a 6-layer student, index-to-index pairing is meaningless - the student's third layer is not doing the teacher's third layer's job. Pair at **stage boundaries**. Most convolutional backbones are organised into a handful of stages separated by a stride-2 downsample, and the end of each stage is a natural, resolution-matched correspondence point. That gives three or four pairs, one per stage, no matter how many layers each side spends inside a stage. The same logic applies to a deep stack of identical blocks: pair the points where the representation changes shape or role, not the points where the indices happen to line up. ## Attention transfer: sidestepping channels entirely Attention transfer collapses the channel dimension **before** comparing. From a `C x H x W` activation you build an `H x W` spatial map by summing the squared activations across channels, normalise it, and match teacher and student maps with a squared error. Because the channel axis is gone, no projection is needed and the widths may differ freely; only the spatial size has to agree, which stage pairing already guarantees. The constraint is deliberately weaker than full feature matching - it says *attend to the same regions*, not *compute the same features* - and in practice it is often more robust for exactly that reason. A related family matches channel-correlation statistics instead, which is likewise dimension-agnostic. ## Scheduling The original formulation is two-stage: first train the student's layers up to the guided layer against the hint loss alone, which initialises the lower half of the student; then train the whole student with the output distillation loss. Common practice now is single-stage, with the feature loss added as a weighted auxiliary term, sometimes warmed up and then decayed so it shapes early training and later releases the student. ## When it earns its place Feature losses help most when the output signal alone is thin or hard to exploit: - very deep, very thin students that are hard to optimise from the top alone; - small labelled datasets, where extra supervision per example matters; - a large teacher-student gap; - tasks where the output is low-dimensional relative to the work the backbone does - dense prediction, detection, regression. When the student is close in capacity and the task is ordinary classification, the output loss usually captures most of the available gain and the feature term buys little. ## How it fails The characteristic failure is **over-constraining**. The teacher's activations are one particular solution among many; a narrower student may have no arrangement of its channels that reproduces them, and forcing it to try spends capacity on imitating internals instead of on being accurate. Symptoms: the feature loss plateaus high, or it falls nicely while validation accuracy does not move. The projection adds a second hazard. A sufficiently expressive adapter can absorb the mismatch by itself, driving the loss down without changing the student's representation at all - which is why a linear `1x1` map is preferred over a deeper adapter. And the loss weight is a real hyperparameter: too high and the student chases the teacher's internals at the expense of the task loss. ## Cost Every training step needs a teacher forward pass and enough memory to hold the teacher's intermediate activations at each matched stage, not just its outputs. That is a training-time cost only, but on large inputs it can dominate the budget. ## What to say in an interview Name the mechanism (auxiliary loss on intermediate activations), name the projection and why it is discarded, say how you pair layers across mismatched depths, and name the attention-map alternative and why it dodges the width problem. Then volunteer the ablation: feature distillation is a hypothesis about your student, and you test it by removing the term and seeing whether the student actually gets worse.