When retargeting a 1000-class pretrained backbone to 12 classes, what do you do with its head?

## What the head actually is At the top of a supervised classifier sits a linear layer: it takes the penultimate feature vector — call it a d-dimensional description of the input — and produces one score per source class. Its weight matrix has one row per source category, so with 1000 source categories the matrix is 1000 by d, plus a 1000-entry bias. A softmax turns those scores into probabilities and cross-entropy trains them. That matrix is indexed by the *source* label space. Row 417 means whatever source category 417 was. There is no sense in which row 417 corresponds to one of your 12 retail-shelf classes, and there is no reordering or subsetting that makes it correspond. That is why the entire layer is removed rather than trimmed. ## The surgery 1. Cut the network at the penultimate representation — the output of the last pooling or the last hidden block, whichever your architecture exposes as the feature vector. 2. Attach a new linear layer with 12 outputs and d inputs, initialised randomly (a small-variance initialisation appropriate to the layer's fan-in) with zero or small bias. 3. Keep everything below unchanged, and decide separately how much of it you will freeze or retrain. Nothing constrains the new head to be smaller than the old one. A target with 1200 classes is just as valid as one with 12; the head's width is a free choice and what limits you is whether the d-dimensional feature vector carries enough information to separate the target classes, not the arithmetic of 12 versus 1000. ## Why the random head is a hazard At initialisation the new head predicts roughly a uniform distribution over 12 classes, so the loss is near log 12 and the gradients flowing out of it are large. Those gradients propagate backwards into a backbone whose weights are, by assumption, already good. A few large steps early in training can undo a substantial part of what pretraining bought — the effect is sometimes described as the random head washing out the pretrained features. The standard mitigations follow directly: - **Warm up the head.** Train only the new layer for a short phase with the backbone held fixed, so the head becomes sensible before any gradient is allowed to reshape the features. - **Use a smaller learning rate below.** Give the backbone a fraction of the head's learning rate, so early noise perturbs it only slightly. - **Warm up the learning rate itself.** Ramping the step size from near zero over the first fraction of training limits the damage from the initial large loss. ## What is discarded and what is not A candidate who says 'we throw away everything the source labels taught us' has the accounting wrong. Two different things are being discarded: - **The source label space**: the identity and semantics of the 1000 categories, the class prior over them, and the linear boundaries between them. Gone, and irrecoverable from the truncated network. - **What learning those labels produced**: the representation. Fully kept. The reason the penultimate vector is useful is precisely that separating 1000 diverse categories forced the network to encode textures, parts and configurations at a level of detail no small target set could have taught it. ## The alternative worth mentioning Instead of cutting the head off, you can keep the full source model as a fixed black box and feed its 1000 output scores into a target model as an input vector. This is a legitimate technique — the source logits form a compact, semantically rich descriptor, and on a very small target set they can be competitive. But it is usually the weaker option: a 1000-dimensional vector of source-category decisions is a lossy summary that has already collapsed everything the source model knew into source-taxonomy terms, whereas the penultimate feature vector is typically higher-dimensional and not yet committed to any label space. It also permanently ties the target model to the source taxonomy. Knowing both options and why the default is head replacement is the level of nuance an interviewer is listening for. ## Related surgery you should expect to be asked about The input side needs the same care: the target images must be preprocessed the way the source model was trained to expect — same resizing convention, same channel ordering, same normalisation statistics — or the pretrained filters see a distribution they were never trained on. Head replacement gets all the attention, but a silently mismatched input pipeline produces the same symptom, a backbone that appears not to transfer.