Why train a fine-tune's randomly initialised head with the pretrained backbone frozen first?

## The problem: an untrained head attached to a trained body Transfer starts by replacing the pretrained output layer with a new one sized for the target label set, and that new layer is randomly initialised. The backbone below it is the opposite: every weight carries information earned over a long pretraining run. The first fine-tuning steps therefore mix two components in very different states, and the untrained one drives the gradient. Concretely, with softmax and cross-entropy the gradient at the logits is `dL/dz = p - y`, where `p` is the predicted distribution and `y` the one-hot label. A random head produces `p` far from `y`, so this quantity is large. Backpropagation carries it into the backbone through the head's random weight matrix, which means the direction the backbone is pushed in is determined by numbers that encode nothing. The result is a large update in an essentially arbitrary direction applied to the most valuable part of the model — the exact mechanism behind features being wrecked in the first few hundred steps. ## The fix: warm the head, not the step size Head warmup makes the risk structurally impossible for a while. Freeze every pretrained parameter, train only the new head, and let it run for a short spell — commonly one to two epochs on a modest target set. With the features fixed, the head's problem is small and well behaved: it is fitting a shallow classifier on top of static representations, so it converges quickly. Once its loss flattens, unfreeze the backbone and continue at a small backbone rate. The gradients now entering the backbone come from a head that already predicts sensibly, so they encode real task signal instead of noise. Two details matter in practice. First, the head can afford a much larger rate than the backbone will later use, precisely because nothing valuable is downstream of it. Second, the signal to unfreeze is the head's own loss curve levelling off — not a fixed step count copied from someone else's recipe. On a very small target set, watch that the head does not start overfitting during a long warmup; if it does, warm up for less time. Do not confuse this with a learning-rate warmup schedule, which ramps the step size up from near zero over the first updates. Those are different mechanisms answering overlapping concerns: one changes *which parameters* are trainable, the other changes *how big the steps are*. ## Extending warmup into gradual unfreezing Head warmup is the first move of a longer protocol. Gradual unfreezing continues it: after the head is warm, release the topmost pretrained block for one epoch, then the next block down for the following epoch, and so on until the whole network is trainable. The ordering follows what the layers hold. Top blocks encode the most task-specific abstractions and are the ones that most need to change; bottom blocks encode generic structure that usually transfers as is. Releasing top-down means the layers that must move are trained longest, and at any point in the schedule only a small part of the network can drift, so a bad batch damages less. A worked contrast: a four-class legal-document topic classifier built on a pretrained text encoder. Unfreezing everything on epoch one puts the entire encoder under gradients driven by a head that has seen the target labels once, and the run is fragile — its first-epoch validation number is often below what a frozen encoder gives. Warming the head for two epochs and then releasing one block per epoch keeps every intermediate checkpoint at or above that reference and reaches a higher final score on the same budget. ## The cost, and when to skip it Gradual unfreezing is not free. Each staged epoch is an epoch not spent training the full network, so on a fixed budget the schedule trades adaptation time for safety, and it adds a decision — how many blocks, how many epochs each — to your tuning surface. When the target domain is close to the source, the dataset is reasonably large, and the backbone rate is already small, unfreezing everything after a short head warmup usually performs just as well and is simpler to run. The staged version earns its cost when the target set is small, the domain is further away, or the run has already shown you a first-epoch collapse. What almost never pays is skipping the head warmup itself. It costs one or two cheap epochs — cheap because the frozen backbone means no gradients through most of the network — and it removes the single most common way a fine-tune destroys the thing it started from.