How do you decide how many pretrained blocks to reuse versus retrain for a new target task?

## The decision, stated precisely A pretrained network is a stack of blocks. You are choosing two things, and it helps to separate them: - **How much to transfer**: which prefix of the stack, blocks 1 to k, is initialised from the source model rather than randomly. - **How much to freeze**: of that prefix, which blocks are held fixed during target training and which are allowed to keep learning. Agents and candidates often collapse these into one knob. They are different: you can transfer the entire backbone and still fine-tune all of it, or transfer everything and freeze everything but the head. ## The sweep The honest way to answer is empirical. For k spanning the depth of the network, build a model that keeps the first k blocks from the source model, reinitialises the rest plus a target-sized head, trains on the target set, and reports target validation performance. Run each k in two variants — transferred blocks frozen, and transferred blocks fine-tuned — and plot both curves against k. It is a modest sweep on a small target set and it replaces an argument with a graph. ## What the curves show, and why Two mechanisms act at once, and reading the sweep means separating them. **Specificity rises with depth.** The first blocks encode edges, colours and textures that any image task needs; the last blocks encode configurations that separate the source's own categories. So the deeper you transfer, the more of what you brought along is about a task you do not care about. This effect penalises large k, and it penalises it harder the further the target domain sits from the source domain. **Co-adaptation is fragile.** Within a trained network, adjacent layers are jointly optimised: a layer's units are tuned to the specific, sometimes idiosyncratic, features the layer below produces. If you freeze blocks 1 to k and randomly reinitialise everything above, the newly trained upper half has to relearn how to talk to a fixed lower half, and gradient descent from a random start often cannot recover the interaction that existed before. The signature of this is a dip in the frozen-transfer curve at *middle* values of k — layers deep enough to be entangled with what is above them, but not yet so specific that specificity explains the drop. Transferring the whole co-adapted group, or fine-tuning the transferred blocks so they can re-adapt, largely removes the dip. Because the two effects have different shapes — one monotonic in depth, one worst in the middle — the frozen curve can be non-monotonic while the fine-tuned curve is much flatter and usually higher. ## Priors to bring before you measure - **Target dataset size.** With few target labels, retraining deep layers overfits: you cannot re-estimate millions of parameters from a small set. Transfer more, freeze more, fit little. With a large target set, freezing wastes capacity and fine-tuning deeper wins. - **Domain distance.** Retail-shelf photographs sit close to a natural-image source corpus, so deep features stay useful. Spectrograms, overhead imagery or medical scans diverge earlier in the stack, so you keep the low-level filters and retrain more of the top. - **Label-space relatedness.** A source whose categories demand the same discriminations as your target — the same kinds of texture, part or motion distinctions — extends the useful depth. - **Compute and serving cost.** Freezing a prefix means you can precompute those activations once for the whole target set and train only the top, which is dramatically cheaper and worth real accuracy in some settings. ## Operational details people get wrong - **Normalisation layers are not fully frozen by freezing weights.** Layers that maintain running input statistics keep updating those statistics from target batches even when their learnable parameters are fixed. If you intend a block to be genuinely fixed, you must put it in inference mode as well, otherwise its behaviour drifts as target data flows through it, and your 'frozen' baseline is not the baseline you think it is. - **Staged unfreezing.** A common recipe is to train the head with everything frozen, then progressively unfreeze from the top down at a reduced learning rate. This is a cheap approximation of the sweep and reduces the risk of early gradients damaging the transferred features. - **Per-depth learning rates.** Rather than a binary frozen/trainable split, you can assign smaller learning rates to lower blocks and larger ones nearer the head, which makes the split continuous instead of a hard cut. - **Compare fairly.** Each k must get its own tuned learning rate and its own early-stopping decision; a sweep where every configuration shares one learning rate mostly measures which k happened to suit that learning rate. ## How to answer it in the room Say that it is a hyperparameter, describe the sweep in one sentence, then name the two competing effects — depth-increasing specificity and split-induced co-adaptation loss — and state that fine-tuning rather than freezing repairs the second. Finish with the two priors that decide the starting guess: how much target data you have and how far your domain is from the source's. That sequence demonstrates you have actually run this rather than repeated a recipe.