Why judge layer pruning sensitivity after recovery retraining rather than right after the cut?

## The trap The cheap version of a sensitivity sweep zeroes a layer's weights, evaluates immediately, and ranks layers by the drop. It is cheap because nothing is retrained. It is also measuring the wrong quantity: nobody ships a network in the state it is in one second after a cut. The deployed model has been fine-tuned afterwards, so the number that matters is how much accuracy is missing *after recovery*, and the two orderings are not the same. ## What the instant drop actually contains The immediate degradation is a sum of two things. **Genuine information loss.** The removed weights were computing something the rest of the network cannot reconstruct. No amount of fine-tuning brings it back, because the capacity is gone. **Recoverable disturbance.** The cut changed the distribution of activations flowing through the network, and several things downstream are now mismatched in ways that a small amount of adaptation fixes: - Normalization layers carry running statistics — the mean and variance estimates collected during training and used at inference. After a cut, the activations they normalize have a different distribution, so the stored statistics are stale and every downstream layer receives badly scaled inputs. Recomputing those statistics by running a few hundred batches of unlabeled data forward, with no gradient steps and no labels, frequently recovers a large fraction of the drop on its own. Reporting a sensitivity number without doing this attributes a normalization bookkeeping problem to the pruning. - Remaining weights and biases are no longer centred for the new activation scale, which a short fine-tune corrects quickly. - The surviving units can often re-specialize to cover part of what was removed, provided they are allowed to train. Only the first component is sensitivity in the sense you care about. The naive protocol measures their sum and calls it sensitivity. ## Why the ranking flips A wide, redundant layer can produce an enormous instant drop — a large perturbation entering a deep stack — and yet recover completely, because the information was distributed and the surviving weights can carry it. A narrow bottleneck can look almost harmless at first and never fully recover, because what it lost was not represented anywhere else. Ranking on instant drop therefore protects the wrong layers: you spend budget defending a layer that would have healed and cut one that will not. ## A protocol that survives review 1. **Recalibrate first.** After each candidate cut, recompute normalization running statistics on a fixed set of unlabeled batches before measuring anything. This is nearly free and removes the largest source of misleading drop. 2. **Give every candidate the same short recovery budget.** A fixed small number of steps at a low learning rate, identical across layers and ratios. Equality is what makes the comparison a comparison; a layer given twice the fine-tuning will look twice as robust. 3. **Keep the proxy honest about what it is.** The short fine-tune is a ranking device, not the shipped recipe. If you let it grow long enough to be a training run in its own right, you are no longer measuring sensitivity — you are measuring whether the architecture can be retrained from a degraded start, which is a different and much less useful question. 4. **Confirm the chosen configuration with the full schedule.** Once the uneven allocation is picked, retrain properly: a fraction of the original schedule, low learning rate with a short warmup, the whole network trainable so the untouched layers can adapt around the cut ones. Then re-measure the layers you cut hardest, because the profile has moved. 5. **Watch the recovery curve, not just its endpoint.** If a configuration's accuracy is still climbing when the recovery budget ends, the comparison is budget-limited rather than capacity-limited and you have not learned what you think you have. ## The cost tension Full recovery retraining for every layer at every ratio is layers times ratios fine-tunes, which is prohibitive for anything real. That is precisely why the two-stage design exists: a cheap no-retrain sweep to eliminate the obviously safe and obviously fatal ratios, then short equal-budget recovery runs on the shortlist, then one full retrain of the final candidate. Interviewers like this question because it separates candidates who have run a compression project from candidates who have read about one — the instant-drop trap costs a real team a week before anyone notices the retrained rankings disagree. ## Recovery after an uneven cut One last practical point specific to uneven allocations: the layers that were cut hardest need the most adaptation, but you do not train them alone. Gradients through the untouched layers are what let the network route around the damage, so the standard recipe trains everything, at a reduced learning rate, for longer than the instinct suggests — heavily compressed regions often keep improving well past the point where accuracy looks converged on the lightly cut ones.