In layer-wise discriminative fine-tuning, why does the bottom block get a much smaller rate than the head?

## The premise: layers differ in how much they must change A single fine-tuning rate assumes every depth needs the same amount of movement. Transfer learning makes that assumption obviously false. Representations in a pretrained network get more general the lower you go and more task-specific the higher you go. The earliest layers of a vision backbone compute edges, colours and simple textures; the earliest layers of a text encoder compute broad token and morphology statistics. Those things are true of the target task too, so the correct amount of change there is close to zero. The upper blocks compose those primitives into abstractions shaped by the *source* task, and those must be re-shaped. Above them sits a head that is random and has to be learned from scratch. Discriminative fine-tuning encodes that gradient of need directly into the step sizes: one rate per layer group, increasing with depth toward the output. ## The ladder and why it is geometric The usual arrangement spans roughly two orders of magnitude, something like 1e-5 on the bottom block rising to about 1e-3 at the head, with a constant *ratio* between adjacent groups rather than a constant difference. The reason to make it multiplicative is that what matters at each depth is the update size relative to that layer's own weight scale, not an absolute number of units in parameter space. A constant ratio means each group moves in proportion to the one below it, so the ladder is scale-free and keeps its shape whether you span four groups or twelve. The arithmetic follows from the span and the number of groups. Spreading 1e-5 to 1e-3 — a factor of 100 — across four groups means three gaps, so each adjacent pair differs by about 4.6x. Across more groups the per-step ratio shrinks: five groups gives four gaps and a ratio near 3.2. The published discriminative-fine-tuning recipe from ULMFiT used a fixed divisor of about 2.6 per group going downward, which lands in the same family. The precise number matters much less than the shape: strictly increasing toward the output, multiplicative, and anchored by a top rate you actually tuned. ## Why not just tune every layer separately Because the search cost explodes and the payoff does not. Parameterising the ladder as (top rate, ratio) gives you two knobs to tune and a monotone structure you can reason about; per-layer rates give you as many knobs as blocks, no structure, and enormous room to overfit the validation set with hyperparameters. The two-knob version also degrades gracefully: set the ratio to one and you are back to a single global rate, which is a sensible fallback rather than a broken configuration. ## Relationship to freezing Freezing a block is the same idea with the rate set to zero. That makes the two techniques points on one axis: freeze the bottom (hard zero), give it a tiny rate (soft, still nearly fixed), or give the whole network one rate (no discrimination at all). The soft version is often preferable when the target domain has drifted a little from the source, because even the generic layers benefit from a small correction that a hard freeze forbids. ## When the ladder is worth it, and when it is not The ladder pays off most when the target dataset is small, when the domain is somewhat distant, or when a single global rate has already shown you both failure modes — collapse at the high end of your sweep and no movement off the frozen baseline at the low end. That combination is exactly the sign that different depths want different step sizes and no single number satisfies both. It pays off least in three situations. First, when the dataset is large and the domain is close: everything can move at one modest rate and the extra structure buys little. Second, when the tuning budget is tight: two extra hyperparameters on a small validation set is a real risk of tuning noise. Third, when you are training with an adaptive optimizer, which already rescales updates per parameter and so partially flattens the differences the ladder is trying to create — the effect is real but smaller than it is with plain gradient descent. ## What to say when asked Good answers name the mechanism (rates increasing with depth), the reason (generality at the bottom, specificity and randomness at the top), the parameterisation (constant ratio, two hyperparameters), and the honest cost (more knobs, diminishing returns near domains, softened by adaptive optimizers). Weak answers describe the ladder as a universal best practice and cannot say what it would cost or when a single rate is fine.