A linear probe hits 68% where full fine-tuning hits 79% - what does that gap tell you?

## What the two numbers are The 68% comes from a frozen encoder plus a fitted hyperplane: the features are fixed and only the read-out adapts. The 79% comes from letting every encoder parameter move on the target labels, which changes the features themselves. The difference is therefore *the return on being allowed to reshape the representation for this task* - not a generic property of the encoder, and not a property you can quote without naming the target. ## Four substantive causes **1. The pretraining objective threw the signal away.** Every objective is a compression rule about what to keep. If the target depends on a distinction the pretraining task never had to make - fine-grained texture, a rare category, an attribute deliberately made invariant by the augmentations used in pretraining - the frozen features simply do not carry it, and no read-out recovers it. Fine-tuning can rebuild it from the lower layers. **2. The signal is present but not linearly decodable.** The classes may sit in a curved or interleaved arrangement that no hyperplane separates. Here the information is there; only the read-out is too weak. A small non-linear head on the same frozen features will close part of the gap and is the cheapest test of this hypothesis. **3. Domain shift.** Frozen features are only meaningful on inputs that resemble what the encoder was trained on. Move to a different sensor, a different language register or a heavily different image statistic, and the frozen vectors become poorly spread and near-degenerate for the new inputs. Fine-tuning adapts the early layers to the new input statistics, which is exactly where a large gap most often comes from. **4. Wrong layer.** You probed one layer. If that layer has specialised toward the pretraining objective, the property may live several blocks earlier. Probing a range of depths often recovers points without touching the encoder. ## Artefacts to rule out first A gap can be manufactured rather than real: - **An under-fitted probe.** Probes need enough epochs and a sensible regularisation strength. A probe stopped early or regularised hard is a weak measurement, and the missing points belong to the probe, not the features. - **Unequal preprocessing.** If the fine-tuning run saw augmentation and the cached probe features did not, part of the gap is a data-pipeline difference. - **Noise on a small target set.** With a few thousand validation examples, eleven points may be two or three standard deviations of seed-to-seed variation - or may not be. Repeat both sides across seeds before treating the gap as a finding. - **Fine-tuning overfitting a favourable split.** A fine-tuned model has far more capacity to exploit leakage or near-duplicates between train and validation; a frozen encoder with a linear head largely cannot. Check the split before crediting the extra points to representation reshaping. ## How to localise the cause A workable diagnostic ladder, cheapest first: 1. Refit the probe properly with a real hyperparameter budget. If the gap shrinks, it was measurement. 2. Probe several layers. If a middle layer beats the one you used, the property was there and you were reading the wrong depth. 3. Fit a two-layer head on the same frozen features. If it closes much of the gap, the information is present but non-linear; if it barely moves, the information is genuinely missing. 4. Compare probe accuracy on target data with probe accuracy on data resembling the pretraining distribution. A large drop points at domain shift rather than task difficulty. 5. Look at *which* classes lose. A gap concentrated in a few confusable classes reads very differently from one spread evenly. ## Reading the size of the gap A **near-zero gap** says the frozen representation already exposes everything the target needs along linear directions; extra capacity buys nothing, and any remaining error is task noise or label noise rather than representation quality. A **modest gap** is the normal case for a target close to the pretraining domain. A **very large gap** - the probe near the majority-class rate while fine-tuning works well - usually means the encoder is being used far outside the distribution it was trained on, and the fine-tune is effectively relearning features rather than adapting them. One caution: the gap ranks nothing beyond this target. Two encoders with identical probe scores can have very different fine-tuning ceilings, and a probe-versus-fine-tune gap measured on one dataset does not transfer to the next one.