A linear probe on frozen features scores near chance while a two-layer head succeeds - what does that prove?

## Why the result is not a contradiction Both runs consume the *same cached feature vectors*. Nothing about the encoder differed. The only thing that changed is the function class fitted on top: a hyperplane in one case, a hyperplane composed with a hidden non-linearity in the other. When the second works and the first does not, the deduction is forced: the information required to make the decision is contained in the vectors, and the linear read-out was the bottleneck. ## What linear inseparability looks like Three familiar geometries defeat a hyperplane while leaving the information fully present: - **Interaction (XOR-like) encoding.** The label depends on the *agreement* of two feature directions, not on either one. Neither direction alone correlates with the label, so every hyperplane is at chance, while one hidden layer of two units resolves it. - **Shell or ring structure.** One class surrounds another in feature space. Distance from a centre separates them perfectly; no linear boundary does. - **Conditional encoding.** The direction that carries the property flips sign depending on some other attribute of the input, so a single global weight vector averages the two cases into nothing. None of these is exotic. Networks have no incentive to make anything linearly readable unless the training objective's own read-out was linear. ## The bound, stated precisely Let *A_lin* be the accuracy of the best linear read-out and *A_f* the best accuracy achievable by any function of the same features. Then *A_lin <= A_f*, always. A probe measures the left-hand side. So: - A **high** probe score is strong evidence: the property is present *and* easy to reach, which is the useful claim for anyone planning to put a light head on frozen features. - A **low** probe score is weak evidence: it constrains the linear case only. Reporting it as "the encoder does not represent X" confuses absence of evidence with evidence of absence. ## The symmetric trap: capacity in the probe Once you allow the read-out a hidden layer, the natural next question is why not go further. The problem is that a sufficiently expressive read-out learns the task itself. Push far enough and you can extract almost any property from almost any representation, including a randomly initialised one - at which point the score describes the probe's training, not the encoder. Two controls keep a stronger probe honest: - **Random-encoder control.** Fit the identical head on a randomly initialised encoder of the same architecture. Only the margin over that control is attributable to what pretraining learned. - **Random-label control.** Fit the identical head on shuffled labels. If it fits them well, the head has the capacity to memorise, and its score on real labels is partly memorisation. The difference between real-label and random-label performance is the standard selectivity check in the probing literature. A disciplined probing report therefore fixes a read-out family, states it, and reports these controls alongside - rather than quietly upgrading the read-out until the result is the one expected. ## Decodability is not use A further limit worth stating out loud in an interview: even a high probe score only shows a property *can* be read out of a layer. It does not show the network's own downstream computation uses it. Probing is a claim about what is recoverable from an activation, not a causal claim about the mechanism. Establishing use requires an intervention - removing or perturbing the direction and observing whether behaviour changes - which is a different tool entirely. ## What to do with the finding If a linear probe fails and a small head succeeds, the practical readings are: 1. Do not delete the encoder or the layer from consideration; it carries the signal. 2. Match the probe to the deployment. If the plan is to serve a light head over frozen features, then a *linear* failure is a real, decision-relevant failure for that plan - it says the plan needs a non-linear head. If the plan is to fine-tune, linear decodability was never the requirement. 3. Report both numbers. "Linear 51%, two-layer head 74%, random-encoder two-layer head 55%" is a complete statement; either number alone is misleading. 4. Consider whether the linear failure is fixable by a different layer, since a property often becomes more linearly exposed a few blocks earlier or later in the stack.