Why would a classifier get worse after PCA retained 95% of the input variance?

## The core reason: PCA is unsupervised PCA is fitted on the input matrix alone. It searches for directions of maximum variance in the mean-centred inputs and orders components by that variance. The target never enters the computation. So the ordering answers the question "along which direction do these rows differ from one another the most?" and *not* "along which direction do the two classes differ the most?" Those can be completely different directions. A concrete way to see it: imagine one input that is a loud, high-amplitude measurement whose value is unrelated to the label, and another that varies over a narrow range but shifts consistently between the two classes. PCA puts the loud one in the leading component and may push the narrow, discriminative one into the tail. If your rule was "keep 95% of the variance", the useful direction is exactly what got cut. Nothing has gone wrong with PCA - it did what it optimises. The mistake is treating a variance ranking as a signal ranking. This also explains why "we kept 95% of the variance" is a misleading sentence in a report. Ninety-five percent of the *inputs'* spread is retained. How much of the *label-relevant* structure is retained is unmeasured, and could be 100% or could be nearly none. ## Second cause: components are dense mixtures Each component is a weighted blend of every original column. That matters differently by model family. For a linear model with strong collinearity among inputs, replacing raw columns with orthogonal components often *helps*: coefficients become stable, the problem is better conditioned, and the model has fewer parameters to fit. For tree-based models the trade usually runs the other way. A tree splits on one feature at a threshold. When a single raw column carried a clean decision boundary - a ratio above a level, a count below one - the tree found it in one split. After the rotation, that same boundary is a slanted plane in the component space, and the tree has to approximate it with a staircase of axis-aligned splits, costing depth, data and accuracy. Distance-based models sit in between: they usually benefit when the raw columns are numerous and correlated, and lose when the discarded directions mattered. ## Third cause: the count was never tuned A fixed threshold picks the number of components from a property of the inputs. If nobody cross-validated that count against the downstream metric, the pipeline is running on an untuned hyperparameter. It is common for a sweep to show that a noticeably larger count - or none at all - wins. ## Fourth cause: the structure is not linear PCA finds flat subspaces. If the class boundary curves through the input space, no linear projection is guaranteed to preserve it, and a low-variance direction may be exactly what a curved boundary needs. ## How to diagnose it 1. **Establish the baseline honestly.** Score the same model on the raw features, with the same folds. If PCA is a regression relative to that baseline, you have a real finding, not noise. 2. **Interrogate the discarded components.** Fit with all components retained and look at how the model weights each one, or measure each individual component's association with the target on the training split. If a component that the 95% rule discarded ranks high, you have located the cause directly. 3. **Sweep the count.** Cross-validate over a range of component counts against the metric you actually optimise. A curve that keeps rising as you add components tells you variance-based truncation is not the right filter for this problem. 4. **Check the model family.** If it is a tree ensemble, try the raw features; the rotation is fighting how the model represents decisions. 5. **Check the leading component.** If component one is dominated by a single loud, uninformative input, the variance ranking is being driven by an artefact. ## What to do about it If discarded directions carry signal, the fix is to stop letting an unsupervised variance criterion do your feature reduction: select or reduce features using their relationship with the target instead, or keep more components, or keep the raw features and address collinearity another way. If the loss is small and PCA buys something real - inference latency, memory, a much better-conditioned model, stability under collinearity - it can still be the right call; just make that trade explicitly, on the metric that matters, rather than on the sentence "we retained 95% of the variance". ## The interview answer in one line "Because variance is a property of `X` alone. PCA ranks directions by how much the inputs move, and the direction that separates the classes may barely move at all - so a 95% variance cut can throw away exactly the signal the classifier needed."