A hard-margin SVM separates 200 rows of 5,000-feature data perfectly. Why is that unsurprising?

## The claim that needs deflating "The classifier separates the training data perfectly" sounds like evidence, and in a low-dimensional problem it is. In a wide problem - many more features than rows - it is nearly automatic, and treating it as evidence is one of the classic ways to fool yourself with a margin classifier. ## Why separability is cheap when features outnumber rows Take `n` points in general position in a `d`-dimensional feature space. "General position" means no degenerate coincidences - no more of them lying on a common flat than necessary, which is the typical case for real-valued measurements. If `d >= n - 1`, those points are affinely independent, and an affine hyperplane exists that realises *every possible* assignment of the two labels among them. Not a particular convenient labelling: all of them, including labels generated by coin flips with no relationship to the features whatsoever. With 200 rows and 5,000 features you are far above that threshold. A hard-margin fit that separates the data is doing what the geometry guarantees it can do. The information content of "it separated" is close to zero. This is not an argument that the model must be wrong. Wide problems with real signal exist everywhere - gene expression, text, sensor arrays. It is an argument that separation alone cannot distinguish the two cases. ## What does carry information The margin's **width**, read relative to the scale of the data. Generalisation arguments for large-margin separators are usually stated in terms of the ratio `R^2 / margin^2`, where `R` bounds the radius of the region containing the data. The number of features does not appear explicitly. That is precisely why margin classifiers can be sane in high dimensions at all - but the guarantee is contingent on the margin staying wide relative to `R`, and in a labelling that is separable only because `d` is large, the achievable margin is typically razor-thin. So the question to ask is not "did it separate?" but "how much room did it have?", and "how much room would it have had on data with no signal at all?" ## The permutation baseline The second question has a direct empirical answer. Shuffle the labels so any relationship to the features is destroyed, refit the hard-margin problem, and record the resulting margin. Repeat many times to build the distribution of margins achievable on pure noise with these exact feature dimensions and this exact sample size. Then compare: - If the real-label margin sits comfortably above that whole distribution, the separation reflects structure the features actually carry. - If it sits inside the distribution, you have measured the geometry of your feature space, not a signal. The baseline is valuable because it is calibrated to your `n` and `d` rather than to an asymptotic rule of thumb, and because it is easy to explain to a stakeholder who has just been shown a perfect training separation. ## What to do about it The honest reading of a perfect separation in a wide problem is: unmeasured. Estimate performance on data the model never saw, with the split made before any feature selection - selecting features using all the labels and then splitting is a leak that manufactures separation on the held-out set too. Expect the held-out estimate to be noisy with 200 rows, and report an interval rather than a point. If the effective dimension can be reduced on principled grounds before fitting, do it, but not by peeking at the outcome. Also record the support-vector fraction. A hard-margin fit on wide data frequently makes a large share of the rows support vectors, which says the frontier is crowded and the boundary is held in place by most of the data - the opposite of the compact, well-separated regime the margin argument is comfortable in. ## The general lesson Any statement of the form "the model fits perfectly" is only informative relative to how easy fitting perfectly was. In wide problems it is easy by construction, so the burden shifts entirely onto out-of-sample evidence and onto quantities - margin width, permutation baselines - that are not automatically satisfiable.