Why can a linear classifier not separate an XOR pattern over two binary features?

## Linear separability Two classes are **linearly separable** when some hyperplane puts every positive example strictly on one side and every negative example strictly on the other. In two dimensions that means one straight line with all of one colour above it and all of the other below. Separability is a property of the *data arrangement*, not of any particular fitted model: if it fails, no training run, no learning rate, no extra epochs and no extra rows of the same pattern will fix it. ## The XOR arrangement The canonical counterexample: a machine with two binary settings, A and B, where the part passes quality control only when **exactly one** of the two is switched on. Four cells, one per combination: ``` (A=0, B=0) -> fail (A=0, B=1) -> pass (A=1, B=0) -> pass (A=1, B=1) -> fail ``` Plot them at the corners of a unit square and colour them. The two passes sit on one diagonal, the two fails on the other. The diagonals cross in the middle of the square. A straight line divides the plane into two half-planes; to be correct it would need a half-plane containing both passes and neither fail — but any half-plane containing two points on one diagonal necessarily contains the crossing point of the diagonals, and shrinking it to avoid one fail always drops a pass. ## The same argument algebraically The geometric hand-wave becomes a two-line proof. A linear rule predicts pass when `s = w1*A + w2*B + b > 0`. Impose the four labels: - (0,0) fail: `b < 0` - (0,1) pass: `w2 + b > 0` - (1,0) pass: `w1 + b > 0` - (1,1) fail: `w1 + w2 + b < 0` Add the two pass constraints: `w1 + w2 + 2b > 0`, hence `w1 + w2 + b > -b`. Since `b < 0`, `-b > 0`, so `w1 + w2 + b > 0`. That directly contradicts the fourth constraint. No weights and no intercept satisfy all four at once. The best any linear model can do here is three cells right out of four, which on balanced data is 75% accuracy at most and 50% if the four cells are equally frequent and it settles on a degenerate cut. ## The continuous cousin XOR is the binary version of a general shape problem. Vibration data from a rotating machine often has the healthy regime forming a **ring** around a fault regime near the centre: healthy readings are moderate on both axes, faults are either very quiet or very loud on both. No straight line carves an annulus away from its own centre — a line separates two half-planes, and the inner region is surrounded on all sides. The same diagnosis applies: the model class cannot express the region shape. ## How you notice it without a picture With more than two features you cannot just look. Practical signals: - A linear model plateaus near the base rate while a flexible baseline on the *same* features does much better. That gap is the shape gap. - Errors are not scattered — they clump in particular regions of feature space. Group the residuals by feature buckets and look for whole cells that are systematically wrong in one direction. - The fit is unstable: small changes in the sample swing the boundary a lot, because no orientation is much better than any other. ## The fix that keeps the model linear You do not have to abandon linear models. Hand the model an extra feature and the boundary bends in the original coordinates while staying flat in the enlarged ones. For XOR, add the product `A*B`. Then ``` s = A + B - 2*(A*B) - 0.5 ``` gives `-0.5` at (0,0), `+0.5` at (0,1) and (1,0), and `-1.5` at (1,1) — all four correct. Geometrically you have lifted the four corners into three dimensions where a plane does separate them; projected back down, the boundary is a curve. The general recipe is the same: interactions, powers, buckets, and other basis expansions buy you shape at the cost of more parameters, more variance and features that need to be justified. ## The other end of the scale Separability is not always the enemy — sometimes you have too much of it. Add enough features and points in general position become separable almost automatically (any `n` points in general position are separable once the dimension reaches `n - 1`). When training data is perfectly separable, the maximum-likelihood weights of a logistic regression diverge: pushing all the weights up scales every score away from zero, drives every probability to 0 or 1 and drives the loss towards zero without ever reaching a finite optimum. The fit does not settle until a penalty on the weights pins it down. So "is it separable?" is worth asking in both directions: too little separability means the wrong model class, too much means the fit is not identified.