Why does the L1 norm push coordinates to exactly zero while the L2 norm only shrinks them?

## The two norms, side by side For a coefficient vector `w`, the L1 norm is `|w_1| + |w_2| + ... + |w_n|` and the L2 norm is `sqrt(w_1^2 + ... + w_n^2)`. Both are legitimate measures of how large `w` is, and both are minimised at `w = 0`. Yet if you hold an objective fixed and constrain the size of `w`, L1 produces solutions with exact zeros in them and L2 essentially never does. The reason is geometry, and it is worth being able to draw it. ## The unit balls The set of vectors with `||w||_1 <= 1` in two dimensions is the square rotated 45 degrees, with vertices at (1, 0), (0, 1), (-1, 0) and (0, -1). It is usually called the diamond. Every one of its four vertices lies **on a coordinate axis**, which means the other coordinate is exactly zero there. The set with `||w||_2 <= 1` is the ordinary circle: perfectly smooth, no vertices, no distinguished directions. The set with `||w||_inf <= 1` is the axis-aligned square, whose corners sit at (1, 1) and its reflections, so its corners are the points where *no* coordinate is zero. ## The tangency argument Picture the level sets of a smooth objective as nested contours in the plane, centred on the unconstrained optimum. Shrink the size budget until the constraint bites, and the solution is the first point where an expanding contour touches the ball. On a smooth round ball, tangency happens at a generic point on the boundary, essentially never exactly on an axis. Both coordinates come back toward zero, but neither arrives. That is what shrinkage means. On the diamond, the corners are the points of highest curvature, in fact points where the boundary is not differentiable at all. A whole range of contour orientations touches at the same corner, because a corner presents a fan of supporting directions rather than a single tangent line. That range is what makes exact zeros not merely possible but common: a corner is a target with positive width, whereas a specific point on a circle is a target of measure zero. In `n` dimensions the picture generalises. The L1 ball is a cross-polytope with `2n` vertices on the axes, plus lower-dimensional faces on which some subsets of coordinates are pinned at zero while others vary. So the solution does not have to collapse to a single nonzero coordinate; it can land on a face where, say, three of twenty coordinates are free and seventeen are exactly zero. ## The gradient argument The same conclusion falls out of calculus. Consider the pull that the size term exerts on one coordinate `w_j` as it approaches zero. - For the absolute value `|w_j|`, the derivative is `+1` for any positive value and `-1` for any negative value, however small. The push toward zero has constant strength all the way in. At zero itself the function has a kink and the derivative is undefined; the set of subgradients there is the whole interval from -1 to +1. If the objective's own pull on that coordinate is weaker than the size term's constant pull, zero is a genuine stationary point and the coordinate stays pinned there. - For the square `w_j^2`, the derivative is `2w_j`, which shrinks in proportion to the coordinate itself. As `w_j` gets small the restoring force gets small with it, so the two pulls balance at some small nonzero value. Zero is reached only in the limit of an infinite size penalty. The non-differentiable kink is the whole mechanism. Any smooth approximation to `|w|` restores a vanishing derivative near zero and destroys the sparsity, which is why smoothed absolute values are a poor substitute when exact zeros are the goal. ## Practical consequences and caveats Exact zeros mean a solution you can read: coordinates that are gone are gone, not merely small, so the result doubles as a selection. L2 shrinkage keeps every coordinate present, which is often the more faithful description when the coordinates genuinely share the signal. Two caveats belong with the geometry. First, when two coordinates point in nearly the same direction, the diamond offers several nearly equally good corners, so which one is selected can flip with a small change in the data. The L2 ball, having no corners, splits the weight between them stably instead. Second, exact zeros are a property of the constrained or penalised solution, not of the norm on its own: computing `||w||_1` for an existing vector creates no zeros at all. ## Checking your understanding If someone claims sparsity comes from L1 being smaller than L2, correct it. For the vector (3, 4) the L1 norm is 7 and the L2 norm is 5, so L1 is the *larger* of the two here. Magnitude is not the point. The corners are.