What penalty does elastic net add to a linear model, and why mix L1 with L2?

## What an elastic net actually minimises An unpenalised linear fit picks coefficients `w` to minimise the training loss alone — for squared-error regression, `loss = sum over rows of (y - w.x)^2`. When predictors are many or strongly related to each other, that objective has too much freedom: coefficients grow large, absorb noise, and swing dramatically if you refit on a slightly different sample. A penalty puts a price on large coefficients: ``` objective = loss + lambda * penalty(w) ``` Two classic penalties sit at either end: - **L2 (ridge)**: `penalty = sum w_j^2`. Every coefficient is pulled toward zero roughly in proportion to its size. Nothing ever lands exactly on zero, so no predictor is removed. - **L1 (lasso)**: `penalty = sum |w_j|`. Coefficients are shrunk and, once the penalty outweighs a predictor's contribution to the loss, pinned at exactly zero — so the fit performs selection. Elastic net applies **both at once**, inside a single convex objective: ``` penalty = alpha * sum |w_j| + (1 - alpha) * sum w_j^2 / 2 objective = loss + lambda * penalty ``` `alpha` in [0, 1] is the **mixing ratio** — the share of the penalty that is L1. `alpha = 1` is a pure lasso, `alpha = 0` is a pure ridge, and anything between is a genuine mixture. `lambda` is the **overall strength**: how hard the whole penalty presses. A warning worth carrying into interviews: symbol conventions differ between implementations — some use `alpha` for the strength and a separate name for the ratio — so always confirm which dial you are turning. ## Why mixing beats either penalty alone A pure L1 fit has three well-known weaknesses on wide, correlated data: 1. **Arbitrary choice among correlated predictors.** Given a group of predictors that carry essentially the same information, an L1 fit tends to keep about one of them and zero the rest, and which one it keeps is close to a coin flip decided by sampling noise. 2. **A hard cap when there are more predictors than rows.** With `p > n`, an L1 solution cannot place more than `n` non-zero coefficients; the path saturates there. 3. **Non-uniqueness.** With collinear columns the L1 objective is convex but not *strictly* convex, so several coefficient vectors can achieve the same optimum. Refits on resampled data then disagree about which predictors were "selected". A pure L2 fit has none of those problems — it is strictly convex, unique, and stable — but it never zeroes anything, so on 8,000 predictors you ship 8,000 coefficients and no selection story. The mixture keeps the useful half of each. The absolute-value term still creates a kink at zero, so exact zeros and genuine selection survive. The squared term makes the objective **strictly convex** even when `p > n`, which guarantees a unique solution, lifts the `n` cap, and pulls the coefficients of correlated predictors toward each other rather than letting one win — the **grouping effect**. In geometric terms, the feasible region keeps the L1 corners that sit on the axes (that is where zeros come from) while the rest of its boundary is rounded by the squared term (that is where the grouping and the stability come from). ## Three things the mix does not mean - **A mixing ratio of 0.5 does not mean half the coefficients become zero.** How many zeros you get depends on the overall strength and on how much unique signal each predictor carries. - **Adding an L2 share does not weaken regularisation.** It adds a second penalty term; the fit is more constrained, not less. What changes is the *character* of the constraint. - **The two penalties are not applied in sequence.** There is no "run lasso, then ridge the survivors". One objective, both terms, solved together. ## Choosing the regime - Genuinely sparse signal with near-independent predictors: a pure L1 fit is fine and is more parsimonious. - Dense signal where nearly every predictor contributes a little: a pure L2 fit is the honest model; forcing zeros only discards signal. - Wide and correlated — the case elastic net was designed for: mix, and let cross-validation tell you how much L1 the data will support. If your tuning lands on `alpha = 1` or `alpha = 0`, that is information, not a failure: the data is telling you it is in one of the pure regimes. ## What the penalty costs you Two hyperparameters now need tuning instead of one, and they interact. Every penalty also buys its variance reduction with bias: the surviving coefficients are shrunk toward zero, so they are *not* unbiased effect-size estimates and should not be read as if they were. The intercept is normally left out of the penalty, so that shifting the target up or down does not change the fit's character.