Regularization

Early stopping halts an iterative fit once a held-out validation score stops improving, and keeps the best-scoring iterate. Each extra iteration lets the model absorb finer detail from the training rows, so stopping sooner limits effective capacity and cuts variance.

Elastic net penalises the sum of absolute coefficients (L1) and the sum of squared coefficients (L2) together. The L1 part drives weak predictors to exactly zero; the L2 part keeps correlated predictors together and makes the fit stable.

Ridge with lambda = 0 reproduces the ordinary least squares fit. As lambda grows, every slope coefficient shrinks smoothly toward zero and the model flattens toward a constant, trading variance for bias. No coefficient reaches exactly zero at finite lambda.

An L1 or L2 penalty adds up coefficient sizes, and a coefficient's size depends on the units of its predictor. Standardising puts every predictor on a common spread so one lambda penalises them all comparably.

Correlation between their errors. Averaging M fits whose errors have pairwise correlation rho leaves rho times the single-fit variance no matter how large M grows; only the independent share, (1 - rho)/M of the variance, is averaged away.

An L1 penalty adds lambda times the absolute value of each coefficient, and the slope of that term stays at lambda right up to zero. That constant pull can push a coefficient exactly to zero; an L2 penalty's pull fades away and never does.

A ridge penalty adds lambda times the sum of squared coefficients to the squared-error objective. Every coefficient is pulled proportionally toward zero, none lands exactly on zero, and the fit trades a little bias for much steadier estimates.

Raising lambda shrinks every coefficient toward zero and pushes more of them to exactly zero, so the fit uses fewer predictors. Past a large enough lambda every slope is zero and only the intercept survives.

In expectation over the noise, the jittered squared error equals the clean squared error plus n*sigma^2 times the squared weight norm — the ridge objective. Large weights amplify input noise, so the fit keeps them small.

The mixing ratio sets the penalty's character — how much of it is L1 versus L2 — and the strength sets how much shrinkage is applied in total. They interact, so the best strength changes whenever the ratio changes.

Lambda = 0.05. Take the lowest mean error, 2.41, add the standard error at that point, 0.09, to get a threshold of 2.50, then pick the largest lambda whose mean still sits under it.

The intercept only sets the model's overall level, and zero is not a neutral level. Shrinking it would drag every prediction toward zero and make the fit depend on the target's arbitrary origin, so it is fitted freely.

Entry order records which predictor was most correlated with the leftover residual at that penalty, in this one sample. It shifts with scaling, with correlations between predictors, and across resamples, and says nothing about effect size.

Set early-stopping patience from the noise scale of the monitored curve: it must outlast the runs of worsening checks that random variation alone produces. Patience of one halts still-improving fits; over-long patience costs only compute.

A pure L1 fit can place at most 200 non-zero coefficients when there are only 200 patients, and inside a co-expressed module it keeps roughly one gene chosen by sampling noise. Adding a squared-penalty share lifts that cap and keeps correlated genes together.

The L1 penalty charges the same total for splitting one effect across two nearly identical columns as for loading it all on one, so it is indifferent between them. A tiny noise-level difference decides the winner, and resampling can reverse it.

On a flat curve the winner is decided by fold noise, not by any real difference: reshuffle the folds and the minimum jumps to a neighbouring lambda. That lowest mean is also optimistically low.

Three sensors on one housing carry nearly the same signal, so least squares gives huge cancelling coefficients. The squared penalty crushes exactly that badly determined direction and spreads the shared effect evenly across the channels.

Read parameter sharing as a regularizer: a shared coefficient block is estimated from all the data and cuts variance, at the cost of bias if the lines really differ. Decide on per-line volume, similarity, and cold start.

You rarely know the right penalty in advance, so the whole path is needed anyway. The descending grid makes it cheap: at lambda_max the solution is known to be all zeros, and each fit resumes from the previous one.

Gradient descent started at zero moves fastest along directions the data determines well and slowest along weak ones, leaving the weak ones near zero. A ridge penalty shrinks exactly those most, and more iterations act like a smaller penalty.

Log loss against a hard 0/1 target keeps falling as the prediction approaches it, so the fit inflates coefficients without limit. Smoothing targets to 0.95 and 0.05 puts the minimum there, capping the log-odds and the weights.

An independent zero-mean Gaussian prior on the weights. With Gaussian noise of variance sigma squared and prior variance tau squared, the posterior mode is exactly the ridge estimate, with lambda equal to sigma squared divided by tau squared.

There is no universal answer. Dividing a rare 0/1 flag by its tiny standard deviation stretches its range and lets it escape most of the shrinkage, though it rests on a handful of rows.

Take the minimum when accuracy is the entire objective and the dip is deep relative to the error bars. The one-standard-error rule spends a little expected accuracy to buy a simpler, more reproducible model.