Linear and Logistic Regression

They differ in how many rows each gradient step averages. Full-batch uses all n rows for one update per epoch; stochastic uses one row, giving n updates; mini-batch averages a chunk of B rows, giving about n/B updates.

Log loss is convex in a logistic model's weights, so the error surface has one global minimum and no local minima. Every run that converges lands on the same coefficients, which is why random restarts add nothing.

The sigmoid squashes any linear score into the open range 0 to 1, monotonically: large positive scores approach 1, large negative scores approach 0, and a score of 0 maps to 0.5. The result is read as the event probability.

Logistic regression scores each point with one weighted sum. The sigmoid is monotone, so a 0.5 probability cut is exactly the rule score above zero, and the set where the score equals zero is flat: a line, plane or hyperplane.

Softmax exponentiates each of the K class scores and divides by their total, turning them into K positive numbers that add up to one. The largest score keeps the largest probability, so the predicted class is unchanged.

Sixteen free coefficients against twenty noisy points leave the fit almost enough freedom to pass through every observation, so it chases noise. Polynomial columns are global and highly correlated, so small data changes swing the curve enormously.

Because a larger weight always lowers the loss. When a feature splits the classes with no overlap, scaling that weight up pushes predictions toward 0 and 1 and log loss toward zero, so no finite maximum-likelihood estimate exists.

Cross-entropy is the Bernoulli negative log-likelihood, and its gradient on the linear score is simply predicted minus actual. Squared error through a sigmoid is non-convex in the weights and its gradient nearly vanishes exactly where the model is confidently wrong.

A linear classifier cuts feature space with one flat boundary and labels each side uniformly. XOR puts its two positive cells on one diagonal and its two negative cells on the other, and crossing diagonals cannot be split by a straight line.

Pinball loss at tau = 0.9 charges 0.9 per unit of under-prediction and 0.1 per unit of over-prediction. That nine-to-one asymmetry is minimised by the value the target falls below 90 percent of the time.

Squared error penalises a residual by its square, so its gradient grows with the residual: a point ten times further off pulls ten times harder. Absolute error's gradient has fixed size, so extreme points get no extra vote.

Squared error is smooth everywhere, so setting its derivative to zero gives one exact formula for the best-fitting weights. Absolute error has a kink at zero, needs iterative fitting, and squaring also makes large misses far more expensive.

A linear model is additive: every feature contributes its own amount, and nothing lets one feature change another's contribution. If discount depth works differently for each loyalty tier, that column has to be built by hand.

Shuffling makes each batch a random sample of the training set, so its averaged gradient fairly estimates the full-data gradient. Without it, a sorted file gives correlated batches, a sawtooth loss curve, and weights biased toward the pass's last rows.

Rows are touched once, in a streaming pass building a p-by-p system. Solving that system costs about p^3 and stores p^2 numbers, so total cost is roughly n*p^2 + p^3: wide designs break the exact solve, not tall ones.

Delta is the residual size at which Huber switches from squared to absolute behaviour. Below delta a row's pull grows with its error; above delta the pull is capped at delta, so extreme rows stop dominating the fit.

Multiclass log loss averages the negative log of the probability given to the true class. Uniform guessing over 20 languages scores log 20, about 3.0, so 1.2 beats chance and means an average true-class probability near 0.30.

One multinomial fit trains all 40 classes jointly through a shared normaliser, so its probabilities already sum to one. Forty one-vs-rest fits are independent, so their scores need renormalising, while one-vs-one would need 780 models.

Log the loss per update, not per epoch. A loss climbing geometrically with weights flipping sign means the step size is too large: cut the learning rate tenfold. A NaN on the very first update points to bad inputs or a broken loss.

Those two symptoms together are the signature of separated data, not slow learning. The loss is creeping toward zero along a direction with no finite optimum, so the weights grow without bound and extra training will not help.

A wrong-sign weight tilts the decision boundary against domain sense, so treat it as a suspected defect: check how the label and the feature are coded and computed, whether a correlated predictor absorbs the effect, and whether the sign is stable.

Near-duplicate columns leave the cross-product matrix almost singular, so the two coefficients come out enormous, opposite in sign and wildly unstable across refits. Predictions inside the training range stay fine; the individual coefficients are not interpretable.

A four-to-one cost ratio is pinball loss with tau = 4/(4+1) = 0.8, so forecast the 80th percentile of demand rather than the mean. Confirm the ratio is real, linear and stable before baking it into training.

Squaring the gap between a label and a sigmoid composes a bowl with an S-curve, producing flat saturated regions and stationary points that are not the global best. Cross-entropy instead collapses to a convex function of the linear score.

Polynomial terms grow without bound, so once you leave the fitted range the highest power dominates and the prediction diverges. Least squares constrains the curve only where observations existed, and nothing pins down the tails.

Three, used together: the full-data gradient norm falling below a relative tolerance, the relative improvement in the epoch-average loss falling below something like 1e-6, and a maximum-epoch budget as a backstop. Record which one fired.

The label becomes the treatment indicator and the probability level matters more than the ranking. Keep confounders even when they add no accuracy, exclude anything measured after treatment, and do not rebalance the classes or over-penalise the weights.

Two independent fits can cross, putting the P10 above the P90 for some inputs, and their 80 percent width is only nominal. Tail quantiles rest on few effective rows, so verify empirical coverage per segment.

Softmax divides by the sum of the exponentials, so a constant added to every score contributes the same factor to numerator and denominator and cancels. Only differences between class scores are identified, which leaves softmax over-parameterised.