skip to content

Why do a logistic regression's weights run to infinity when a feature perfectly separates the label?

level: middleimportance: must knowfreq 55%

answer

  1. no overlap between the two classes
  2. scaling the weight always lowers the loss
  3. the infimum is never attained
  4. convex, but with no bottom
  5. the estimate does not exist

basics

~20 s

Because a larger weight always lowers the loss. When a feature splits the classes with no overlap, scaling that weight up pushes predictions toward 0 and 1 and log loss toward zero, so no finite maximum-likelihood estimate exists.

solid answer

~50 s

Complete separation means some combination of features classifies every training row correctly with no overlap — for example an HR promotion dataset of 60 rows where every certified employee was promoted and no uncertified one was. Take the weight on that indicator and double it: every predicted probability moves closer to 0 or 1, every row's loss falls, and the total log loss drops. The loss decreases along that ray forever, approaching zero but never attaining it, so no finite maximum-likelihood estimate exists. The surface is still convex — a convex function is perfectly entitled to have no bottom. What you see is a coefficient that grows every epoch and never settles, reaching 20 or more with a standard error in the thousands, 100% training accuracy, and a fit that stops only because it ran out of iterations. Quasi-complete separation is the same pathology confined to a subgroup, with ties on the boundary.

code

python · 23 lines
python
import math

# 6 employees: certified (x=1) always promoted, uncertified never -> complete separation
X = [0, 0, 0, 1, 1, 1]
Y = [0, 0, 0, 1, 1, 1]
w = b = 0.0
lr = 0.5

for epoch in range(1, 20001):
    gw = gb = 0.0
    for x, y in zip(X, Y):
        p = 1.0 / (1.0 + math.exp(-(w * x + b)))
        gw += (p - y) * x
        gb += (p - y)
    w -= lr * gw / len(X)
    b -= lr * gb / len(X)
    if epoch in (100, 1000, 10000, 20000):
        print(epoch, round(w, 3), round(b, 3))

# 100 4.413 -1.976
# 1000 9.18 -4.385
# 10000 13.845 -6.719
# 20000 15.236 -7.415

go deeper

for a junior

Recall the symptom picture: one coefficient in the tens, an enormous standard error, perfect training accuracy, and a fit that never settles. Knowing this pattern means a perfect predictor is a warning sign, not a win.

for a middle

Explain the mechanism: scaling the weight vector up strictly lowers every correctly classified row's loss, so the loss decreases forever along that ray and no finite estimate exists. Be clear that the loss is still convex.

for a senior

Show how you find it in a real dataset — cross-tabulating levels against the label to find empty cells — and how you choose between pooling rare levels, dropping an artefact column, gathering data, or moving to a penalised fit.

for a principal

Own the policy question: high-cardinality categorical encoding plus small strata makes separation routine, so decide up front on minimum level counts, pooling rules and whether unpenalised maximum likelihood belongs in your pipeline at all.

## The definition A binary logistic fit has **complete separation** when there exists a weight vector such that every positive row gets a score above zero and every negative row a score below zero, with no row on the boundary. In the simplest case a single indicator does it: the column is 1 for every positive and 0 for every negative. **Quasi-complete separation** is the same thing with ties: the classes can be split with no misclassification, but one or more rows sit exactly on the boundary and share a score. A rare-disease study with 9 positive cases that all carry one indicator, while some negatives carry it too, is the usual shape — the pathology is confined to that subgroup, but it is still enough to prevent a finite estimate for the parameters involved. ## Why the weight runs away Write the per-row log loss in terms of the score `z`. For a correctly classified row the loss is `log(1 + e^-m)` where `m` is the row's margin, the score signed so that positive means correct. Now scale the whole weight vector by a factor `t > 1`. Every margin scales by `t`, every `e^-m` shrinks, and **every row's loss strictly decreases**. Nothing pulls back, because there is no row on the wrong side to pay for the confidence. So the loss is strictly decreasing along the ray `t*w` as `t` grows, and its limit is zero. The infimum is zero, but no finite weight vector attains it. There is no maximum-likelihood estimate. Gradient descent keeps taking a real, downhill step every epoch and the run only ends when it hits the iteration cap. Note what has *not* gone wrong. The loss is still convex; convexity says any minimum you find is global, not that a minimum exists. The optimiser is not buggy, the data are not corrupt, and the arithmetic is not unstable in the usual sense. The estimand simply is not there. ## What it looks like in practice - A coefficient in the tens — 15, 20, 30 — where every other coefficient is under 2. - A standard error in the thousands for that coefficient. The likelihood is flat out along the ray, so the curvature used to compute the error is nearly zero and its inverse explodes. - Fitted probabilities pinned at 0 and 1, and training accuracy of exactly 100%. - The fit ending at the iteration limit rather than at a convergence tolerance. - Refitting with more iterations produces a *larger* coefficient, not a stable one. That is the giveaway: a converging fit repeats, a separated one keeps climbing. ## Where separation comes from - **Rare categorical levels.** A one-hot column for a city with 3 rows in it, all of them positive, separates within that level. This is the most common cause by far and it multiplies with high-cardinality categorical encoding. - **Small samples with many features.** Once you have as many features as rows, the data are almost always separable by chance alone, with no signal involved. Adding interaction terms accelerates this. - **Genuinely deterministic rules.** Sometimes a business rule really does decide the outcome, and the model is discovering the rule rather than a relationship. - **A column that restates the label.** Any feature computed from the outcome will separate it perfectly. ## What separation is not It is not collinearity. Two duplicated features give a *flat* direction — many weight vectors with identical loss, so the coefficients are not identified but the fit still converges and the loss stops moving. Separation gives a *downhill* direction that never runs out, so the loss keeps improving and the weights keep growing. Flat versus endlessly downhill is the distinction to keep straight. It is also not overfitting in the usual sense, though it accompanies it. Overfitting is a statement about generalisation; separation is a statement about the estimate not existing. ## What to do about it Start by locating it: cross-tabulate each categorical feature level against the label and look for cells with zero rows of one class. For continuous features, sort by the feature and check whether one class occupies a contiguous end. In practice the separating column is usually obvious once you look. Then decide. Pooling rare levels into an "other" bucket removes the empty cell and typically restores a finite fit. Dropping the offending column is right when it is an artefact rather than a real predictor. Collecting more rows in the empty cell is the honest fix when the level matters. Fitting with a penalised estimator rather than plain maximum likelihood is the other standard route, and it keeps the coefficient finite by construction. What does not work is training longer, tightening the tolerance, or raising the iteration cap. There is nothing to converge to; you are just choosing where along an infinite ray to stop. ## The consequence for interpretation Even if you stop the run somewhere reasonable and the classifier ranks well, that coefficient and its standard error mean nothing — their values are an artefact of when you halted. Any statement of the form "this feature multiplies the effect by X" is unsupportable while the fit is separated. Fix the pathology before reporting anything about that variable.

  • How does quasi-complete separation differ from the complete case?
    Complete separation classifies every row correctly with no row on the boundary. Quasi-complete separation still misclassifies nothing but has rows tied exactly on the boundary, usually inside one subgroup — nine positive cases all sharing one indicator, say. The estimate for the parameters involved still fails to exist, but the rest of the model may be fine, so it is easier to miss.
  • You have 40 rows and 60 features. Why should you expect separation before you even fit?
    With at least as many features as rows, a hyperplane that separates any labelling almost always exists purely by dimension counting — no signal is required. So a perfectly separating fit in that regime is evidence about the shape of your design matrix, not about your predictors. Reduce dimensionality or use a penalised fit before drawing any conclusion.
  • Is the classifier useless while it is separated?
    Its ranking on the training data is perfect and it may still rank new rows sensibly, but its probabilities are pinned at 0 and 1, so anything that depends on calibrated output is broken. The runaway coefficient and its standard error carry no information, since their values depend only on when you stopped. Fix the separation before shipping or interpreting.
  • How do you tell separation apart from two duplicated features?
    Duplicated features give a flat direction: the loss stops improving, the fit converges, and only the split between the two coefficients is arbitrary. Separation gives a permanently downhill direction: the loss keeps creeping toward zero and the weight norm keeps growing. Refit with more iterations — a flat direction reproduces the same loss, separation returns a larger coefficient.

It is like being told to price confidence in a bet you cannot lose. Every extra unit of certainty pays a little more and costs nothing, so there is no point at which stopping is optimal — you just stop when the clock runs out.

saying these in an interview costs you the question

  • Calls it a bug in the optimiser or a numerical overflow
  • Confuses separation with collinearity between two features
  • Says the loss stops being convex under separation
  • Interprets the runaway coefficient as a huge real effect
  • Suggests raising the iteration cap until it converges
  • Treats 100% training accuracy as evidence the model is good

context

resolving links…