How does iteratively reweighted least squares fit a logistic regression model?

## Why an iterative fit at all A logistic model gives each observation a fitted probability `p_i` that depends on the coefficient vector `beta` through the linear predictor `eta_i = x_i^T beta`. The log-likelihood is ``` l(beta) = sum_i [ y_i * eta_i - log(1 + exp(eta_i)) ] ``` Setting its derivative to zero produces equations that are nonlinear in `beta` — unlike ordinary least squares, there is no formula that hands you the answer. So the fit is an optimization: maximise a smooth concave function of p parameters, iteratively. ## The derivatives Differentiating gives the score vector ``` grad l = X^T (y - p) ``` where `y` is the vector of 0/1 outcomes, `p` the vector of current fitted probabilities, and `X` the n-by-p design matrix. The interpretation is clean: each column of X is weighted by how far the outcomes sit from what the model currently predicts. Differentiating again gives ``` Hessian l = -X^T W X, W = diag(p_i (1 - p_i)) ``` Two facts about this Hessian matter. First, it depends only on the fitted probabilities and not on the observed outcomes `y` at all — a special property of this link function. Second, `X^T W X` is positive semidefinite for any non-negative weights, so the log-likelihood is concave: no local maxima to get trapped in, and a unique maximiser whenever `X` has full column rank and the data are not separable. ## The Newton step, rewritten Newton's method for maximising subtracts the inverse Hessian times the gradient with the appropriate sign: ``` beta_new = beta + (X^T W X)^-1 X^T (y - p) ``` Now do a small algebraic rearrangement. Define the **working response** ``` z = X beta + W^-1 (y - p) ``` that is, the current linear predictor plus a residual scaled by the inverse weight. Substituting shows ``` beta_new = (X^T W X)^-1 X^T W z ``` which is exactly the formula for a weighted least-squares regression of `z` on `X` with weights `W`. So every Newton step is a weighted linear regression. Recompute `p`, recompute `W` and `z`, refit. That is the entire algorithm, and it is where the name comes from: iteratively **reweighted** least squares. ## Reading the weights The weight for observation i is `p_i (1 - p_i)`. This is maximised at `p_i = 0.5`, where it equals 0.25, and falls toward zero as the fitted probability approaches 0 or 1. Observations the model is uncertain about therefore dominate the update, while ones it is already confident about barely influence it. This is a genuine property of the curvature: the log-likelihood surface is most sensitive to coefficient changes for observations sitting near the decision boundary. ## Convergence and its failures Because the objective is concave and the Newton step is exact for the local quadratic model, IRLS typically converges in fewer than ten iterations from a zero starting vector, and the convergence is quadratic near the optimum. Standard stopping rules watch the change in the log-likelihood or the relative change in coefficients. The well-known failure is **complete or quasi-complete separation**: some linear combination of the predictors perfectly (or almost perfectly) splits the two classes. The likelihood then has no finite maximiser — it keeps increasing as the coefficients grow — so the fitted probabilities are driven to 0 and 1, the weights `p(1-p)` collapse toward zero, `X^T W X` becomes numerically singular, and the coefficients diverge while the algorithm reports huge, meaningless estimates. Diagnosing it means looking at coefficient magnitudes and fitted probabilities rather than trusting a convergence flag. The standard remedies are to add a penalty that keeps coefficients finite, to drop or combine the offending predictor, or to accept that the data do not identify the effect. A second, milder failure is **collinearity**: if `X` has near-dependent columns, `X^T W X` is ill-conditioned and coefficient estimates are unstable across refits even when the algorithm converges. ## What the question is testing An interviewer asking this wants to see that you know a logistic fit is an optimization rather than a formula, that you can identify the algorithm as Newton's method in disguise, and that you can explain the reweighting in terms of curvature. Naming the separation failure and its symptom — exploding coefficients with vanishing weights — is what separates a candidate who has fitted these models in anger from one who has only read about them.