Likelihood Asymptotics

## Setting up Suppose data `x_1, ..., x_n` are modelled by a density or mass function `f(x; theta)` indexed by an unknown parameter `theta`. The likelihood `L(theta)` is that model evaluated at the observed data and read as a function of `theta`, and the log-likelihood is `l(theta) = log L(theta)`. Everything in likelihood asymptotics is built from the first two derivatives of `l`. ## The score function The **score** is the first derivative of the log-likelihood: `U(theta) = d/dtheta l(theta)` Setting `U(theta) = 0` is the estimating equation whose solution is the maximum-likelihood estimate, but the score is interesting in its own right as a random quantity: it is a function of the data, so it has a distribution. Under **regularity conditions** — chiefly that the support of `f` does not depend on `theta`, and that you may swap the order of differentiation and integration — the score has mean zero when evaluated at the true parameter: `E[U(theta)] = 0` The one-line derivation: the density integrates to 1 for every `theta`, so differentiating `integral f(x; theta) dx = 1` gives `integral d/dtheta f(x; theta) dx = 0`. Writing `d/dtheta f = f * d/dtheta log f` turns the left side into `E[U(theta)]`, hence zero. ## Fisher information: two equivalent definitions **Definition 1 — variance of the score.** Because the score has mean zero, its variance is just its second moment: `I(theta) = Var(U(theta)) = E[U(theta)^2]` **Definition 2 — expected negative curvature.** Differentiating the mean-zero identity a second time gives the *information equality*: `I(theta) = -E[d^2/dtheta^2 l(theta)]` The second form is usually the easier one to compute, and it is the one that gives information its interpretation. `d^2 l / dtheta^2` is the curvature of the log-likelihood. A sharply peaked log-likelihood (large negative curvature) means that moving `theta` even slightly away from its best value makes the data much less probable — the data strongly discriminate between nearby parameter values. A flat log-likelihood means many parameter values explain the data about equally well, and information is small. Note that the two definitions coincide **only** under the regularity conditions above; the information equality is exactly what fails for misspecified models, which is why robust (sandwich) standard errors combine both quantities instead of using either alone. ## Additivity over observations For independent observations the log-likelihood is a sum, so the score is a sum and — by independence — the information adds: `I_n(theta) = n * I_1(theta)` where `I_1` is the information in a single observation. This additivity is why sample size shows up as a factor of `n` in every downstream result, and ultimately why precision improves like `1/sqrt(n)`. ## Worked example: Bernoulli For one observation `x` in {0, 1} with success probability `p`: - log-likelihood: `l(p) = x log p + (1-x) log(1-p)` - score: `U(p) = x/p - (1-x)/(1-p)` - second derivative: `-x/p^2 - (1-x)/(1-p)^2` - negated expectation, using `E[x] = p`: `p/p^2 + (1-p)/(1-p)^2 = 1/p + 1/(1-p) = 1 / (p(1-p))` So `I_1(p) = 1 / (p(1-p))`. Sanity checks: the score has mean zero, since `E[U(p)] = p/p - (1-p)/(1-p) = 0`. And the information is minimised at `p = 0.5` and blows up as `p` approaches 0 or 1 — a coin that almost always lands heads is highly informative per flip about *how* extreme `p` is, whereas a fair coin's outcomes are maximally ambiguous. With `n` independent flips, total information is `n / (p(1-p))`, and its inverse `p(1-p)/n` is the asymptotic variance attached to the estimate — the precision statement falls straight out of `1 / sqrt(n * I_1)`. ## Multiple parameters When `theta` is a vector, the score is a gradient vector and the information is a matrix: `I(theta) = E[U U^T] = -E[H]`, where `H` is the Hessian of the log-likelihood. Off-diagonal entries record how much two parameters trade off against each other. Large off-diagonal information means the parameters are hard to separate — the log-likelihood ridge runs diagonally, and each parameter individually is poorly determined even when their combination is well determined. ## Why interviewers ask Score and information are the two objects everything else in likelihood theory is written in: the precision bound on unbiased estimators, the asymptotic distribution of the maximum-likelihood estimate, the standard errors reported next to fitted coefficients, and the three classical test statistics. A candidate who can define the score, state the mean-zero property, give both forms of information, and explain the curvature intuition has the vocabulary for all of it.