Maximum Likelihood Estimation

## The same formula, two different free variables Start from one object: a model that assigns a number to data given a parameter. Write it `f(data | theta)`. Nothing about that expression tells you which symbol you are allowed to vary. That choice is what separates a probability from a likelihood. - **Probability**: hold `theta` fixed at some assumed value, let `data` vary. The result is a genuine distribution over outcomes. It sums (discrete case) or integrates (continuous case) to one across all possible datasets. - **Likelihood**: hold `data` fixed at what you actually observed, let `theta` vary. The result is a function of the parameter, written `L(theta) = f(observed data | theta)`. It is *not* a distribution over `theta`. ## Worked case: 7 heads in 10 flips Suppose a coin is flipped 10 times and lands heads 7 times. The model is binomial with unknown heads probability `p`: `L(p) = C(10,7) * p^7 * (1-p)^3` Evaluate it at a few parameter values: `L(0.5) = 120 * 0.5^10 ~ 0.117`, `L(0.7) = 120 * 0.7^7 * 0.3^3 ~ 0.267`, `L(0.9) = 120 * 0.9^7 * 0.1^3 ~ 0.057`. Reading these correctly: 0.267 is the probability of getting exactly 7 heads in 10 flips **in a world where `p` really is 0.7**. It is not "the probability that `p` is 0.7", and it is not a statement about the coin's bias at all until you compare it with other values. The comparison is the whole point. `L(0.7)` beats `L(0.5)` by a factor of roughly 2.3, so the data are about 2.3 times more probable under `p = 0.7` than under a fair coin. Maximum likelihood estimation formalises exactly this: pick the parameter value under which the observed data would have been most probable. Here that is `p-hat = 0.7`. ## Why the likelihood is not a probability distribution over the parameter Two concrete consequences follow, and interviewers probe both. First, **the likelihood need not integrate to one over the parameter**. For this example, integrating `L(p)` over `p` from 0 to 1 gives exactly `1/11`, not 1. Nothing normalises it, because nothing in the model claims `p` is random. Second, **the absolute value of a likelihood is meaningless on its own**. In the continuous case the model returns a *density*, not a probability, so `L(theta)` can exceed 1 -- a Normal density with a very small standard deviation is tall and narrow, and its value at the peak is large. That is not a broken probability; it is a density evaluated at a point. Only ratios and the location of the maximum are interpretable. This is also why any multiplicative constant that does not involve the parameter can be dropped. The binomial coefficient `C(10,7) = 120` counts the orderings of heads and tails; it does not contain `p`, so `p^7 * (1-p)^3` has its maximum at exactly the same place. Most derivations drop such constants immediately. ## Independence and the product form With independent observations `x1, ..., xn` the joint model factorises, so the likelihood is a product of per-observation terms: `L(theta) = f(x1 | theta) * f(x2 | theta) * ... * f(xn | theta)` Products of many small numbers underflow quickly and are awkward to differentiate, so in practice the **log-likelihood** is used: `l(theta) = sum of log f(xi | theta)`. Because the logarithm is strictly increasing, it has its maximum at exactly the same parameter value, so nothing is lost. ## Saying it cleanly in an interview A crisp phrasing: "Probability runs forward from a parameter to data; likelihood runs backward from data to a parameter, using the same formula. The likelihood is a function of the parameter given fixed data, and it is not a probability distribution over that parameter -- it does not normalise, and its height only matters relative to other parameter values." The classic trap is to answer "`L(0.7) = 0.267` means there is a 27% chance the coin's bias is 0.7". That statement flips the conditioning, and interviewers hear it as a sign that the candidate has not internalised what is being held fixed.