What is the jackknife, and why does it break down for the sample median?

## The mechanics The jackknife is the oldest of the resampling methods and the simplest to state. Given a sample of n observations and a statistic theta_hat computed on all of them: 1. For each i from 1 to n, delete observation i and recompute the statistic on the remaining n - 1 observations. Call the result theta_(i). 2. Let theta_(.) be the average of the n leave-one-out values. 3. The **bias estimate** is (n - 1) times (theta_(.) - theta_hat). Subtracting it gives the bias-corrected estimate, which can also be written as n times theta_hat minus (n - 1) times theta_(.). 4. The **variance estimate** is (n - 1)/n times the sum over i of (theta_(i) - theta_(.)) squared. There is no randomness anywhere: run it twice and you get identical numbers, with no seed and no B to argue about. The cost is exactly n recomputations. ## Pseudo-values An equivalent presentation defines the i-th **pseudo-value** as n times theta_hat minus (n - 1) times theta_(i). The average of the pseudo-values is precisely the bias-corrected estimate, and their sample variance divided by n is precisely the jackknife variance estimate. The appeal is conceptual: the pseudo-values behave, for smooth statistics, roughly like n independent observations of the quantity of interest, so ordinary formulas for a mean apply to them. For a sample mean the pseudo-values are exactly the original observations, which is a useful sanity check on the definitions. ## Why the (n - 1) scaling appears twice Both formulas carry an inflation factor, and for the same underlying reason. Leave-one-out samples are enormously similar to each other: any two of them share n - 2 observations. Their raw spread therefore badly understates how much the statistic would move across genuinely independent samples, and the (n - 1) factors scale that shrunken spread back up. In the bias formula, (n - 1) converts the small observed shift between theta_(.) and theta_hat into an estimate of the bias at sample size n, exploiting the fact that bias of a smooth estimator typically behaves like a constant over n. ## The relationship to the bootstrap The jackknife can be read as a **linear approximation** to the bootstrap. It probes the statistic's sensitivity to each single observation, which is a first-order, one-point-at-a-time picture of how the statistic responds to perturbing the empirical distribution. The bootstrap perturbs the whole distribution at once and captures higher-order behaviour. When the statistic is smooth, that linear picture is enough, and the two agree closely. When it is not, the approximation misses the point entirely. ## Why the median defeats it Consider an odd-sized sample, say n = 11, sorted. The median is the sixth value. Delete an observation from the upper half and the median of the remaining ten values becomes the average of the fifth and sixth. Delete one from the lower half and it becomes the average of the sixth and seventh. Delete the median itself and you get the average of the fifth and seventh, or a similar neighbouring pair. Across all eleven deletions the leave-one-out median takes only about three distinct values, all sitting within one order statistic of each other. The resulting spread is a function of the local spacing between two or three central order statistics, not of the sampling variability of the median, and it does not converge to the right answer as n grows. The jackknife variance estimate for the sample median is **inconsistent**: more data does not fix it. The technical statement of the problem is that the median is not a smooth functional of the distribution; it responds to a small perturbation in a way that a one-point-at-a-time probe cannot see. The same warning applies to other statistics defined by ranks or thresholds, and to any estimate produced by a selection step. ## The delete-d jackknife The standard repair deletes a block of d observations at a time rather than a single one, averaging over many such subsets. Letting d grow with n rather than staying pinned at one restores consistency for statistics such as the median, because removing a block actually moves the statistic by an amount that reflects genuine sampling variability. The costs are that you now have to choose d, and that enumerating all subsets is infeasible, so you sample them and reintroduce the Monte Carlo noise that the plain jackknife avoided. ## Where the jackknife still earns its place It has two live roles. First, it is the customary way to estimate the acceleration constant in a BCa bootstrap interval, which is defined directly in terms of the leave-one-out values. Second, it gives a cheap, deterministic, seed-free standard error for smooth statistics such as means, ratios and regression coefficients, which is genuinely convenient when reproducibility matters and nobody wants to negotiate over the number of replicates. For anything non-smooth, reach for the bootstrap or a delete-d variant instead.