Statistical Inference & Hypothesis Testing

An estimator is unbiased when its expected value across all possible samples equals the true parameter. Its estimates are centred on the target: too high as often, and by as much, as they are too low.

A probability treats the parameter as fixed and asks how likely the data are; a likelihood fixes the observed data and reads the same formula as a function of the parameter. Likelihood values do not sum to one over parameters.

The standard error of the mean measures how precisely a sample mean estimates the population mean; the sample standard deviation measures how spread out individual observations are. Only the standard error shrinks as the sample grows.

Deviations are measured from the sample mean, which is pulled toward the data and makes those deviations as small as they can possibly be. Dividing their squares by n therefore underestimates the population variance by the factor (n-1)/n.

The score is the derivative of the log-likelihood with respect to the parameter; Fisher information is the variance of the score at the true parameter, equivalently minus the expected second derivative. It measures how sharply data pin down the parameter.

The 95% describes the procedure, not one interval. If you repeated the sampling and rebuilt the interval many times, about 95% of those intervals would contain the fixed true parameter. Any single interval either covers it or does not.

Higher confidence needs a larger critical value. On a normal reference the two-sided values are 1.645 at 90%, 1.96 at 95% and 2.576 at 99%, so nothing but that multiplier changes and the interval stretches.

A sample quantile's standard error contains the unknown probability density at that quantile: roughly sqrt(p(1-p)/n) divided by that density. The density has to be estimated from the few points sitting near a p95, so practitioners resample instead.

Use t whenever the population standard deviation is unknown and estimated from the sample, which is almost always. The z shortcut is exact only when that spread is known, and adequate once the sample is large.

Only about five of 500 observations exceed the p99, so the estimate rests on two or three sorted values. Resampling only reshuffles those, giving a wide, chunky interval. The width is real information, not a defect.

The null hypothesis (H0) is the default claim of no effect or no difference, written as an equality so its sampling distribution can be computed. The alternative (H1) is the competing claim that only evidence in the data can support.

A p-value of 0.03 says that if the null hypothesis of no difference were true, data at least this extreme would arise about 3% of the time. It is not the probability that the null hypothesis is true.

Use a one-tailed alternative only when a result in the opposite direction would be acted on exactly like no effect, and only when the direction is fixed before the data. Otherwise two-tailed is the honest default.

A p-value blends how big an effect is with how precisely it was measured. Standard error shrinks roughly as one over the square root of the sample size, so at a million observations even a trivial difference becomes statistically detectable.

A test controls only the risk of wrongly rejecting a true null; it gives no guarantee that a null which survives is true. A non-significant result is a verdict of not proven, not evidence of no effect.

One-way ANOVA tests whether the population means of three or more groups defined by a single factor are all equal. The null says every group mean is the same; the alternative says at least one differs.

A two-sample t-test assumes observations are independent within and between groups, that the sampling distribution of each group mean is approximately normal, and, for the pooled version, that the two populations share a common variance.

A chi-square goodness-of-fit test compares observed category counts with the counts a hypothesised distribution predicts. A fair die over 600 rolls predicts 100 per face; the statistic sums (observed minus expected) squared, divided by expected, across the six faces.

A two-sample Kolmogorov-Smirnov test compares the two samples' entire distributions. It builds a step-shaped empirical CDF for each sample and takes the largest vertical gap between them, so a difference in location, spread or shape can all trigger rejection.

Rank-based tests replace each value with its position in sorted order, so a mistyped 10,000 becomes only the largest rank, not a huge number. Its influence is capped at one rank; a mean and a variance have no such cap.

A Type I error rejects a null hypothesis that is actually true, a false alarm whose long-run rate is alpha. A Type II error fails to reject a null hypothesis that is actually false, a missed effect whose rate is beta.

Power is the probability a test rejects the null when a specified real effect exists. 80% power means that if an effect of that size is truly present, the test detects it 80% of the time.

Divide the difference in means by the pooled standard deviation: 3 / 15 = 0.2. The two groups sit one fifth of a standard deviation apart, which Cohen's conventional benchmarks call a small effect, and their distributions overlap heavily.

Lowering the significance level raises the Type II error rate. Demanding stronger evidence shrinks the rejection region, so with the same data and the same true effect, more real effects fall short of the threshold and go undetected.

Power rises with sample size, with the size of the true effect, with a larger alpha, and with lower outcome variance. Three of those are design choices; the true effect is not yours to set, only to assume honestly.

The bootstrap treats your sample as a stand-in for the population: draw many new samples of the same size with replacement, recompute the statistic on each, and read the interval off the middle 95% of those values.

About 64 percent. Each test independently avoids a false positive with probability 0.95, so all 20 stay clean with probability 0.95^20 = 0.36, leaving roughly a 64 percent chance that at least one comes back significant by luck.

Not on that evidence. On 1,000 examples, 0.3 accuracy points is three examples, and a net of three can never reach statistical significance in a paired comparison. The gap sits inside the test set's noise.

A permutation test shuffles the group labels thousands of times, recomputing the test statistic on each shuffle to build the null distribution from the data itself. The p-value is the share of shuffles at least as extreme as the observed statistic.

Bonferroni tests each of the 15 endpoints at 0.05/15 = 0.0033 instead of 0.05. That keeps the family-wise error rate under 5 percent whatever the dependence between endpoints, but the tiny per-test threshold makes real effects far harder to detect.