When is a bagged ensemble's out-of-bag error a trustworthy substitute for a hold-out set?

## How the estimate is built In a bagged ensemble of B base models, each model was fit on its own bootstrap resample and therefore missed roughly 36.8% of the rows. Invert that view and look at one row: about `0.368 * B` of the base models never saw it. Those are the models allowed to predict it. Aggregate just their predictions — average the probabilities, or take their vote — and you have an out-of-bag prediction for that row that no in-sample information contaminated. Do that for every row and score the result with whatever loss you care about; that is the out-of-bag error. The appeal is that nothing was set aside. On a 1,200-row soil-nutrient agronomy table, carving off 20% for validation costs 240 rows of training signal and leaves an estimate computed on 240 rows, which on a noisy agronomic target is a very wobbly number. Out-of-bag gives you an estimate computed over all 1,200 rows while every row still contributed to training. It costs no extra fits either — it falls out of the resampling you were doing anyway. ## The assumption that makes it honest Out-of-bag error is an honest generalization estimate only if a row being absent from a resample means the model learned nothing about that row. That holds when rows are independent draws from the population. It fails whenever rows come in related clusters: - **Repeated measurements.** If the agronomy table holds four samples per field, an out-of-bag row from field 12 has three near-twins that are almost certainly in-bag in most base models. Those models effectively memorised the field, so predicting the held-out sample is easy and the out-of-bag error reads far better than performance on a genuinely new field. - **Near-duplicates.** The same problem in a subtler form: rows that were copied, joined in twice, or that describe the same physical entity under different identifiers. - **Time order.** Bootstrap resampling draws uniformly from the whole history, so a base model routinely trains on rows from after the out-of-bag row's timestamp. If the target drifts, the out-of-bag estimate is measuring interpolation while production will demand extrapolation. - **Preprocessing fit on everything.** If a transform — an imputation statistic, a target-based encoding, a scaling constant — was computed over all 1,200 rows before bagging, then every base model carries information about its out-of-bag rows regardless of resampling. The common thread is that out-of-bag protects against reusing a *row*, not against reusing the *information in a row*. ## Two biases that pull in opposite directions **Pessimistic direction.** Each row is scored by roughly a third of the base models. A smaller ensemble is a slightly worse predictor than the full one, so out-of-bag error usually overstates the error of the model you actually ship. The gap shrinks as B grows — at B = 200 each row gets about 74 voters, which is close enough to the full ensemble that the effect is small; at B = 20 it is about 7 voters and the estimate is both noisy and clearly pessimistic. If you are going to lean on out-of-bag error, use plenty of base models. **Optimistic direction.** Out-of-bag error is unbiased for one fixed configuration. Compute it for forty configurations and pick the best, and the winner's out-of-bag score is the maximum of forty noisy numbers — it is optimistically biased by exactly the amount you would expect from any repeated selection on the same estimate. The estimate itself was never the problem; reusing it as a selection criterion is. ## What to do in practice On a small, independent-rows table, use out-of-bag error as the working number and say so plainly: it is a legitimate estimate, obtained for free, computed over every row. Sanity-check it once against a small hold-out or a resampling scheme if you can spare the data, and expect the two to agree within noise — a large disagreement is a signal that the exchangeability assumption is broken and worth chasing down. When rows are grouped, out-of-bag error is not repairable by tuning: the resampling unit is the row and the leakage unit is the group, so you need a validation scheme whose splits respect groups. When the data is time-ordered, the same applies with time as the unit. And whatever the case, if out-of-bag error has driven a long search over settings, report a final number from data that search never touched. ## What it is not Out-of-bag error only exists because the base models were trained on resamples that left rows out; it is a property of bagged ensembles, not a general validation technique you can bolt onto any model. It also says nothing about whether the predicted probabilities are well calibrated or how errors are distributed across subgroups — it is one aggregate number, and it should be read as one.