How do you construct a learning curve over training-set size so its shape is trustworthy?

## The curve is an experiment, and it has one independent variable Everything below follows from a single principle: a learning curve is only interpretable if **training-set size is the only thing that changes** between points. Every classic mistake in drawing one is some other variable moving at the same time. ## Hold the evaluation set fixed Score every trained model on the **same** held-out rows. A tempting alternative — re-split the data at each size, so that a smaller training set leaves a larger test set — sounds efficient and quietly ruins the plot: the validation numbers are now computed on different samples of different sizes, and a step in the curve could equally be a step in the difficulty of that particular test sample. Fix the evaluation set once, before the ladder starts, and never touch it during the sweep. ## Average repeated draws at each size At small sizes, which model you get depends heavily on **which** rows you happened to draw. Two independent 1,000-row subsamples can differ by several points of held-out error. Plotting one draw per size gives a jagged line whose bumps are pure sampling noise and which people then over-interpret. Instead, draw several subsamples per size — five is a common working number, more at the small end where variability is worst — refit each, and plot the mean along with a band showing the spread. The band is itself informative: a wide band at small n is the variance story showing up before the gap does. A useful refinement is **nested subsamples**: make each larger sample a superset of the smaller one within a replicate, so the curve traces the effect of *adding* rows rather than the effect of drawing an unrelated sample. ## Space the sizes logarithmically Going 1k, 2k, 5k, 10k, 20k tells you far more than 4k, 8k, 12k, 16k, 20k, because gains arrive per doubling, not per row. Evenly spaced sizes waste most of the compute on the flat right-hand end of the curve and leave the steep, informative region under-sampled. Log spacing also matches how the curve will be read and extrapolated later. ## Fit preprocessing inside each subsample Every quantity learned from data — a scaling mean and spread, a category-to-number mapping, an imputation fill value, a target-based encoding, a feature-selection step — must be recomputed **from the subsample being trained on**. If you compute them once on the full training pool and reuse them at every size, then the 1,000-row model quietly benefits from statistics over all 20,000 rows. The small-n points come out too good, the curve looks flatter than it is, and you conclude data does not help when it does. This is the subtlest and most common defect in a hand-rolled curve. ## Decide, and state, the hyperparameter convention There are two defensible conventions and they give different pictures: - **Fixed settings.** Use one configuration at every size. Simple and cheap, but a configuration chosen for 20,000 rows is usually too flexible for 1,000, so the small-n end looks worse than it needs to and variance is exaggerated. - **Re-tuned at each size.** Repeat the selection procedure within each subsample. This shows what a competent practitioner would actually achieve at each data scale, which is the honest input to a data-acquisition decision — at the cost of much more compute, and with the requirement that all tuning happen inside the subsample. Neither is wrong. Reporting which one you used is mandatory, because a reader comparing your curve with another's will otherwise compare two different experiments. ## Keep the subsamples representative Draw **stratified** subsamples when the target is imbalanced, so a 500-row point is not accidentally almost free of positives. When rows are grouped — several records per patient, per document, per customer — subsample by **group**, not by row, or the same entity appears in both training and evaluation and the small-n points are flattered again. If the data is a time series, respect ordering rather than sampling rows at random. ## Read the noise band before reading the trend When the plot is finished, look at the spread bands first. If successive points overlap heavily, the apparent slope between them is not evidence of anything, and the honest conclusion is "cannot tell at this resolution" — either add replicates or accept a coarser reading. A curve reported without any notion of its own uncertainty invites confident conclusions the experiment does not support. ## The checklist One fixed evaluation set; several replicates per size with a spread band; log-spaced sizes; preprocessing refit inside each subsample; a stated hyperparameter convention; stratified or group-aware sampling. Miss any one and the shape you read may be an artefact of the procedure rather than a property of the problem.