Given a learning curve over training-set size, how do you decide whether 20,000 more labels are worth buying?

## The question is an investment question A learning curve's whole commercial purpose is to answer "should we buy more data?" — and that is a question about **cost per unit of realised value**, not about whether the curve is still descending. It is nearly always still descending a little. The discipline is turning the slope into a number that can sit next to a price. ## Step 1: read the slope on the right axis Put training-set size on a **logarithmic** axis. Empirically, over the practically relevant range, held-out error falls by a roughly constant amount for each doubling of the data and then bends toward its floor. Two consequences follow immediately: - **Gains per row collapse.** Going from 1,000 to 2,000 rows and from 10,000 to 20,000 rows can buy comparable improvements — but the second costs ten times as much labelling. On one scanned-document field-extraction task, halving the error rate required roughly ten times the rows. - **Absolute row counts are the wrong unit.** "We will add 20,000 rows" means something very different at 5,000 existing rows (two doublings) than at 200,000 (a rounding error). Always convert the proposed purchase into doublings before reasoning about it. ## Step 2: extrapolate, but barely Project **one doubling, at most two**, beyond your largest measured point. The curve's shape further out is unknown: it may bend toward the floor much sooner than a straight-line extension suggests, and the floor is exactly where the extension is least reliable. Anyone confidently quoting the error at ten times the current data, from a curve measured over one decade, is quoting an assumption, not a measurement. State the projection as a range and carry the uncertainty band from your replicates into it. ## Step 3: price the projected drop, in the units the business uses Twenty thousand labelled scans is not a number of rows; it is a number of expert hours, a scheduling problem, and an opportunity cost for the people who would otherwise be doing something else. Set that against the projected improvement — say one and a half points of error — and then ask the question that actually decides it: **does one and a half points change any decision the system makes?** - If the model feeds a triage queue with a fixed capacity, the relevant quantity is how many more true cases surface in the top of the queue, not the aggregate error. - If a review threshold is set by a fixed tolerance, the relevant quantity is how much manual review volume falls at that tolerance. - If the improvement is real but no threshold, queue, or automation rate moves, the purchase buys a nicer number in a report. This translation is the part junior analyses skip, and it is the part that makes the recommendation credible to whoever signs off. ## Step 4: check the projection against the floor If the projected error lands close to where the curve appears to be asymptoting — the level set by label ambiguity and by what the current model and features can express — then the purchase is buying the last sliver of an exhausted resource. A projection that comfortably beats the visible plateau, on the other hand, should make you suspicious of the extrapolation rather than optimistic about the purchase. ## Step 5: de-risk with a staged buy The cheapest way to test an extrapolation is to buy a small piece of it. Commission a tranche that adds perhaps a quarter of the proposed volume, re-plot the curve, and check that the new point lands where you predicted. If it does, the remaining purchase is now supported by measurement instead of by a fitted line. If it does not, you have spent a fraction of the budget to avoid a bad decision. Staging also protects against a subtler failure: new data collected under a different protocol, from a different source, or at a different time can fail to behave like the old data at all, and a small tranche exposes that immediately. ## Step 6: compare against other uses of the same budget More rows of the same kind is one option among several with the same price tag: - **Better labels on the cases that matter.** If a meaningful share of remaining errors sit on cases the current labels get wrong, re-examining those may move the achievable floor rather than just walking further down the curve. - **Broader coverage instead of deeper volume.** Rows from a site, device, language, or customer segment the sample barely contains change what the model can generalise to; the existing curve says nothing about them, because it was drawn from the population you already have. - **Nothing at all.** If the projected gain is small, the floor is near, and no decision threshold moves, the honest recommendation is to stop investing in data for this model and say so plainly. ## What a strong answer sounds like "Our curve spans 1k to 20k. Adding 20,000 rows is one doubling; the last doubling bought about two points, and the curve is bending, so I would project one to one and a half points with a wide band. That is roughly this many expert hours. At our current threshold, one and a half points changes the review queue by this much, which is worth about that much per year. I want to buy 5,000 first and confirm the next point before we commit the rest." Numbers, a doubling-based projection, a staged commitment, and an explicit link to a decision the business makes.