On the quality-cost frontier, how do you choose an operating point for an LLM feature?

## What the frontier is Every routing decision — which model, which effort level, what escalation threshold, whether a semantic cache tier serves the request at all — produces a (quality, cost) pair for your feature. Plot them and most points are *dominated*: some other configuration is both better and cheaper. Throw those away. What remains is the Pareto frontier, the set of configurations where buying more quality strictly costs more money. Choosing an operating point means picking one of those, and the whole value of drawing the frontier is that it turns an argument about opinions into a choice between measured options. ## Building it The frontier has to be measured on *your* task, with your eval suite, on your traffic distribution. Sweep the parameters you can actually change in production — escalation threshold, effort level per class, model pairing, cache similarity threshold — and for each configuration record end-task quality (the score your eval suite produces, not a proxy), blended cost per request, and latency at p50 and p99. Run enough repetitions per configuration that the quality difference between neighbouring points is bigger than the run-to-run noise; otherwise you will "choose" a point that is indistinguishable from its neighbour. Two things make a frontier misleading. First, evaluating on a dataset that does not match production traffic mix — the cheap configurations look far better than they are because the hard segment is underrepresented. Second, measuring quality as an average when the distribution is what matters: a configuration with the same mean quality but a fat tail of catastrophic answers is not equivalent. ## Choosing a point The frontier does not tell you where to sit; the cost of being wrong does. **Quality-floor mode.** Where a bad answer is expensive — advice with regulatory exposure, an irreversible action, anything a customer acts on without review — derive a floor from that cost and take the cheapest frontier point that clears it, with margin for the fact that production quality is usually a little worse than eval quality. Below the floor, cost savings are not savings; they are deferred liability. **Budget-ceiling mode.** Where output is a draft a human edits, or an internal summary, fix an acceptable cost per request from unit economics and take the best quality inside it. **What not to do:** maximize quality per dollar. That ratio is maximized by cheap mediocrity whenever a floor exists, and it is insensitive to exactly the thing that matters — whether the point clears the bar. Ratios are for comparing marginal moves, not for selecting an operating point. ## Marginal accuracy per dollar The useful ratio is the *marginal* one: what does the next increment of spend buy? Frontiers are strongly concave. Moving from a weak configuration to a competent one often costs little and buys a lot; the last few points of quality cost superlinearly, because they come from routing the hardest residual traffic to the most expensive tier at the highest effort. Compute the slope between adjacent frontier points and you can say, concretely, "raising effort from low to high for amended filings costs 2.4x on that class and buys 1.9 points of accuracy" — a sentence a finance partner can act on, unlike "the expensive model is better". ## The frontier is per segment A single global operating point is almost always dominated by a set of per-class points. Split traffic into classes with genuinely different difficulty and different error costs, draw a frontier per class, and choose per class. The routine, high-volume class usually sits far down the cheap end with no measurable quality loss; the rare, high-stakes class sits at the expensive end, and because it is rare its cost barely registers. That asymmetry — most volume in the cheap class, most risk in the rare one — is why segmented routing beats global model selection, and it is the core argument for building a router at all. ## Latency is a third axis Quality and cost make a curve; add latency and it is a surface. A configuration that is attractive on the (quality, cost) plane can be unshippable because escalation doubles the tail or because high effort adds tens of seconds of thinking before anything is visible. Carry p99 alongside every frontier point, and for interactive paths treat the latency budget as a hard constraint that filters the frontier before you choose on it. ## The point is a parameter, not a decision The frontier moves under you. New model versions shift both axes, prices change, your traffic mix drifts, and a prompt improvement can lift the cheap tier enough to redraw everything. Treat the operating point as something with an owner and a re-tuning cadence, re-swept after every model or pricing change, with the sweep results stored so the choice is auditable. Frontier configurations chosen offline still have to be validated on live traffic before they become the default — that rollout machinery is a separate concern from the frontier analysis itself, but skipping it turns a measured choice back into a guess.