When does distilling a frontier model into a small tuned model beat prompting it?

## What distillation means here In this decision, distillation means using a strong model's outputs as the training data for a smaller model that you then serve. You are not trying to make the small model smarter than its teacher; you are trying to buy most of the teacher's task-specific competence at a fraction of the per-call cost and latency. It is the fourth rung of the prompt → retrieval → fine-tune → distill ladder, and it is the rung that most often has a clear, calculable business case. ## The preconditions **The teacher already works.** If a frontier model cannot do the task acceptably, distillation has nothing to copy. Fix quality first; a student trained on mediocre traces inherits the mediocrity and adds its own. **The task is narrow and stable.** A small model gives up general capability. That is only safe when the job is bounded — classify this alert, extract these fields, draft this memo type — and when the specification is not going to change next month, because every change means regenerating traces and retraining. **Volume makes cost the binding constraint.** At a few thousand calls a month, engineering time dwarfs inference cost and distillation is a distraction. At millions of calls a month it can be the difference between a programme that ships and one that does not. Compute the actual monthly bill at both price points before proposing it. **You can legitimately use the teacher's outputs.** Provider terms differ on training competing models from their outputs. This is a real gating condition, not a footnote. ## The shape of the project Collect traces from production or from a representative input set: the input, the teacher's reasoning where it is exposed, and the final output. Filter aggressively — a judge model or programmatic checks to drop wrong, malformed or degenerate rows — and keep an eye on diversity, because traces sampled from real traffic skew toward whatever is most common and the student inherits that skew. Hold out an evaluation slice that never touched training. Then train a small strong open-weight base. In 2026 the usual configuration is a lightweight adapter on a capable small model rather than a full-parameter run; the adapter approach reaches comparable quality at a fraction of the compute, which matters because you will do this again every time the task or the base moves. ## Serving the result The realistic deployment is a cascade, not a replacement. Route the routine tier to the student and keep an escalation path to the teacher for inputs the student is unsure about, for high-stakes decisions, or for anything a confidence signal flags. A KYC operation running three million alerts a month might send the overwhelming bulk of clean, low-risk alerts to the tuned small model and everything with adverse-media or complex-ownership signals to the frontier model. The cost saving comes from the volume tier; the risk protection comes from the escalation tier. Retrieval does not go away. The student is smaller and therefore *more* dependent on being handed the relevant records rather than recalling them. ## Why the crossover keeps moving As of mid-2026 two things have made the naive "small model is cheaper, therefore distil" argument weaker than it was. Caching of repeated prompt prefixes has cut the marginal cost of a long, stable system prompt on a large model substantially. And the price of capable small hosted models has fallen far enough that the gap you are arbitraging is narrower than the headline flagship price suggests. Measure against the actual configuration you would ship — cached prompt, cheapest adequate hosted model — not against the flagship list price. ## What you are signing up for A distilled model is a system you now operate. You own the trace pipeline, the filtering, the eval suite, the serving path for the small model, and the decision of what to do when a materially better base ships — which, at current cadence, is every few months. Many teams find that a new small base plus a good prompt matches last quarter's distilled model with none of the maintenance. Budget a periodic re-measurement rather than assuming the tune stays ahead. ## Answering well Lead with "it is a cost decision, not a quality decision", give the preconditions, describe the cascade rather than the replacement, and mention that you would re-derive the crossover with current prices instead of quoting a rule of thumb.