How do you tell that a fine-tune has overfit its training set?

## What overfitting means for a fine-tuned language model Overfitting is the model reducing loss by absorbing properties of the specific training examples that do not transfer - the annotators' phrasing, the exact entities that appeared, the layout of the reference answers - rather than the underlying behaviour you wanted to teach. Fine-tuning is unusually prone to it because the datasets are small (hundreds to tens of thousands of rows against a base model with billions of parameters) and the examples are often stylistically homogeneous, having come from a handful of authors or a single synthetic generator. The distinctive thing about overfitting in a language model is that it does not look like noise. It looks like *confidence*. An overfit model produces well-formed, domain-shaped, plausible text, and only fails when the input drifts away from the training distribution - which is exactly what production traffic does. ## Signal one: curve divergence The classic evidence is the gap between the two loss curves. Training loss falls monotonically because that is what the optimiser is doing. Held-out loss, computed on examples never used for updates, falls while the model is learning transferable structure and turns upward once it starts memorising. In a concrete case - a crop-disease advisory fine-tune - held-out loss bottomed at epoch two and rose steadily while training loss kept falling through epoch four. The best checkpoint was the one at the held-out minimum, and everything after it was strictly worse for deployment despite looking better on the training curve. Two cautions. First, the divergence point is a property of *this* dataset and *this* configuration, not a universal epoch count, so it must be measured per run. Second, loss is only a proxy; on some runs the graded task metric peaks a little before or after the loss minimum, and when they disagree the task metric wins. ## Signal two: verbatim recitation The most legible output symptom is the model reproducing training-set phrasing word for word. It is directly measurable: take the generated answers for your held-out inputs, and compute the fraction that share a long n-gram (eight tokens is a common threshold) with any training completion. Track that number across checkpoints. A sharp rise means the model is retrieving stored strings rather than composing an answer, and the strings it retrieves will be wrong whenever the retrieved case does not match the new input. This also has a privacy dimension. If the training data contained customer names, addresses or identifiers, a memorising model can surface them in unrelated answers, which turns an eval finding into an incident. ## Signal three: paraphrase brittleness A model that learned the task should be robust to how the question is asked. A model that learned the dataset is not. The test is cheap: take a slice of held-out inputs, rewrite each into different wording that preserves meaning - reorder the facts, change register from clinical to conversational, translate the units, vary the length - and re-score. A model whose score holds within noise has learned behaviour. A model that drops sharply has learned surface form. This is one of the most informative checks available and it is routinely skipped, because it requires building a paraphrased variant of the eval set. A related probe is *distribution shift by construction*: hold out inputs that come from a period, region or entity absent from training, and compare the gap against the in-distribution slice. A widening gap across checkpoints tracks overfitting directly. ## Signal four: diversity and calibration collapse Late-stage overfit models become templated. Every answer follows the same three-paragraph shape, opens with the same clause, and reaches for the training set's most frequent conclusion. Two symptoms are worth measuring: **output diversity** - the number of distinct answer templates or the type-token ratio across a batch of varied inputs - and **abstention behaviour**. If the training data contained no examples of the model declining or expressing uncertainty, the fine-tune learns that a confident answer is always correct, and abstention rates fall to zero even on inputs where the right answer is "insufficient information." On a diagnostic advisory task, that is the most dangerous failure of the four, because it converts uncertainty into an authoritative recommendation. ## Reading the signals together No single signal is decisive. The pattern that confirms overfitting is *coherent movement across several of them at the same checkpoint*: held-out loss turning up, n-gram overlap climbing, the paraphrase slice diverging from the literal slice, output diversity falling. When they move together, the diagnosis is settled and the remedy is a smaller effective training budget, a more diverse dataset, or a lighter-touch adapter - and the deployable artifact is an earlier checkpoint, not the last one. What should not be accepted as evidence either way is a good score on the training set, or on any eval set whose items resemble training items closely. Overfitting is precisely the condition under which those numbers look excellent.