How do you choose the number of epochs for a small fine-tuning dataset?

## Epochs are a convenience; steps are the currency An epoch is one pass over the training set. A step is one optimizer update. They relate through the effective batch: steps_per_epoch = ceil(examples / effective_batch) This matters because everything else in the run is defined over steps: the warmup fraction, the decay curve, the evaluation cadence, checkpoint saving. Two runs at "3 epochs" with different batch sizes are different-length runs. When comparing configurations, fix the step count; when sizing a job, convert epochs to steps first and check that the number is large enough for the schedule to make sense — a schedule with 3% warmup over 40 total steps is barely a schedule at all. ## Why the answer is small on a small dataset Fine-tuning starts from a model that already writes fluent language. What the dataset supplies is form: a voice, a format, a decision policy. That signal is absorbed quickly. Beyond a few passes, additional epochs stop teaching the pattern and start teaching the specific rows — the model reproduces training examples, hallucinates specifics it saw in them, and loses the ability to generalise to inputs the set did not contain. A concrete shape from a 3,000-example sports-commentary style-transfer run: - **1 epoch** — output drifts between the target commentary voice and the base model's neutral register; the pattern is present but not reliable. - **3 epochs** — the voice holds across inputs; held-out loss is at its minimum. - **8 epochs** — training loss keeps falling while held-out loss rises; the model emits player names and phrasings from the training rows into unrelated inputs. That divergence between training and held-out loss is the signal that decides the run — not a fixed epoch number carried over from another project. ## Early stopping in practice Early stopping is a stopping *rule*, and it needs four decisions: 1. **The signal.** Held-out loss on a split never trained on is the default because it is cheap and computed every evaluation. Task-specific scoring is better aligned but slower, so it usually runs at a coarser cadence. 2. **The cadence.** Evaluate every N steps, where N is small enough to catch the turn but large enough not to dominate the run — commonly a few dozen steps on a short fine-tune. 3. **Patience.** Stop after some number of consecutive evaluations without improvement, rather than at the first uptick, because the curve is noisy at small validation-set sizes. 4. **Checkpoint selection.** Keep the checkpoint with the best held-out score, not the last one written. This is the step people skip, and it wastes the entire mechanism. One interaction to state explicitly: if the schedule was planned for 1,000 steps and you stop at 400, you stop while the learning rate is still high and the checkpoint is coarser than a run planned to be 400 steps. Prefer planning the run at the length you expect and treating early stopping as a safety net. ## Weight decay and other mild levers Weight decay shrinks weights slightly at every step, discouraging large parameter values. In fine-tuning it is a minor knob: typical values are 0 to 0.1, decoupled from the gradient update in AdamW-style optimizers, and conventionally excluded from biases and normalization parameters. It will not rescue a run that is training for eight epochs on 3,000 rows — reducing epochs is the first-order fix, and decay is a second-order refinement. Adapter training brings its own regularization: with far fewer trainable parameters, the adapter has less capacity to memorise, so adapter runs are somewhat more forgiving at the top of the epoch range than full-parameter runs on the same data. It is not immunity — an adapter trained long enough on a small set still memorises — but it shifts the practical ceiling. ## The interview answer Give the range, give the reason, and give the arbiter. The range is 1–3 epochs for a small curated dataset. The reason is that fine-tuning teaches form quickly and then starts teaching rows. The arbiter is the held-out curve with a patience rule and best-checkpoint selection, converted to steps so it lines up with the schedule. Candidates who name a fixed epoch count with no measurement behind it are the ones who get followed up on.