Five checkpoints from one run: ensemble their predictions or collapse them into one averaged weight vector?

## The setup A snapshot ensemble spends one training budget and saves the model at several points along the way - for instance five checkpoints harvested at the ends of successive cycles of a rate that is repeatedly raised again. The appeal is that the checkpoints are free: no additional training runs, no additional data. What is not free is what you do with them at serving time, and that is the decision. ## Option A - average the predictions Keep all five parameter sets. At inference, run all five and average their output distributions. - **Accuracy:** normally the best of the options. It works because the snapshots make partly different errors; averaging cancels the uncorrelated part. The gain scales with how much they disagree. - **Cost:** five parameter sets resident in memory, five forward passes per request. On a throughput-bound service you can sometimes batch across the five and hide part of the cost; on a latency-bound service you cannot hide it unless you have spare parallel capacity. - **Operational weight:** five artifacts to version, load, warm and roll back together. Any one of them going missing changes the model silently. ## Option B - collapse into one weight vector Average the five parameter vectors into one and serve that. - **Accuracy:** typically better than the single best checkpoint, typically worse than the prediction ensemble. It cancels the optimization noise but cannot exploit functional diversity, because there is only one function left. - **Cost:** identical to the original single model - one artifact, one forward pass, no memory increase. That is the entire argument for it. - **Preconditions:** the checkpoints must lie in one low-loss region connected by straight lines, which holds for checkpoints from one run after the trajectory has settled but not in general; and any normalization running statistics must be re-estimated with one forward-only pass over training data afterwards, or the collapsed model can score near chance. ## How to decide 1. **Start from the serving constraint, not the leaderboard.** If the request budget has no room for four extra forward passes and you cannot run them in parallel, option A is not on the table and the discussion is over. State this before anyone runs an experiment. 2. **Measure three numbers, not two.** The single best checkpoint, the collapsed average, and the prediction ensemble - on the same held-out data, through the same serving path. 3. **Measure the noise floor too.** Retrain with two or three seeds and look at the spread of the single-model score. Any gain smaller than that spread is not a result. This is the step most often skipped, and it is why many reported ensembling gains evaporate. 4. **Check diversity before you pay for it.** If the snapshots agree on almost every input, the ensemble cannot help much. The disagreement rate between snapshot pairs is a one-line diagnostic and predicts the ensemble gain better than intuition does. Snapshots taken after the rate has been raised again diverge far more than consecutive epochs do. 5. **Price the whole cost, not just compute.** Five artifacts mean more failure modes, more warm-up, a bigger image, slower deploys and a rollback story with five moving parts. A one-artifact model that is slightly worse is often the better engineering decision. ## The third option If the ensemble is genuinely better and the budget genuinely cannot absorb it, train a single student model to match the ensemble's output distribution. It costs an extra training run but yields one artifact at one forward pass, often recovering a good share of the ensemble's advantage. Whether that extra run is worth it depends on how long the model will be in production - a model serving for a year amortizes a distillation run trivially; a weekly-retrained model may not. ## What to say in an interview The answer an interviewer wants is not 'ensemble, it is more accurate'. It is: the training cost is identical so the decision is a serving-cost decision; here is what each option costs in latency, memory and operational surface; here is the measurement plan including the seed-variance baseline; and here is the fallback if the accurate option does not fit. Then state a default - collapse to one vector unless the measured ensemble gain clearly exceeds the noise floor and the budget has room - and say what evidence would change it.