Your model sees a 2-billion-token stream once — how do you plan and report the run without epochs?

## Why the epoch stops working An epoch is defined as one complete pass over the training set. That definition is useful only when the number of passes is greater than one and is a design choice. On a 2-billion-token stream consumed exactly once, the epoch count is 1 at the very end and 0 before that — it carries no information about progress, cannot be used to schedule anything, and cannot compare two runs. This is not an exotic regime; it is the default for large-corpus training, and interviewers use it to check whether a candidate treats the epoch as a fundamental unit or as the bookkeeping convenience it is. ## The units that replace it **Optimizer steps.** The loop still does exactly what it always did: take a batch, forward, loss, backward, update. The step counter is monotone, it is what every schedule can be written against, and it is what the run's budget is denominated in. **Tokens (or examples) consumed.** `tokens_seen = steps x tokens_per_batch`. This is the unit that describes how much *data* the model has been exposed to, independent of how it was chopped into batches. You need both, because they answer different questions. Steps tell you how many times the parameters moved; tokens tell you how much evidence produced those movements. Two runs at the same step count but different batch sizes consumed different amounts of data; two runs at the same token count but different batch sizes took different numbers of updates. Reporting only one of the two makes runs incomparable — this is the single most common reporting error in this regime. ## What has to be decided up front - **A total step budget.** Not a nice-to-have: the learning-rate schedule is defined over the whole budget, so the budget must be known before the first step. Changing your mind about the length mid-run leaves you with a schedule that does not land where intended. - **Evaluation cadence.** With no epoch boundary to hang it on, validation runs every N steps against a **fixed held-out shard** that the training stream never yields. Choosing N is a real tradeoff: too frequent and evaluation dominates the run's cost, too rare and you learn about a problem tens of thousands of steps late. - **Checkpoint cadence**, likewise in steps, sized so that the work lost to a crash is acceptable. - **Data ordering.** "Reshuffle before every epoch" has no meaning here. Randomness comes from shuffling the order of shards and from a shuffle buffer that mixes examples locally as the stream is read. The property you want — that consecutive batches are not correlated by how the corpus happened to be written — still has to be engineered, just differently. ## Resuming is a first-class design problem A long single-pass run *will* be interrupted. A checkpoint that stores only parameters is not resumable in any meaningful sense: restarting the stream from the beginning re-shows data the model has already been updated on and guarantees the tail of the corpus is never reached. A correct checkpoint stores parameters, optimizer state, the step counter, and the **position in the data stream**, plus enough of the shuffling state that the resumed order is the intended one. Teams discover this after the first crash, and the tell in an interview is whether you list the stream position without prompting. ## A genuine property of the single-pass regime Within one pass, every batch's loss is computed in the forward pass *before* that batch has ever contributed to an update. So the running training loss is measured on data the parameters have not fitted — it is close to a held-out measurement, and the usual train-versus-held-out gap is largely absent. This is a real and useful property: it means the training curve itself is informative about generalisation while the data lasts. It stops being true the moment any data repeats, which is exactly why the distinction between a single-pass run and a multi-epoch run is worth stating explicitly rather than treating the two as the same loop with a different counter. ## What you actually report A defensible run summary in this regime names: total steps taken, tokens consumed, batch size in tokens, the fraction of the corpus consumed, the held-out metric and the step at which it was measured, and the checkpoint that produced it. An epoch count either does not appear or appears as a fraction below one, which usually confuses more than it informs. ## The principal-level judgment The strategic call is the reporting convention itself. If your team reports epochs, then the moment someone doubles a batch size or swaps a corpus, the historical numbers stop meaning anything and nobody notices for a quarter. Standardising on steps and tokens costs nothing and keeps every run in the org comparable — including runs that pass over data once, runs that pass over it many times, and runs on corpora of different sizes. Own that convention, and require the stream position in every checkpoint contract.