A run trains cleanly but validation is far below expectation — how do you decide: data bug or model bug?

## Framing the question The run finished. No exception, no NaN, a loss curve that looks reasonable — and a validation number far below what the task should support. The bug lives in exactly one of three places, and they cost very different amounts to check: 1. **The evaluation path** — the model is fine, the number is measured wrong. 2. **The data that reaches the loss** — the model is fine, it is being taught the wrong thing. 3. **The model or the optimization** — architecture, capacity, learning rate, loss weighting. Engineers reach for (3) first because it is the interesting one. It is also the most expensive to explore and the least often responsible. Work the list in order. ## Step 1: is the measurement stable? Load one saved checkpoint and score the validation set twice, changing nothing. If the two numbers differ, your evaluation path is stochastic and you cannot trust any comparison you have made all week. The usual culprit is regularization left in its training behaviour. Dropout during training zeroes a random subset of units and rescales the survivors so the expected activation is preserved; at evaluation the units are all kept and nothing is rescaled, which computes an approximation to the average over that ensemble of thinned subnetworks. Leave it sampling at evaluation and you are scoring one random thinned subnetwork instead — a few points of accuracy worse, and different every run. This looks *precisely* like a generalization gap: validation below training from the very first epoch, by a stable-ish margin. It is not a generalization gap. It is a measurement bug. The same step catches random crops, flips or colour jitter left in the evaluation transform, and any metric accumulated in a nondeterministic order with numerically sensitive reduction. ## Step 2: separate the harness from generalization Now score a slice of the **training** data through the **evaluation** code path — the same transforms, the same batching, the same metric function. Two outcomes, two different worlds: - Training data scores well through the evaluation path, held-out data scores badly: the harness works and you have a genuine generalization or distribution problem. Now questions about the validation set's labels, its distribution, and about overfitting become legitimate. - Training data also scores badly: the model cannot even reproduce what it was trained on when read through this path. The bug is in the path or in what was loaded — not in generalization at all. This single comparison eliminates the largest class of wrong hypotheses. ## Step 3: look at what actually reaches the loss Most teams never do this, and it is the highest-yield step. Take a batch **after every transform in the pipeline**, invert the encoding, and inspect it as a human would: the images, the text, the numbers, and the labels beside them. What to look for: - **Label histogram.** One or two classes per batch says the order is wrong; a distribution unlike the dataset's says the sampling is wrong. - **Value ranges.** Inputs normalized twice, or not at all, or clipped by an unexpected cast. - **Label-destroying augmentation.** An augmentation is an assertion that the label is invariant to a transformation. When the task is *defined* by the symmetry you are augmenting over, the assertion is false. A horizontal flip on a left-versus-right lane-change classifier turns a left example into a right example while keeping the old label, so the training set contains directly contradictory pairs; the achievable loss floors well above zero and accuracy on the affected distinction sits near chance no matter what you do to the model. The same trap catches vertical flips on digit or arrow recognition and aggressive crops that remove the object being labelled. - **Shapes and dtypes** at the boundary into the loss. ## Step 4: only now, the model With the measurement stable, the harness cleared and the batch inspected, model-side hypotheses are worth spending on: capacity, learning rate and schedule, initialization, loss weighting across terms, the objective itself. ## The discriminating heuristic Data bugs impose a **floor** the model cannot cross, because the information needed is absent or contradictory. Model bugs respond to model knobs. So the sharpest single signal is this: run two variants that *should* behave differently — a much larger model, a very different learning rate — and see whether the result moves. If a ten-times-larger model and a ten-times-smaller learning rate both land on the same disappointing number, stop touching the model. Something upstream is capping you, and no architecture will lift it. A related smell is a validation metric that is suspiciously *stable* across configurations that should have separated. Identical results are rarely a coincidence; they usually mean the thing you changed never reached the computation. ## Make it standing practice Every check above is cheap enough to automate: an assertion that evaluation is deterministic given a fixed checkpoint, an assertion on batch label diversity, a stored decoded sample from the first batch of every run, saved next to the metrics. Teams that write these down stop rediscovering the same three bugs, and the checklist is what turns a two-day investigation into a ten-minute one.