A training loss decreases smoothly yet the model is useless — how do you check target-to-input alignment?

## The premise to reject "The loss goes down, so the pipeline is right" is the assumption that makes these bugs expensive. A loss is a function of two arrays. It has no notion of where either array came from, whether row *i* of one describes the same real-world event as row *i* of the other, or whether the shapes you handed it are the shapes you meant. Gradient descent will happily minimize a well-formed but meaningless objective and produce a smooth, convincing curve. ## Family one: silent broadcasting Suppose the network emits predictions of shape `[B,1]` — batch dimension plus a trailing size-one axis — while the targets arrive as `[B]`. Elementwise arithmetic between them does not fail. Broadcasting aligns from the trailing axis, expands both to `[B,B]`, and computes every prediction against every target: element `(i,j)` is the per-element loss of prediction *i* against target *j*. The reduced loss is then the mean over all B-squared pairs. This number is not garbage, which is what makes it dangerous. With a squared error, the value of prediction *i* that minimizes its row is the **mean of the targets in the batch**, so training pushes every output toward the batch mean and the loss settles near the variance of the targets. It decreases. It is smooth. And the model has learned a constant. Two details make it survive casual testing. A smoke test with batch size one has `[1,1]` against `[1]`, which broadcasts to `[1,1]` and is numerically correct, so the bug hides. And the magnitude of the loss is in a believable range, so nobody stops to hand-check it. The defence is to stop relying on broadcasting silently: assert the exact expected shape immediately before the loss call, and once, on a two-sample batch with numbers you chose, compute the loss on paper and compare it with what the code returns. If those two disagree, no amount of curve-watching would have told you. ## Family two: correspondence bugs Here the shapes are right and the pairing is wrong. **Row-wise misalignment** — an off-by-one index, a sort applied to the feature array but not the label array, a join that silently reordered one side — pairs each input with somebody else's target. The signature is that generalization sits at chance while the training loss still falls, because a high-capacity network can memorize an arbitrary input-to-label assignment given enough steps; it simply falls far more slowly than a learnable mapping would, and nothing carries over to held-out data. If your training loss is descending but validation never leaves the level of predicting the marginal, suspect the pairing before you suspect capacity. **Component-order misalignment** is nastier, because the run is genuinely fine. Take a robot-manipulation dataset whose action vector is stored in a different joint order than the observation assumes, so every target is a fixed permutation of the truth. The mapping from observation to permuted action is still a function, and the network learns it. Training error is low, validation error is low, every dashboard is green — and the arm moves wrongly the first time the outputs are sent to hardware, because index 0 of the emitted vector is interpreted as joint 0. No training-time metric can see this, because the training data is self-consistent. Only an end-to-end round trip of one known record, compared against the raw log **by name rather than by position**, exposes it. ## A procedure that catches all three 1. **Assert shapes** at the boundary into the loss, and hand-verify the loss value once on a tiny batch. 2. **Decode and eyeball.** Take the batch the loader actually produces — after every transform — invert the encoding, and inspect a handful of pairs against the source records. This one step catches more pipeline bugs than any amount of metric-watching. 3. **Carry an identifier.** Give every sample a stable id that travels alongside it through the pipeline, and assert after each stage that the id attached to the input still matches the id attached to the target. Positional correspondence is an assumption; an id makes it a checked fact. 4. **Run a permuted-target control.** Deliberately shuffle targets against inputs and train briefly. That gives you the loss curve of a *known* misalignment. If your real run's curve resembles it, the real run is misaligned too. 5. **Prefer names to positions.** Where a vector's components mean distinct physical things, address them by name at the pipeline boundaries and convert once, in one place, with a test. The theme underneath all of this: the loss can tell you whether an objective is being optimized. It can never tell you whether it is the objective you meant.