Should label smoothing be on for a shared backbone serving both classification and retrieval?

## The thing people miss Label smoothing is usually introduced as a change to the reported probabilities. It is more than that: it changes the objective, so it changes the function that gets learned, including the hidden representation. That matters the moment more than one consumer reads the model — for example a trunk whose final classification layer serves a labelling product, while its penultimate activations are indexed and served as an embedding for nearest-neighbour retrieval. ## What smoothing does to the penultimate geometry With a hard one-hot target, the loss rewards pushing the correct logit as far above the others as possible, and it stays rewarding forever. The way a network achieves that is to keep extending each example's penultimate activation along its class direction. The spread is large, and crucially the *relative* positions carry information: examples of confusable classes end up near each other, because the network never needed to fully separate two things that look alike in order to keep reducing the loss on the easy ones. With a smoothed target the optimum is a fixed, finite logit gap that is the same for every wrong class. Reaching it, and not exceeding it, is best served by placing every example of a class at a tight cluster that is equally distant from all the other class templates. This has been reported directly by visualising penultimate activations: smoothing produces tighter, more uniformly separated clusters. Better cluster structure sounds like an improvement, and for the classification decision it is. But equidistant clusters are precisely a representation in which the similarity between classes has been flattened away — the geometry no longer says that two bird species are near-neighbours and a third is far. ## The concrete failure A fine-grained bird-species head trained with smoothing can post a small top-1 gain while recall at k on the same penultimate features falls. The retrieval consumer's complaint looks mysterious from the classification team's dashboard, because nothing there regressed. This is the whole reason to treat it as a cross-team decision: the metric that moves down is not the metric that owns the training run. ## How to make the call **Measure both, across epsilon.** Run a sweep and plot classification accuracy, calibration error and the retrieval metric together on the same held-out data. They do not peak at the same epsilon. Choosing on accuracy alone is not a defensible decision when a second consumer exists. **Look at the geometry, not just the metric.** Within-class versus between-class distance in the penultimate space, measured per epsilon, tells you whether the degradation is the mechanism described above or something unrelated, such as an index or preprocessing mismatch. **Then choose, in ascending order of cost.** - *Smaller epsilon.* Often the two metrics are not in hard conflict, and a value in the low range keeps most of the accuracy gain with less geometric collapse. - *Epsilon zero, plus a different fix for confidence.* If the reason smoothing was on was probability quality, that requirement can often be met after training instead, leaving the representation alone. - *Smoothed loss on the classification head only.* This helps at the margin but does not insulate the trunk: the head's gradient flows into the shared layers, so the geometric pressure is reduced, not removed. Claiming otherwise in a design review is a mistake an interviewer will notice. - *Two models.* The clean answer, and the expensive one. It buys independent objectives at the cost of a second training run, a second deployment and a drift problem between them. ## The organisational half The technical content here is a paragraph; the judgment is the rest. A shared backbone is a shared interface, and a change to its training objective is a breaking change to every consumer even though no signature moved. Two things are worth owning as a lead. First, the shared trunk needs a regression suite that includes the *downstream* consumers' metrics, not only the training task's, or this class of failure will always be found by the other team in production. Second, someone has to be allowed to decide when the metrics conflict — the default of letting whoever owns the training script pick epsilon is how a retrieval product silently loses recall to another team's accuracy win.