Under domain shift, why is a network's confidence a poor filter for target pseudo-labels?

## The procedure and where the risk sits Self-training on an unlabelled target domain is simple: run the source-trained network over target data, treat high-scoring predictions as if they were labels, retrain or fine-tune including them, repeat. Everything hinges on the selection step, because a pseudo-label is indistinguishable from a real one once it enters the training set. A wrong pseudo-label is not noise that averages out; it is a supervised instruction to be wrong. ## Why the softmax number is not a probability The maximum softmax output is a monotone score, not a calibrated posterior. Trained to convergence with cross-entropy, deep classifiers push logits apart until predictions are near-saturated, so a value of 0.99 routinely corresponds to an empirical accuracy well below 99 percent. Any calibration measured on held-out **source** data does not transfer: calibration is itself a property of the input distribution, and it degrades under shift in the same direction as accuracy. So the one number the threshold reads is least trustworthy exactly where you are relying on it. ## The selection bias Even if confidence were perfectly calibrated, a global high threshold selects a biased subsample. The target examples that score highest are the ones lying closest to the source distribution — the frames that already look like the training data, the clean audio, the canonical viewpoints. Two consequences follow: 1. **Low information gain.** You retrain on the examples the model already gets right. The hard region — the part of the target that actually caused the accuracy drop — is exactly what the filter excludes. 2. **Class skew.** Confidence is not uniform across classes. Easy, frequent, visually distinctive classes clear a 0.95 bar constantly; rare or ambiguous classes almost never do. The pseudo-label set therefore misrepresents the target's class distribution, and training on it shifts the decision boundaries further toward the majority classes, which raises their confidence again next round. ## The spiral Put those together and the loop is self-reinforcing. Round one admits a confident, easy, class-skewed set including some confident errors. Training on it increases confidence in the same regions — including in the errors, which are now explicitly supervised. Round two's threshold therefore admits more of the same, plus new errors that inherited confidence from the first round's mistakes. Average target confidence climbs monotonically while target accuracy stagnates or falls. The rising-confidence curve looks like progress, which is what makes this failure so effective at surviving review. ## Selection rules that hold up better - **Per-class quotas rather than a global cut.** Take the top fixed proportion of examples *within each predicted class*. The pseudo-label set keeps a sane class balance, and rare classes contribute their best examples instead of none. - **Agreement instead of magnitude.** Require the same prediction across several augmented views of the input, or across an ensemble or across weight-averaged copies of the model. Consistency under perturbation is a far better correctness signal under shift than a single peaked softmax. - **A curriculum on the threshold.** Start strict and loosen over rounds as the model genuinely improves, rather than fixing one number for the whole run. - **Keep source data in the mix.** Continuing to train on labelled source examples anchors the model against drifting into its own fiction. - **Soft targets or loss down-weighting.** Weighting a pseudo-labelled example below a real one limits the damage a wrong one can do. - **Refresh cheap corrections first.** Adapt normalization statistics before generating the first round of pseudo-labels, so the predictions being filtered are the best free predictions available. ## Monitoring it without target labels Even with no labels you have three usable signals. Track the **class histogram of accepted pseudo-labels** across rounds — collapse toward a few classes is the spiral becoming visible. Track the **number of examples clearing the threshold**: an explosive rise means confidence inflation, not learning. Track **prediction churn**, the fraction of target examples whose predicted class changes between rounds; healthy adaptation churns then settles, a spiral freezes early. And if it is at all affordable, hand-label a few hundred target examples for evaluation only. That set never trains anything, and it is the only thing that will tell you the truth about whether the last three rounds helped.