For a rare-positive classifier missing its F1 target, do you train soft-F1 or tune the threshold?

## Two levers, one metric F1 is the harmonic mean of precision and recall, computed from hard counts of true positives, false positives and false negatives. Those counts come from comparing a *thresholded* prediction to a label, so F1 depends on the model through a step function twice over: once in the counts and once in the threshold. Neither is differentiable. You have two distinct ways to attack it, and knowing which to reach for first is the substance of this question. **Lever one: leave training alone and move the cut.** Train with cross-entropy, which is a proper scoring rule — its minimiser is the true conditional probability, so it produces a well-ordered, reasonably calibrated score. Then sweep the decision threshold over a held-out split and pick the value maximising F1, or the one hitting a required precision floor. This is a one-dimensional search over a metric you can evaluate directly; no gradient needed. **Lever two: change the objective so the metric itself is differentiable.** Replace each hard count with its expectation under the model's own probabilities: - `soft_tp = sum over i of p_i * y_i` - `soft_fp = sum over i of p_i * (1 - y_i)` - `soft_fn = sum over i of (1 - p_i) * y_i` and define `soft_F1 = 2 * soft_tp / (2 * soft_tp + soft_fp + soft_fn)`, minimising `1 - soft_F1`. Every term is a smooth function of the sigmoid outputs, so gradients flow. Note that no threshold appears anywhere: the objective is threshold-free by construction, which is exactly its appeal and exactly where its problems begin. ## Why the threshold sweep is the default The 0.5 threshold is not a property of the sigmoid or of the problem — it is the value you get by accident when nobody chooses one. Under a 2% positive rate, the score distribution is squeezed hard toward zero and the F1-optimal cut frequently lands somewhere well below 0.5. A model that looks broken at the default is very often a model that has been read at the wrong point. The sweep is also cheap, auditable and reversible. It changes no weights, so you can re-run it when the class balance shifts without retraining. It keeps the probabilities interpretable for anything downstream that needs an expected value rather than a decision. And it decouples the modelling question ('does the score order examples well?') from the product question ('what precision-recall trade do we want?'), which are usually owned by different people. Three disciplines make it trustworthy. Tune on a split that is neither the training data nor the split you report on, or the reported F1 is optimistically biased by the selection. Check how many positives that split contains: with a few dozen, the argmax of the F1-versus-threshold curve is mostly noise, and you should prefer a smoothed or more conservative choice. And treat the threshold as a parameter with a shelf life — it is a function of the prevalence and score distribution at tuning time, and both drift. ## When soft-F1 earns its place There are cases the sweep does not cover. In extreme multi-label settings — thousands of labels, each with its own operating point — per-label threshold tuning is a large, noisy estimation problem in itself, and an objective that already optimises the metric shape can be the cleaner design. Similarly, when the metric is very far from what cross-entropy prioritises, the surrogate can steer the *representation*, not merely the cut point, toward the positives that matter. But price it honestly: - **Non-decomposability.** Mean cross-entropy over a batch is an unbiased estimate of mean cross-entropy over the corpus, because it is an average of per-example terms. F1 is a ratio of sums, so a batch value is a biased and noisy estimate of the corpus value — biased in a way that depends on batch size and on how many positives happen to land in the batch. Batch size becomes a semantic hyperparameter, not just a throughput knob, and a batch with no positives gives a degenerate or undefined objective. - **Calibration loss.** The soft counts reward pushing every probability toward 0 or 1, because that is what makes the soft counts match hard counts. The resulting scores are no longer usable as probabilities, which forfeits the flexibility that made the threshold sweep attractive in the first place. - **Optimisation behaviour.** The objective is non-convex and its gradient for one example depends on the whole batch's counts, which makes it noisier and more finicky early in training than a per-example loss. A common compromise is to warm up with cross-entropy and switch or blend the metric surrogate in later. ## How to answer this in a loop Say the order out loud: check the threshold before changing the loss; verify the ordering quality is actually adequate before blaming the objective; and reach for a metric-shaped surrogate only when the operating point genuinely cannot be chosen post-hoc. Then name the costs of that surrogate unprompted — the batch dependence and the ruined calibration — because that is what shows you have run it rather than read about it.