Why does a Keras function metric report the wrong F1 for an epoch?

## Two kinds of thing in the metrics list `compile(metrics=[...])` accepts either a callable of the form `fn(y_true, y_pred)` returning a per-sample or per-batch value, or an instance of `keras.metrics.Metric`. They are handled very differently, and the difference is invisible in the code that passes them. A plain function is wrapped in `keras.metrics.MeanMetricWrapper`. Its state is a running sum and a count, so the number displayed for the epoch is the **mean over batches** of whatever your function returned. A `Metric` subclass carries its own variables. Keras calls `update_state(y_true, y_pred, sample_weight=None)` on every batch, `result()` whenever a value is needed for display, and `reset_state()` at each epoch boundary within `fit()` and at the start of each `evaluate()` call. ## Why averaging breaks ratios F1 is `2TP / (2TP + FP + FN)` — a ratio of counts accumulated over a *set* of samples. Averaging per-batch F1 values computes `mean(ratio_i)` instead of `ratio(sum of counts)`, and those are different numbers. Jensen's inequality is the formal statement; the intuition is easier: - With a rare positive class, most batches contain **zero** positives. F1 on such a batch is 0 (or undefined and reported as 0). Those zeros drag the mean down no matter how well the model does on the batches that do contain positives. - Conversely a batch with one positive that happens to be caught yields F1 = 1.0, a wildly over-confident contribution. - The distortion grows as batches get smaller and as the class gets rarer — exactly the regime where you cared about F1 in the first place. The same argument applies to precision, recall, AUC, and any metric involving a division or a global ranking. It does **not** apply to accuracy or MAE, which really are means over samples: the mean of per-batch means equals the overall mean when batches are the same size. ## What a stateful metric does instead A correct F1 keeps true positives, false positives and false negatives as variables, adds each batch's counts into them, and divides only when `result()` is called. That yields the epoch-level ratio because the division happens once, over the totals. `keras.metrics.F1Score`, `keras.metrics.Precision`, `keras.metrics.Recall` and `keras.metrics.AUC` are all built this way, which is why the fix is usually to pass the built-in object rather than write anything. A custom stateful metric follows the same three-method shape: - `update_state(y_true, y_pred, sample_weight=None)` — accumulate into variables; no division here. - `result()` — compute the reportable number from those variables. - `reset_state()` — zero them; Keras calls this between epochs and before `evaluate()`. Write the internals with `keras.ops` so the metric runs unchanged on the TensorFlow, JAX and PyTorch backends — `keras.ops` is Keras 3's backend-agnostic numerics layer, and a metric written against raw `tf` calls pins your model to one backend. ## Statefulness explains other confusions too Once you see metrics as accumulators, several familiar oddities stop being mysteries: - **The progress bar's training number is an epoch-so-far average.** It includes early batches computed with worse weights, which is one reason it looks worse than the validation number measured once at the end with final weights. - **`evaluate()` on the training data does not match the last training number.** Different weights, dropout off, and a fresh accumulation over the whole set instead of a running mean. - **Re-calling `compile()` resets metric state**, because it builds new metric objects. - **Thresholded metrics need probabilities.** `Precision` and `Recall` compare against a threshold (0.5 by default), so they are meaningless on a logits output. `keras.metrics.AUC` exposes `from_logits` for that reason. A model can be training perfectly while its thresholded metrics are nonsense. ## Diagnosing it in the wild The symptom is an F1 or precision reported by `fit()` that does not match what you compute yourself over the full predictions after training. People usually blame the split or the threshold; the cause is the wrapper. Two checks settle it in a minute: run `evaluate()` with a single batch covering the whole set — if the number now matches your offline computation, batch averaging was the culprit — and check whether you passed a function or a `Metric` object. With a large batch size and a balanced class the gap can be small enough to hide, which is exactly why it survives into production dashboards.