A segmentation model reports 95% pixel accuracy but 42 mIoU — what explains the gap?

## The two metrics **Pixel accuracy** is the simplest thing you can compute: count pixels whose predicted label equals the ground-truth label, divide by the number of labelled pixels. It is a single number over the whole dataset, and every pixel votes equally. That last property is the problem: pixels are not distributed equally across classes. **Intersection over union** for one class `c` is computed from that class's pixel counts: `IoU_c = TP_c / (TP_c + FP_c + FN_c)` where `TP_c` is pixels correctly labelled `c`, `FP_c` is pixels labelled `c` that are something else, and `FN_c` is pixels of `c` labelled as something else. Equivalently it is the area of the intersection of the predicted and true regions divided by the area of their union. **Mean IoU** averages `IoU_c` over the classes, unweighted. ## Why the numbers diverge so far Consider a street scene with roughly 19 classes. Road might be 35 percent of all pixels, building and vegetation another 30 percent between them, sky maybe 4 percent, and at the other end a pole class at 0.3 percent, a traffic-light class lower still. For **accuracy**, the arithmetic is a pixel-weighted average. Get road, building, vegetation, sky and car right and you have already accounted for well over 90 percent of pixels. Every rare class combined can contribute at most one or two points. Accuracy is therefore almost a measure of how well you segment the largest handful of classes, and 95 percent is unimpressive rather than good. For **mIoU**, the arithmetic is a class-weighted average: each class carries weight `1/19` regardless of its pixel share. If ten common classes score 0.75 and nine rare ones score 0.05, the mean is about 0.42. Every rare class the model gives up on costs roughly five points of mIoU, and none of them costs more than a fraction of a point of accuracy. On top of the weighting, IoU is a **stricter** per-class score than recall. Its denominator contains both false positives and false negatives, so a model that hedges by painting a rare class generously is penalised for the over-prediction, not rewarded for the extra recall. A thin two-pixel structure predicted three pixels wide and offset by one has poor IoU even though a human would call the prediction correct. ## Which to report Report **mIoU as the headline plus the full per-class IoU table**. The mean is what makes the metric comparable across models, but it is an average over things you care about differently. In practice the per-class column is where the decisions live: it tells you whether the failure is concentrated in two classes you could fix with data, or spread across every small class, which points at a resolution or architecture limit instead. Use pixel accuracy only as a sanity check. It is useful for spotting a catastrophic regression and almost useless for comparing two working models. ## Aggregation detail that changes the number There are two ways to compute mIoU over a dataset, and they do not agree. 1. **Accumulate globally.** Sum `TP_c`, `FP_c` and `FN_c` over every image in the dataset, then compute each class's IoU once from those totals and average over classes. This is the standard and the one you should use. 2. **Average per image.** Compute mIoU on each image, then average over images. This is unstable, because a class absent from an image and correctly not predicted gives `0/0`, which is undefined. Whatever convention you pick for that case — skip it, call it one, call it zero — changes the score substantially, and images containing only easy classes get the same weight as crowded ones. If two teams report different numbers on the same model, aggregation convention and the treatment of ignore or void labels are the first two things to check. ## What you do about a real gap - **Look at the per-class table first** and separate classes that are rare from classes that are thin. They have different fixes. - **Sample training crops** so rare classes appear more often than their natural pixel frequency, rather than accepting whatever a uniform random crop gives you. - **Reweight the per-pixel loss** by inverse class frequency so the gradient is not dominated by road and sky. - **Raise the effective output resolution** for thin classes, since a structure narrower than the decoder can resolve will never reach good IoU no matter how it is weighted. - **Check the annotation** on the failing classes. Thin-structure boundaries are exactly where human labels are least consistent, and a class can cap out below what the model is capable of. ## What weak answers get wrong - Treating the gap as a bug in the metric. It is a correct report of imbalanced performance. - Defining IoU with recall's denominator, `TP / (TP + FN)`, which ignores over-prediction entirely. - Assuming mIoU weights classes by pixel share. If it did, it would just be a variant of accuracy. - Quoting a mIoU without saying how it was aggregated or how ignore pixels were handled.