Why does RoIAlign produce better instance masks than RoI pooling on a detection head?

## The coordinate problem A detector proposes boxes in input-image coordinates. The features those boxes must be read from have been downsampled by the backbone — a stride of 16 means one feature cell covers a 16x16 input patch. Dividing a box's coordinates by the stride almost always lands on fractional feature positions: a box at `x = 100` with stride 16 sits at feature coordinate `6.25`. Both pooling schemes must turn that fractional region into a fixed-size grid of features, because the downstream head expects a fixed input shape. They differ in how honest they are about the fraction. ## What RoI pooling rounds RoI pooling quantises **twice**: 1. **Proposal to grid.** The box's fractional feature coordinates are rounded to integer cell indices, so the region actually pooled is not the region proposed. Error: up to half a cell per edge. 2. **Grid to bins.** The snapped region is divided into the output grid — say 7x7 — and each bin boundary is rounded to integers too, so bins end up with unequal sizes and, again, cover slightly the wrong feature cells. Error: up to another half cell. Each half-cell of error is `stride/2` input pixels: eight pixels at stride 16, and more on the coarser levels of a feature pyramid. The features handed to the head therefore describe a region offset from the box the head believes it is looking at. ## Why classification survives it and masks do not Classification asks one question about the whole region: which class. That question is largely invariant to a small translation — a cat shifted eight pixels is still a cat, and max-pooling over a bin discards much of the spatial detail anyway. The argmax rarely flips. A mask asks a question **at every pixel**: is this one inside the object. The predicted low-resolution mask is placed back onto the box, so a systematic misalignment between features and box coordinates translates into a systematic displacement of the whole predicted silhouette. Nothing downstream can undo it, because the head was never told the features came from somewhere else. The damage scales inversely with object size. An object 300 pixels across shifted by eight pixels loses a sliver of its border. An object 40 pixels across shifted by eight pixels loses a fifth of its extent, and the per-bin evidence for its boundary was thin to begin with. This is why alignment shows up first and worst in small-object mask scores. ## What RoIAlign does instead RoIAlign performs **no rounding at all** on the coordinates: - The box keeps its fractional feature coordinates. - The region is divided into the output grid with fractional bin boundaries, so every bin is exactly the same size. - Inside each bin, a small fixed number of sample points (four is a common choice, placed at regular positions in the bin) is chosen, again at fractional coordinates. - Each sample's value is computed by **bilinear interpolation** from the four feature cells surrounding it, weighting each by the overlapping area implied by the fractional offsets. - The samples in a bin are aggregated — averaged or maxed — to give that bin's output. Bilinear interpolation is differentiable with respect to the feature values, so gradients flow back to the backbone exactly as they did through pooling; nothing about training changes. The exact number of sample points per bin turns out to be relatively unimportant — what matters is that the coordinates are never snapped. ## Reading the symptom in practice The fingerprint of a misalignment problem is a **gap between box quality and mask quality**: boxes match ground truth well, masks do not, and the mask errors look like a consistent shift or a border that is systematically off rather than random noise. Note that mask overlap is a much harsher test than box overlap for slender shapes — a perfectly placed rectangle around a bicycle encloses mostly background, so the same object can score high on box overlap and low on mask overlap even before alignment enters the picture. That makes mask-side metrics the sensitive place to look when you are hunting for feature misalignment. ## Common misconceptions - *"RoIAlign is a bigger pooling window."* The window is the same; only the sampling is exact. - *"RoI pooling rounds once."* Two roundings, and the second is easy to forget. - *"It matters most for large objects because they cover more bins."* The opposite: a fixed pixel offset is a larger fraction of a small object. - *"Interpolation blocks gradients."* Bilinear sampling is a differentiable weighted sum.