What is arithmetic intensity, and how does it decide whether a GPU operation is compute- or bandwidth-bound?

## The quantity Every operation a GPU runs must pull its inputs from the device's off-chip memory onto the chip and push its results back. **Arithmetic intensity** is the ratio between the useful arithmetic and that traffic: floating-point operations performed per byte moved. It is a property of *an operation at a given shape and layout*, not of the hardware — the hardware supplies the yardstick you compare it against. ## The roofline and its ridge point A device has two ceilings: a peak arithmetic rate in FLOP/s and a peak memory bandwidth in bytes/s. If an operation streams its data once, the best it can do is ``` attainable FLOP/s = min(peak FLOP/s, intensity * peak bandwidth) ``` Plot that against intensity and you get the roofline: a sloped section where bandwidth is the limit, then a flat section where the arithmetic units are the limit. The corner between them — the **ridge point** — sits at `peak FLOP/s / peak bandwidth`, expressed in FLOPs per byte. Anything to the left of it is **bandwidth-bound**; anything to the right is **compute-bound**. The practical fact that makes this topic an interview staple is where that ridge sits: on current accelerators it is in the tens of FLOPs per byte, because arithmetic throughput has grown far faster across hardware generations than memory bandwidth has. A great many deep-learning operations therefore land on the bandwidth side, and the flat roof is irrelevant to them. ## Why dense matmul is on the compute side For `C = A * B` with square n x n matrices, the arithmetic is `2*n^3` FLOPs — one multiply and one add per accumulation step. The ideal traffic is reading A, reading B and writing C: `3*n^2` elements. So ``` intensity = 2*n^3 / (3*n^2 * bytes_per_element) = 2*n / (3 * bytes_per_element) ``` At four bytes per element and n = 4096 that is about 680 FLOPs per byte — an order of magnitude past any realistic ridge point. Two things matter here. First, intensity **grows with n**: the larger the problem, the more times each loaded element is reused. Second, the `3*n^2` figure assumes an implementation that loads blocks into fast on-chip memory and reuses them; a naive version that re-reads a row from device memory for every output element has intensity near 1 and runs nowhere near peak. It is the *reuse pattern*, not the raw FLOP count, that makes dense matmul suit massively parallel hardware. ## Why elementwise, normalization and softmax are on the bandwidth side Take `out = a*x + y`: per element that is 2 FLOPs against 12 bytes at four bytes per element (read x, read y, write out), so intensity is about 0.17. A pointwise activation is similar — one element in, one out, eight bytes, a handful of FLOPs. A normalization layer computes a mean and a variance across a feature vector and then scales and shifts each element: a small constant of arithmetic per element, still well under one FLOP per byte. A batched softmax over a large vocabulary is the same story read at scale — per logit you subtract a maximum, exponentiate, accumulate into a sum and divide, which is a few FLOPs against reading and writing several bytes, and an implementation that makes several passes moves the whole tensor several times. All of these are limited by how fast the memory system can stream the tensor past the chip. ## What follows from knowing the side - **The two sides answer to different investments.** Raising the peak FLOP rating lifts the flat roof and pushes the ridge point to the right: compute-bound work speeds up, bandwidth-bound work does not move at all, and some previously compute-bound work quietly becomes bandwidth-bound. - **For bandwidth-bound work, the only lever is moving fewer bytes** — merging consecutive passes so an intermediate never goes back to device memory, keeping a tile resident on-chip, avoiding gratuitous copies and layout changes. - **For compute-bound work, the lever is keeping the arithmetic units fed** — tiles large enough to amortise the loads, shapes that fill the units, enough parallel work to hide latency. - **Shape changes the answer.** The same weight matrix multiplied by a single input vector instead of a batch degenerates to a matrix-vector product: each weight is read once and used in two FLOPs, giving roughly 0.5 FLOPs per byte at four bytes per element. Same layer, other side of the ridge. ## Measure rather than guess Time the operation, divide its FLOP count by the time to get achieved FLOP/s, divide its byte count by the time to get achieved bytes/s, and compare each with the device's ceilings. Whichever is close to its ceiling is the binding one. If neither is close, the limit is elsewhere: not enough parallel work, per-launch overheads on tiny operations, or a device sitting idle waiting for input. ## The classic wrong conclusions Treating a lower FLOP count as automatically faster; assuming a GPU is always compute-limited because it has thousands of arithmetic units; counting only the model's weights as "memory" and forgetting the activation traffic that dominates it; and expecting one more powerful device to speed up every part of a step by the same factor.