What does model FLOPs utilization (MFU) measure for a transformer training run, and how do you compute it?
answer
- useful arithmetic over available arithmetic
- dense peak, not sparse
- about six FLOPs per parameter per token
- forward two, backward four
- recomputed work does not count
basics
~20 sMFU is the share of the GPUs' dense peak FLOP rate spent on the model's own arithmetic: tokens per second times about 6 x parameters (plus an attention term), divided by GPU count times per-GPU peak.
solid answer
~40 sMFU = achieved model FLOP/s ÷ (number of GPUs × per-GPU dense peak). For a decoder transformer with `N` parameters, one trained token costs about `6N` FLOPs — `2N` in the forward pass and `4N` in the backward — plus an attention term of roughly `12 · layers · d_model · seq_len` that grows with context. Multiply by measured tokens per second and divide by the cluster's dense peak for the training precision. It counts only the arithmetic the model needs, so recomputed activations are excluded; a figure that includes them is hardware FLOPs utilization (HFU). It is also not the busy percentage `nvidia-smi` shows, which only says some kernel was running. A 7B model at 48,000 tokens/s on eight GPUs of an illustrative 1,000 TFLOP/s peak is at about 25%.
go deeper
Recall the definition: the model's FLOPs per second divided by the GPUs' combined dense peak, with about six FLOPs per parameter per trained token.
Explain where 6N comes from — 2N forward, 4N backward — when the attention term matters, and why recomputed work is left out of the numerator.
Compute MFU from a measured tokens-per-second figure, then turn a low value into a checklist: idle GPU gaps, exposed communication, small unfused kernels, recompute share.
Use MFU to judge whether a throughput target is credible before committing hardware, and require every team to quote it with the same FLOP convention and peak.
## What MFU measures **Model FLOPs utilization (MFU)** is the share of the hardware's peak arithmetic rate that the model's own mathematics actually uses during training. It answers one question: of all the floating-point operations the GPUs could have performed in a second, how many went into the forward and backward passes the model needs? `MFU = achieved model FLOP/s ÷ (number of GPUs × per-GPU dense peak FLOP/s)` The numerator is computed from the model's size and the measured throughput, not read from a hardware counter. That is what makes MFU comparable across frameworks, parallelism layouts and memory-saving tricks: two runs of the same model at the same tokens per second on the same hardware have the same MFU, whatever each did internally. ## Counting FLOPs per trained token For a dense decoder-only transformer with `N` parameters, nearly all arithmetic is matrix multiplication against the weights. Each weight takes part in one multiply and one add per token in the forward pass. The backward pass computes two gradients — with respect to the activations and with respect to the weights — each about as costly as the forward. | Part of the step | FLOPs per token (approx.) | |---|---| | Forward pass through the weights | `2N` | | Backward pass (activation and weight gradients) | `4N` | | Attention scores and weighted sums | about `12 · layers · d_model · seq_len` | | **Total** | **`6N` plus the attention term** | The attention term grows with sequence length, so it matters for long-context training and is small for short sequences. Conventions differ — some halve it for a causal mask — so state which one you used whenever you quote a number. ## A worked example Take a 7B-parameter code model fine-tuned on one node of eight GPUs. Suppose each GPU has an illustrative dense bf16 peak of 1,000 TFLOP/s and the job processes 48,000 tokens per second across the node. 1. Model FLOPs per token: `6 × 7×10⁹ = 4.2×10¹⁰`. 2. Achieved rate: `4.2×10¹⁰ × 48,000 ≈ 2.0×10¹⁵` FLOP/s, about 2,016 TFLOP/s. 3. Available rate: `8 × 1,000 = 8,000` TFLOP/s. 4. MFU: `2,016 ÷ 8,000 ≈ 25%`. Adding the attention term for 32 layers, `d_model` of 4,096 and 4,096-token sequences (`12 × 32 × 4,096 × 4,096 ≈ 6.4×10⁹` FLOPs per token) lifts the count to about 48.4 GFLOP per token and MFU to about 29%. Either figure says most of the node's arithmetic capacity is going unused. ## What MFU is not - **Not hardware FLOPs utilization (HFU).** If activation checkpointing recomputes the forward pass during backward, the GPUs execute roughly an extra `2N` per token. HFU counts that work; MFU deliberately does not, because the model did not need it. A run can hold or raise its HFU while its MFU and its tokens per second fall. - **Not the busy percentage from `nvidia-smi`.** That figure reflects how much of a sampling window some kernel was running, not how much arithmetic it did. A GPU running small, memory-bound kernels back to back can look fully busy at low MFU. - **Not measured against the sparse peak.** Spec sheets often list a higher figure that assumes structured sparsity; dense training is compared with the dense peak for the precision it trains in. ## Checking the estimate against the step PyTorch can count what a step executes. `torch.utils.flop_counter.FlopCounterMode` intercepts the operators run inside its context and applies per-operator formulas; `get_total_flops()` returns the sum. Two caveats apply: it counts every operator that runs, so recomputation shows up and the result leans towards HFU, and operators without a formula contribute zero. `torch.profiler.profile(with_flops=True)` similarly estimates FLOPs for matrix multiplications and 2D convolutions inside a trace. ## When MFU is low A low MFU is a symptom, not a diagnosis. The usual first checks, roughly in the order they pay off: 1. **Is the GPU idle between kernels?** Gaps on the GPU stream in a profiler trace point to host-side stalls — a synchronizing `.item()`, waits for input — rather than slow kernels. 2. **Is communication exposed?** Collective kernels that do not overlap with compute add step time and no model FLOPs. 3. **Are kernels small and unfused?** Long runs of short elementwise kernels and a per-parameter optimizer loop keep the GPU busy at low arithmetic throughput; a compiler and a fused optimizer address that. 4. **How much is recomputation?** Checkpointing every block adds about `2N` per token to `6N` — roughly a third more compute per step.
- The same run turns on activation checkpointing for every block. What happens to MFU and to hardware FLOPs utilization?Recomputing the forward pass adds roughly `2N` FLOPs per token on top of `6N`, so the GPUs do about a third more work per step. Hardware FLOPs utilization counts that work, so it can hold steady or rise. MFU counts only the `6N` the model needs; as step time grows and tokens per second fall, MFU falls.
- How can you check the 6N estimate against what a PyTorch step actually executes?Run one step inside `torch.utils.flop_counter.FlopCounterMode` and read `get_total_flops()`, or profile with `with_flops=True`. Both apply per-operator formulas. FlopCounterMode counts every operator that runs, so recomputed activations land in its total, and operators without a formula count as zero — use it to cross-check the estimate, not to replace it.
- MFU sits near 25% on the node. What do you check first?Profile a few steady-state steps and read the GPU stream. Gaps mean host-side stalls such as a per-step `.item()` or input waits. No gaps but thousands of tiny kernels means launch-bound work: unfused elementwise ops or a per-parameter optimizer loop. Collective kernels that do not overlap compute mean exposed communication. Then account for recomputation, which costs time without adding model FLOPs.
- MFU is the GPU utilization percentage that nvidia-smi reports.
- Recomputed forward passes count towards model FLOPs.
- Divide by the structured-sparsity peak from the spec sheet.
- Training costs about 2 x parameters FLOPs per token, the same as inference.
- The attention term never matters, whatever the sequence length.