A hot loop specialised into many generated bodies runs slower than before - what could have eaten the win?

## What specialising the loop actually removed When a generic routine is compiled into one body per type argument, each body knows its argument concretely. Two things follow inside that body: operations on the argument are direct calls with known targets rather than selections made through a run-time indirection, and those known targets can be inlined, which in turn lets the optimiser fold constants, delete branches that are dead for this argument, and keep values in registers across what used to be a call boundary. That is a real gain, and it is worth being precise about its size. Removing one indirection is worth a small, fixed number of cycles per call. Inlining is worth more, because of what it enables downstream. Both are per-call savings measured in cycles. ## Why more code can run slower The cost sits on a different axis. A processor does not execute from the artifact; it executes from a small, fixed-size instruction cache, fed by a front end that has to fetch and decode. Code that is resident in that cache runs at full speed; code that is not stalls the front end until it arrives. Specialising a loop for many arguments multiplies the amount of distinct code that does the same job. If one iteration of the loop reaches into several generated bodies, those bodies now compete for the same cache lines, and each can evict the others. The loop thrashes its own instruction supply. A stall waiting for instructions is worth far more than the few cycles an indirection cost, so the arithmetic flips: the per-call gain is paid many times over by a front-end cost the per-call reasoning never accounted for. Two details make this worse than it first looks: - Inlining does not only duplicate a call, it duplicates the callee's body into every site, so aggressive inlining inside many generated bodies multiplies footprint twice over. - The cost is invisible in a microbenchmark that exercises one argument at a time, because with one body in play the cache is never contended. It appears only on the real mix of arguments. ## Reading the symptom before believing it | What you observe | What it usually means | |---|---| | Slower loop, same number of executed instructions | the front end is stalling - fetch, not work | | Slower loop, more executed instructions | specialisation exposed a worse code path, not a cache problem | | Faster in isolation, slower in the application | the microbenchmark ran one body, the application runs many | | Slower only after the artifact grew sharply | a new type argument multiplied bodies through a composed chain | The distinction in the first two rows is the one that matters. If the instruction count is flat and the time is up, the machine is waiting on code rather than doing more of it, and that points at footprint. If the instruction count rose, specialisation is not the villain and something else changed. ## Where the crossover sits There is no fixed number of arguments at which generation stops paying. The crossover depends on how large the generated body is, how many distinct bodies one pass of the loop touches, how much the removed indirection actually cost on this shape of call, and how much cache the rest of the program is already using. That last term is why a change that measured well in a component can regress in the assembled application: the cache is shared with everything else running. ## What to do about it 1. Confirm the diagnosis with a measurement that separates front-end stalls from extra executed work, rather than assuming it. 2. Find which arguments the hot loop actually exercises, and how many distinct bodies one pass touches. 3. Keep per-argument generation for the argument that dominates the loop, and let the long tail run on one shared body; a mixed strategy is normal and is usually better than either extreme. 4. Re-measure on the real workload, with the full application resident, not on the loop alone. The general lesson is that a per-call saving and a per-program cost are measured in different units, and only a measurement on the real workload converts between them. An engineer who says more specialisation is always faster has not met this case; one who says it is always slower has over-corrected. The answer is that it depends on footprint, and footprint is observable.