A load test measures a long-lived request server during its first thirty seconds - why does that figure understate steady-state throughput?

## What warmup is **Warmup** is the interval between a process starting and its performance reaching a stable plateau. In a runtime that compiles while it runs, it is not a vague settling effect; it has specific causes that can be named and measured: - **Hot paths are still in a low tier.** The counters that trigger compilation have not yet crossed their thresholds, so the code doing the work is interpreted or only cheaply compiled. - **The profile is still thin.** Aggressive optimisation needs observations - which branches run, which types appear - and those accumulate only by executing. - **Compilation competes for CPU.** Compiler work happens inside the same process on the same cores. During the ramp, the machine is doing two jobs, and request handling gets less than all of it. - **First-execution costs are being paid.** Code is being located and prepared, lookup structures are being filled, caches at every level are cold. These are separate from compilation but land in the same window. ## Why the early figure is not the system you shipped A long-lived server spends essentially all of its life in the plateau. A number taken during the ramp describes a state the system passes through once, and it is wrong in a specific direction: it is pessimistic about steady state and it hides the shape of the curve. Two systems with identical peak throughput can have very different warmup profiles, and a single early average tells you nothing about which you have. It also mixes two effects that should be reported separately. Some of the deficit is code quality, which compilation fixes; some is compiler threads taking cores, which is a resource cost during the ramp only. Averaging them into one figure loses the distinction that would tell you what to do about it. ## Measuring honestly 1. **Drive representative traffic.** The profile is built from what actually executes, so warming with one code path and then measuring another means measuring a cold path again. Include the mix of request shapes that production sees. 2. **Warm until the curve flattens.** Track per-request time or throughput over time and wait until improvement stops - do not warm for a fixed count picked by habit. The plateau is an observation, not a constant. 3. **Discard the ramp.** Everything up to the plateau is excluded from the reported figure rather than averaged into it. 4. **Measure a long window afterwards.** Long enough that occasional recompilation, rebalancing and background work are represented rather than accidentally excluded. 5. **Repeat in fresh processes.** Compilation decisions vary between runs, so a single process gives no sense of variance. 6. **Report both numbers.** Time-to-plateau and steady-state throughput, side by side, with the shape of the ramp. ## Reading the curve | Observation | Most likely reading | |---|---| | Per-request time falls steadily, then flattens | ordinary warmup: hot paths reaching higher tiers | | Falls, flattens, then worsens and recovers | an assumption stopped holding; code fell back and was recompiled | | Never flattens across a long run | the workload keeps reaching new code, or thresholds are never crossed on the paths that matter | | Flattens immediately at a poor figure | the bottleneck is not code speed - look outside the runtime | ## When warmup is the number that matters The plateau is the right figure for a process that lives for days. It is the wrong figure whenever processes are short-lived or frequently replaced: - A job that starts, processes a batch and exits within seconds may never leave the ramp at all; its **wall-clock time including the interpreted phase** is the only meaningful measure. - A service redeployed many times a day pays the ramp on every instance, every time. - A fleet that scales out under a traffic spike adds instances precisely when it is least able to absorb their warmup, and a new instance given a full share of traffic immediately will show worse latency than its peers. A sound benchmark answers the question the deployment actually asks. For a long-lived server that is the plateau; for a short-lived job it is the total including the ramp; for an elastic fleet it is both, plus how long an instance needs before it deserves full traffic. ## The trap in the other direction Warming is not licence to measure something the system never does. If the warm-up phase runs a narrower set of paths than production, the profile that the optimising tier built is a profile of the benchmark. The measured plateau is then real for the benchmark and optimistic for production, which is the same error as the early measurement with its sign flipped.