After a code change, your second performance trace of the same page load is 400 ms faster than the first. What could make that comparison wrong, and how would you establish that your change actually caused the improvement?

## Why one pair of traces is not evidence A performance recording is a sample from a noisy distribution, not a reading from a ruler. Load two identical builds of the same page twice and the totals will differ — sometimes by tens of milliseconds, sometimes by hundreds. If your change is worth 400 ms and typical noise is worth 300 ms, a single before/after comparison genuinely cannot distinguish "the fix worked" from "the second run got lucky". Senior engineers get caught by this constantly, because the result confirms what they hoped and confirmation feels like proof. ## Where the noise comes from - **Connection and server state.** The first request of a session pays for connection setup; a later one may not. Backend response time varies with its own load, warm caches and neighbours. - **Cache warmth on the client.** Anything already cached — resources, compiled script, DNS answers — makes a later run cheaper for reasons that have nothing to do with your diff. - **Machine state.** Other applications, background tabs, indexing, a build running in another terminal, and thermal throttling on a laptop all steal CPU from the recording. - **The page's own variability.** Different data, a different experiment bucket, a different ad, or a conditional code path can make two runs of "the same page" do genuinely different work. - **The profiler itself.** Recording adds overhead, and that overhead is not perfectly constant. ## Designing a comparison that survives scrutiny Hold everything constant except the change: - The **same build type** (production against production), served the same way. - The **same throttling settings** and the **same cache state** for every run. - The **same data and the same scenario**, performed identically each time. - A **quiet machine**, ideally a clean browser profile without extensions. Then run each build several times and compare medians rather than single values — a median resists the one anomalous run that a mean would absorb. Crucially, **interleave** the runs. Measuring all of build A first and all of build B afterwards confounds your change with everything that drifted in between: server warmth, machine temperature, a colleague's deploy. Alternating A, B, A, B distributes that drift across both arms. And take the magnitude seriously. If the spread within a single build overlaps the difference between builds, the honest conclusion is "no measurable effect", not "probably a small win". ## The mechanistic check, which is stronger than the aggregate Statistics tell you *whether* something changed; the trace tells you *what*. This is the step that distinguishes a real verification from a hopeful one: open the after-trace and look for the specific thing you set out to remove. - You deferred a script — is its evaluation block gone from the critical span, or merely moved somewhere it still blocks? - You removed a repeated computation — has that function's aggregated self time collapsed? - You made a resource load earlier — does its request bar now start earlier in the trace? If the total improved but the targeted cost is unchanged, you have learned that something else moved, and shipping the change on that evidence means shipping complexity that buys nothing. Conversely, if the targeted cost is clearly gone but the total barely moved, you have learned something equally valuable: that cost was not on the critical path, and the fix — however correct — was aimed at the wrong thing. ## When the number improves and the experience does not A change can remove real work and leave the user-visible moment exactly where it was, because the work was running in parallel with something slower, or after the moment that mattered. Always tie the verification back to the number you named at the start of the investigation, and if possible to the moment a user would perceive: when did the content actually appear, when did the click actually respond. Total scripting time is a diagnostic quantity, not a user experience. ## Confirm it survives the trip to production A local verification is necessary and not sufficient. The deployed build is compiled, minified, compressed and served from different infrastructure, and any of those can change the picture — a lazily-loaded chunk that behaved locally may be fetched over a slow connection in production. Re-check after deployment, with the same discipline: same scenario, same conditions, and look for the mechanism, not just the total. ## Knowing when to stop Verification has diminishing returns too. For a change with a large, clearly visible mechanistic effect, a handful of interleaved runs is plenty. For a marginal change where the effect sits inside the noise, the right answer is usually to drop it: an optimization you cannot demonstrate is complexity you cannot justify to the next engineer who reads the code.