How do garbage collection and safepoint pauses distort a homegrown Java microbenchmark, and how should you account for them?

## Two sources of involuntary pauses A Java program does not have uninterrupted control of the CPU. The JVM steps in at two related mechanisms: ### Garbage collection (GC) Java manages memory automatically: objects you allocate live on the **heap**, and a **garbage collector** periodically reclaims those that are no longer reachable. Most collectors have at least some **stop-the-world (STW)** phases — moments where *all* application threads are frozen so the collector can safely scan or move objects. Even "mostly concurrent" collectors (G1, ZGC, Shenandoah) have short STW pauses. A GC pause can be microseconds to many milliseconds depending on collector and heap. ### Safepoints A **safepoint** is a point in execution where the JVM knows the precise state of every thread (where all references are, etc.), so it can safely do global work: GC, **deoptimization** (undoing a speculative JIT optimization), revoking biased locks, taking a thread dump, or applying certain `Thread` operations. To reach a safepoint the JVM asks all threads to pause at the next safe location — this is **time-to-safepoint** plus the **safepoint operation** itself. STW GC *uses* safepoints, but safepoints also happen for non-GC reasons. ## Why this distorts a microbenchmark Your benchmark wants to measure the cost of *one operation*. A GC or safepoint pause can land **in the middle of a timed iteration**, adding, say, 3 ms to an operation that really costs 50 ns. Now: - **If you report the mean**, a handful of multi-millisecond pauses across millions of iterations can dominate or visibly inflate the average — your "50 ns" operation reports as 200 ns for reasons unrelated to the code. - **If you naively discard outliers**, you might *also* be discarding the operation's *legitimate* allocation cost. If the operation itself allocates, the GC it triggers is part of its true cost — throwing it away under-reports reality. So both directions of naive handling lie: include the noise and you over-report; strip it blindly and you under-report. ## Coordinated omission lurks here too If the benchmark loop *waits* for each iteration to finish before starting the next, a long GC pause stalls the whole loop — but you only record *one* slow sample, when in a real system many requests would have queued up behind that pause and all suffered. This is **coordinated omission**: the pause's true impact on latency percentiles is massively under-counted. (It's a distinct pitfall, but GC pauses are a classic trigger.) ## How to account for it correctly 1. **Control allocation.** Minimize *incidental* allocation (autoboxing, temporary objects in the harness) so the GC you observe reflects the *operation's* real allocation, not benchmark scaffolding. 2. **Measure allocation explicitly.** Use a profiler (JMH `-prof gc`) to report allocation rate (bytes/op). An operation's allocation footprint is often more stable and informative than wall-clock for comparing implementations, because it's deterministic. 3. **Run long and representative.** Time long enough that GC happens at a realistic rate; a too-short run might see *zero* GC and under-report, or *one* and over-report. 4. **Report the distribution, not just the mean.** Percentiles (p50/p99/p999) and max separate "typical operation cost" from "perturbation." A clean p50 with a spiky p999 tells you GC/safepoints hit some iterations. 5. **Reduce safepoint noise.** Avoid triggering deopts mid-measurement (warm up the exact paths), and be aware of safepoint-bias in sampling profilers. ## Why a harness exists JMH forks **fresh JVMs** per run (so prior state doesn't leak in), provides `-prof gc` to attribute allocation, reports **percentiles** out of the box, and structures measurement so a single GC spike is visible rather than silently smeared into a mean. Reproducing all of that in hand-rolled timing is error-prone — which is exactly why you don't. ## The takeaway GC and safepoint pauses are *real* and part of how Java runs — you can't pretend they don't exist. The skill is distinguishing **"this operation's own allocation/GC cost"** (keep it) from **"a stray global pause perturbed this sample"** (account for it via distribution, not by faking a clean average).