What is safepoint bias in JVM sampling profilers, and how do modern profilers like async-profiler avoid it?

## Why this exists: how the JVM pauses threads The JVM frequently needs all application threads stopped at a coordinated moment — for stop-the-world GC, deoptimization, class redefinition, and (historically) for sampling stack traces. It can't stop a thread at a truly arbitrary machine instruction, because the GC and runtime need a consistent view (which registers hold object references, etc.). So the JIT inserts **safepoints**: specific locations in compiled code where a thread *polls* a flag and, if asked, parks itself in a state the runtime fully understands. Typical safepoint locations: **method entry/return, loop back-edges, and allocation sites.** When the JVM requests a safepoint, each thread runs until it hits its *next* poll, then stops. The interval where all threads are stopped is a **safepoint** (a stop-the-world pause). ## The JVMTI sampling path and where bias creeps in The standard, portable way to sample stacks is the **JVMTI** (JVM Tool Interface) call `GetAllStackTraces` (or `GetStackTrace`). It only returns valid Java stacks when threads are at a safepoint, so under the hood it triggers a **global safepoint**, walks every thread's stack, then resumes. Two distortions result: 1. **Spatial bias (the classic 'safepoint bias').** The JIT, optimizing a tight counted loop, may **elide** the safepoint poll inside the loop (a poll on every iteration is expensive). Now suppose the timer fires while a thread is deep in that loop. The thread can't stop *there* — it must run on until it reaches the next surviving safepoint (e.g. the loop's exit or the next method return). The recorded stack is *that* location, not the loop. Systematically, time is shifted toward methods that *have* safepoints and away from the hot loop you most want to see. The profile lies about exactly the code under investigation. The famous demonstration of this is Nitsan Wakart's comparison showing different 'hot' methods reported by safepoint-biased vs safepoint-free profilers for the same workload. 2. **Temporal bias / coalescing.** Because reaching a safepoint takes variable time, samples aren't taken at the true uniform intervals; threads that are slow to reach a safepoint are sampled late, and the global stop adds its own overhead. Define the key terms: - **Safepoint:** a runtime-known safe spot to pause a thread; the JIT may remove them from hot loops. - **JVMTI:** the standard C API for tools (debuggers/profilers) to inspect the JVM. - **Deoptimization:** the JIT discarding optimized code back to the interpreter; also requires safepoints. ## How async-profiler avoids it **async-profiler** does *not* go through the safepoint mechanism. Its mechanism: 1. It registers for a periodic OS signal — `SIGPROF` driven by an interval timer, or hardware **`perf_events`** (which can also sample on CPU-cycle/cache-miss events). 2. When the signal fires, it interrupts the target thread **wherever it currently is** — inside the hot loop, in native code, anywhere — because OS signal delivery isn't tied to JVM safepoints. 3. In the signal handler it calls the JVM's semi-private **`AsyncGetCallTrace`** API, which can unwind the *current* Java call stack (plus native frames) from an arbitrary point, without requiring a safepoint. Because the snapshot is taken at the real instant the timer fired, samples land in the genuinely-executing code — **safepoint bias is eliminated.** As a bonus it can profile native and kernel frames, allocations (via TLAB hooks), locks, and wall-clock — not just on-CPU Java time. ## The built-in alternative: JFR **JDK Flight Recorder (JFR)** ships with the JDK and is engineered for *always-on*, sub-percent overhead. Its execution-sampling is also designed to be low-overhead and largely free of the classic safepoint bias for CPU profiling, and it captures a rich event stream (GC, allocation, locks, I/O) viewable in **JDK Mission Control**. In practice teams reach for **async-profiler** (deepest, flame-graph-friendly, native frames) and/or **JFR** (built-in, production-safe) and avoid legacy JVMTI-`GetAllStackTraces` samplers (e.g. some older commercial/IDE samplers) precisely because of safepoint bias. ## Deriving your own answer Safepoint bias = a JVM sampler that needs threads at safepoints records the *next* safepoint, not the true firing point; since the JIT strips safepoints from hot loops, the hottest code is under-reported and safepoint-bearing methods over-reported. The fix is **safepoint-free** sampling: interrupt via OS signal anywhere and unwind with `AsyncGetCallTrace` (async-profiler) — or use JFR. With that, you can explain both the failure and the modern remedy.