skip to content

Flame Graph Profiling

Sampling a live JVM with async-profiler in CPU, wall-clock, allocation or lock mode, then reading the flame graph it draws. Interviewers hand you a graph and ask where the time actually went.

part ofJVM internalsread it →
on this pageshow

questions

5

In async-profiler, which event mode answers which question about a live JVM, and what does each mode's flame graph actually count?

level: middleimportance: must knowfreq 50%

answer

  1. one resource per profile
  2. on-CPU versus elapsed time
  3. running, sleeping or blocked alike
  4. top frame is a class
  5. nanoseconds waiting to enter

basics

~20 s

Use cpu for where CPU time is burned, wall for elapsed thread time including waits, alloc for which call sites allocate heap, and lock for waits to enter contended locks; each graph counts only its own resource.

solid answer

~40 s

Each async-profiler mode samples one resource, so the first step is matching the mode to the question. `-e cpu` (or the `itimer`/`ctimer` engines) samples threads while they consume CPU: one sample is roughly one interval of one core's time, so a blocked thread is invisible. `-e wall` samples all threads at a fixed period whatever their state, which is what you need for latency spent waiting. `-e alloc` samples heap allocations; the top frame is the class of the allocated object and the weight is heap pressure. `-e lock` records contended lock entry; the top frame is the lock's class and the counter is nanoseconds spent waiting to enter. Several modes can be recorded together only in JFR output, then split with `jfrconv --cpu`, `--wall`, `--alloc` or `--lock`.

code

bash · 6 lines
bash
asprof -e cpu -d 30 -f cpu.html <pid>
asprof -e wall -t -i 50ms -d 30 -f wall.html <pid>
asprof -e alloc --alloc 500k -d 30 -f alloc.html <pid>
asprof -e lock -t -i 5ms -d 30 -f lock.html <pid>
asprof -e cpu --alloc 2m --lock 10ms -d 60 -f profile.jfr <pid>
jfrconv --alloc profile.jfr alloc.html

go deeper

for a junior

Recall that async-profiler has different modes for CPU, elapsed time, allocations and locks, and that a CPU profile cannot see a thread that is waiting.

for a middle

Explain what one sample means in each mode, what the top frame is in alloc and lock graphs, and why several modes need JFR output and separate conversion.

for a senior

Show you pick the mode from the symptom: wall for slow requests with idle CPU, alloc for frequent young collections, lock for queueing, and you know each mode's blind spot.

for a principal

Weigh which modes a team can afford to keep recording by default and which stay on demand, since wall and allocation sampling cost more as threads and allocation rates grow.

## Why the mode is the first decision A **flame graph** is an aggregation of sampled call stacks: every sample adds weight to each frame on its stack, and the width of a frame is its share of the total weight. What a sample *is* depends entirely on the **event mode** async-profiler was started with. A CPU flame graph and a wall-clock flame graph of the same minute can look nothing alike, and both are correct, because they count different things. Choosing the wrong mode is the most common reason a profile "shows nothing useful": the evidence is real, it just answers a different question from the one being asked. So before reading any graph, write down the question: - *Why is the process using so much CPU?* That is a CPU profile. - *Where do my request threads spend their elapsed time?* That is a wall-clock profile. - *Why is the collector running so often?* That is an allocation profile. - *Are threads queueing on a shared lock?* That is a lock profile. ## CPU modes: `cpu`, `itimer`, `ctimer` `-e cpu` measures **CPU time consumed by running threads**. On Linux it uses `perf_events`: one descriptor per thread, each firing a signal every *N* nanoseconds of that thread's CPU time (10 ms by default). The documentation's example makes the unit concrete: two cores each 30% busy at a 10 ms interval give about 60 samples per second. One sample therefore means "one core was busy for one interval here". A thread that is blocked in a socket read, parked in a pool or sleeping consumes no CPU and **produces no samples**. `itimer` (`setitimer(ITIMER_PROF)`) and `ctimer` (`timer_create`) also count CPU time, without needing `perf_events`. Neither can collect **kernel stack traces**, and both are limited to jiffy resolution. They matter mostly in containers, where `perf_events` is often blocked. ## Wall-clock mode: `wall` `-e wall` samples **all threads equally every period, regardless of state**: running, sleeping or blocked. Its default interval in async-profiler 4.x is five times the CPU default (50 ms), because it samples many more threads. This is the mode for latency spent *waiting*: downstream calls, connection-pool borrows, lock waits, sleeps. Its trap is the idle thread: a pool thread parked waiting for work is sampled just as often as a busy request thread, so a wall graph is usually read per thread (`-t`) and filtered to the threads that serve requests. ## Allocation mode: `alloc` `-e alloc` samples **heap allocations** rather than time. In async-profiler 4.x on a current HotSpot it uses the JVM's sampled-allocation callback by default (TLAB events with `--tlab`), taking roughly one sample per `--alloc` interval of allocated bytes (512 KiB by default in the source). Two reading rules: - The **top frame of every stack is the class of the allocated object** (for example `byte[]`); the frame beneath it is the code that allocated it. - Only allocations that **actually reach the heap** are measured; an allocation removed by escape analysis does not appear. This is the graph that explains a high allocation rate, which is usually what drives frequent young collections. ## Lock mode: `lock` `-e lock` records **contended** lock acquisition. In this mode the **top frame is the class of the lock or monitor**, and the counter is the number of **nanoseconds it took to enter** it. In the 4.x source it covers contended `synchronized` monitors and parks whose blocker is a `ReentrantLock`, `ReentrantReadWriteLock` or `Semaphore`; an uncontended acquisition is not recorded. The width is waiting time, not holding time: a wide frame says threads queued there, and the owner's slow work has to be found in a CPU or wall profile. ## Summary table | Mode | One unit of weight | Answers | Invisible to it | |---|---|---|---| | `cpu` | an interval of one core's CPU time | where CPU is burned, including kernel frames | blocked and waiting threads | | `itimer` / `ctimer` | an interval of CPU time, no `perf_events` | the same, where `perf_events` is unavailable | kernel stacks, sub-jiffy detail | | `wall` | one sampling tick of one thread | where elapsed time goes, waiting included | nothing by state, so idle threads add noise | | `alloc` | sampled allocated bytes | which code allocates which classes | eliminated allocations, time | | `lock` | nanoseconds waiting to enter | which locks threads queue on | uncontended locks, hold time | ## Recording more than one Modes can be combined in one session, but **only JFR output holds several event types together**: 1. Record, for example, `asprof -e cpu --alloc 2m --lock 10ms -f profile.jfr <pid>`. 2. Convert each view separately with `jfrconv --cpu`, `--alloc` or `--lock`, because a graph that sums CPU samples and allocated bytes means nothing. 3. Read each graph against the question that mode answers.

  • Why might an allocation you can see in the source never appear in an async-profiler alloc flame graph?
    The profile measures only allocations that reach the heap, so an object that escape analysis eliminates, or scalar-replaces, leaves no sample. It is also a sample: roughly one per `--alloc` interval of allocated bytes, so a small, rare allocation can be missed in a short recording. Lowering the interval raises the resolution at a higher cost.
  • A lock flame graph shows a wide frame on one monitor class. Does that tell you what the owner was doing?
    No. Lock mode weighs the time waiters spent entering the lock, so it names the contended lock and the code paths that queued on it. What the owner did while holding it comes from a CPU or wall-clock profile of the same window, or from thread dumps that show the owner's stack.

  • A CPU flame graph shows where request latency goes, waits included.
  • Lock mode measures how long each lock was held by its owner.
  • The top frame of an alloc stack is the method that called new.
  • Wall-clock mode samples only threads that are running.
  • Several modes can be merged into one meaningful HTML flame graph.
open as a page

A pricing service's async-profiler CPU flame graph shows a wide regex plateau, while a wall-clock profile of the same minute shows request threads in a socket read: which explains slow requests, and why?

level: seniorimportance: must knowfreq 52%

basics

~20 s

The wall-clock profile explains the slow requests: latency is elapsed time, and threads blocked in a socket read use no CPU, so a CPU profile cannot see them. The regex plateau is the largest share of CPU, not of request time.

open as a page

In an async-profiler CPU flame graph of a JVM, how do you find where time is really spent, and what do plateaus and colours show?

level: middleimportance: should knowfreq 45%

basics

~20 s

Look for wide frames with nothing on top: a plateau is self time, while a wide frame with children only passes time to callees. Colours give the frame type: compiled, inlined or interpreted Java, native, JVM C++ or kernel.

open as a page

When async-profiler -e cpu on a JVM inside a Docker container reports perf events unavailable, or shows no kernel frames, why does that happen, and what are your options?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Docker's default seccomp profile or a restrictive perf_event_paranoid blocks perf_event_open, so async-profiler falls back to ctimer, which cannot collect kernel stacks. Accept ctimer, force user-space perf_events, profile from the host as a privileged user, or relax seccomp deliberately.

open as a page

As the lead for a fleet of JVM services, would you run async-profiler continuously in production, and if so in which modes, at what interval and overhead budget?

level: principalimportance: should knowfreq 35%

basics

~20 s

Usually yes, narrowly: CPU, and perhaps wall-clock, allocation and lock at coarse intervals, as rotating JFR files with the agent loaded at startup. Set the overhead budget up front and measure it; broad or fine-grained modes stay on demand.

open as a page