In an async-profiler CPU flame graph of a JVM, how do you find where time is really spent, and what do plateaus and colours show?
answer
- look at the tops
- flat top means self time
- green Java, red native, orange kernel
- JVM threads have their own towers
- inlined frames need debug info
basics
~20 sLook for wide frames with nothing on top: a plateau is self time, while a wide frame with children only passes time to callees. Colours give the frame type: compiled, inlined or interpreted Java, native, JVM C++ or kernel.
solid answer
~50 sI read upward from the root and stop at the widest frames whose top is flat: that plateau is where samples were taken in the method itself, so it is the code to look at. A wide frame with a stack of children is only a caller. async-profiler colours frames by type: green for JIT-compiled Java, a lighter yellow-green for C1-compiled, aqua for inlined, pale green for interpreted, red for native code, yellow for the JVM's own C++ code and orange for kernel frames. That tells me whether time is in my code, a native library, the JVM itself (GC and JIT compiler threads form separate towers) or system calls. Because the x-axis is sorted by name, not time, I compare widths only. If the profiler was attached late without `-XX:+DebugNonSafepoints`, small inlined methods can be folded into their caller's plateau.
code
bash · 3 linesasprof -e cpu -d 30 -f cpu.html <pid>
asprof -e cpu -d 30 --reverse -f cpu-by-leaf.html <pid>
jfrconv --cpu --diff before.jfr after.jfr diff.htmlgo deeper
Recall that the graph is read by width, that the x-axis is not time, and that a flat-topped frame is code that was itself running.
Explain self versus total time, how a reversed graph adds up one method's self time, and what each async-profiler frame colour means.
Show you can tell application code from JVM, native and kernel time at a glance, and you know when inlining or truncated stacks make a plateau lie.
Decide whether teams load the agent at startup, or set DebugNonSafepoints at launch, trading a little JIT bookkeeping for profiles that attribute inlined code correctly.
## What the picture is made of A **flame graph** stacks the frames of sampled call stacks: the root at the bottom, each callee above its caller, and the **width** of each frame equal to its share of the samples. async-profiler's documentation makes two points worth stating precisely before reading one: - The **x-axis is not a timeline**. Frames at each level are sorted by name and merged, so left-to-right order says nothing about when anything happened. - The graph shows **how often** a stack was seen, so a frame's width is its share of the profile's weight (in a CPU profile, of the CPU time sampled). ## Plateaus: where the time actually goes The reading rule that matters most is the difference between **total** and **self** time: 1. A wide frame with frames on top of it spent most of its samples *in its callees*. It is on the path to the cost, not the cost itself. 2. A wide frame with **nothing on top** (a flat-topped **plateau**) was itself executing when those samples were taken. That is **self time**, and it is where optimisation pays. 3. A method can be cheap at every individual call site and still expensive in total. Reversing the stacks (`--reverse` on `asprof`, `-r` on `jfrconv`) merges from the leaf frames down, so all of one method's self time adds up into a single wide frame regardless of who called it. In the documentation's worked example, only the leaf functions consume anything, and the deepest wide leaf is "a likely candidate for performance optimization". Everything below it is plumbing. ## Frame colours: which kind of code async-profiler colours frames by **frame type**, so the colour tells you which layer the time is in. The legend in its HTML output reads: | Colour | Frame type | Typical meaning in a JVM profile | |---|---|---| | green | Java compiled | your hot code after C2 compilation | | yellow-green | Java compiled by C1 | code still at the first JIT tier | | aqua | Inlined | a method inlined into its compiled caller | | pale green | Interpreted | code not yet compiled, common during warm-up | | red | Native | C libraries, JNI code, libc | | yellow | C++ (VM) | the JVM itself: GC, JIT compiler, runtime | | orange | Kernel | system calls, page faults, scheduling | Three common shapes follow from that table: - **Yellow towers rooted in JVM threads.** Garbage-collection workers, the VM thread (frames such as `VMThread::run`) and the JIT compiler threads (`CompileBroker::compiler_thread_loop`) appear as their own stacks beside your request threads. A wide yellow tower under GC means CPU spent collecting, which sends you to the allocation profile next. - **Orange on top of Java.** Kernel frames above a Java frame mean the thread was in a system call or a page fault while on the CPU. They appear only with a `perf_events`-based event such as `cpu`, and only when kernel symbols are visible. - **Pale green where you expected green.** A hot path still interpreted, typically in a profile taken during warm-up. The `-a` option also annotates names with suffixes such as `_[j]` (JIT-compiled) and `_[i]` (inlined); the colours are applied either way. ## Traps that distort a plateau - **Inlining without debug information.** async-profiler's troubleshooting notes say that when the agent is attached at runtime rather than loaded at startup, `-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` is highly recommended; without it, simple inlined methods are likely to vanish into their caller, making the caller's plateau look like its own work. - **Truncated native stacks.** On most Linux systems the native call stacks `perf_events` captures stop at 127 frames, so a very deep native stack is cut short. - **Missing frames below Java.** Native frames that precede the first Java frame (thread start-up) are hidden unless `--cstack vmx` is used; this is expected and harmless. ## Comparing two graphs To see what changed between two profiles, `jfrconv --diff baseline new diff.html` draws a **differential flame graph** in the shape of the new profile: red frames gained samples, blue frames lost them, yellow frames are new. Frames that exist only in the baseline are not drawn, so run it a second time with the inputs swapped to see what disappeared.
- In an async-profiler differential flame graph, why can a method that disappeared in the new release be invisible?The diff graph takes the shape of the current profile: every frame has the size it has in the new recording, coloured red if it gained samples, blue if it lost them and yellow if it is new. A frame present only in the baseline has no width in the new profile, so it is not drawn. Swap the two inputs to see it.
- Why does a wide yellow tower under a GC worker thread send you to a different profile next?Yellow frames are the JVM's own C++ code, and a tower rooted in a collector thread is CPU spent collecting. The CPU graph cannot say why the collector runs so often; an allocation profile shows which code allocates which classes, and that is usually where the collection pressure comes from.
- The widest frame at the bottom of the graph is the method to optimise.
- Left-to-right position shows the order in which the work happened.
- Every frame colour in an async-profiler graph is random decoration.
- Kernel frames always appear, whichever CPU engine took the profile.
- A diff flame graph shows every frame from both profiles.