skip to content

As the lead for a fleet of JVM services, would you run async-profiler continuously in production, and if so in which modes, at what interval and overhead budget?

level: principalimportance: should knowfreq 35%

answer

  1. a baseline before the incident
  2. not --all in production
  3. coarse intervals, rotating JFR files
  4. agent at startup, debug info
  5. measure the budget, do not assume

basics

~20 s

Usually yes, narrowly: CPU, and perhaps wall-clock, allocation and lock at coarse intervals, as rotating JFR files with the agent loaded at startup. Set the overhead budget up front and measure it; broad or fine-grained modes stay on demand.

solid answer

~50 s

I would run it continuously where incidents are costly, because the value is the baseline: a profile from before the regression to diff against with `jfrconv --diff`. I would not use `--all`, which the documentation advises against in production. The default set would be `cpu` at a coarser interval than 10 ms, wall-clock only if latency questions are common (it samples every thread), allocation at a large `--alloc` interval and lock at a coarse interval, all into JFR with `loop` and a `%t` file name, loaded as an agent at startup so compiled code carries the debug information for inlined frames. I would cap trace storage with `memlimit`, never expose the agent's insecure `server` endpoint publicly, and set an explicit budget, for example one or two percent of CPU, verified by comparing CPU and p99 on canaries with and without it. Native-memory and method tracing stay on demand.

code

bash · 4 lines
bash
java -agentpath:/opt/async-profiler/lib/libasyncProfiler.so=start,event=cpu,interval=20ms,alloc=2m,lock=10ms,memlimit=256m,loop=15m,file=/profiles/profile-%p-%t.jfr \
  -jar service.jar

jfrconv --cpu --diff /profiles/before.jfr /profiles/after.jfr cpu-diff.html

go deeper

for a junior

Recall that async-profiler can run continuously with the loop option and write a new file per period.

for a middle

Explain why --all is discouraged in production and why each mode's cost scales with a different quantity.

for a senior

Show how a continuous profile from before a regression, diffed against one after it, shortens an incident compared with profiling on demand.

for a principal

Own the trade: which tiers profile continuously, which modes and intervals, the overhead budget and how it is measured, retention, and who may enable the invasive modes.

## The decision a lead owns Running a profiler on demand is cheap until the incident that matters: the regression happened yesterday, the process has since restarted, and there is nothing from *before* to compare with. **Continuous profiling** changes the question from "can we reproduce it?" to "what changed between these two profiles?". async-profiler supports it directly with the `loop` option, which restarts the profile every period and writes a new file each time. The decision has no single right answer; it is a trade between evidence and cost, and it should be made once per fleet rather than per incident. ## What it buys - **A baseline for every regression.** A `jfrconv --diff baseline.jfr new.jfr` graph across a release shows frames that grew (red), shrank (blue) or appeared (yellow). - **Evidence for transient problems** that are gone before anyone attaches. - **Profiles from real traffic**, not from a load test's approximation of it. ## What it costs | Cost | Where it comes from | Lever | |---|---|---| | CPU | a signal and a stack walk per sample | a coarser `interval`, fewer modes | | memory | the call-trace storage inside the JVM | `memlimit` | | storage | one JFR file per loop iteration | loop period, retention, `chunksize` | | risk | a native agent running inside the process | version pinning, canary rollout | | exposure | the agent's optional control server, an insecure HTTP server | never expose `server` on a public network | Sample cost scales with **what is sampled**: CPU samples scale with busy cores, wall-clock samples with the number of threads (the source raises wall mode's default interval for exactly that reason), allocation samples with the allocation rate. ## A defensible default 1. **Load the agent at startup**, not by attaching later. The troubleshooting notes recommend `-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` when the agent is not loaded at startup, and with a late attach debug information is enabled only for methods compiled after attaching. An agent loaded at startup gets that debug information for every method it sees compiled; a team that prefers attaching on demand sets the two flags at launch instead. 2. **Record a narrow set of modes into JFR**: `cpu` at a coarser interval than the 10 ms default; `alloc` with a large interval such as 2 MB; `lock` with a 10 ms interval, so a sample is taken each time accumulated lock-wait time passes it; wall-clock at a long interval only if the fleet's incidents are mostly latency spent waiting. 3. **Rotate files** with `loop` and a `%t` (timestamp) pattern in the file name, or every iteration overwrites the last one. 4. **Avoid `--all`.** It switches on CPU, wall-clock, allocation, live objects, lock, native lock, native memory and, on Linux, process statistics at once; the documentation calls it not recommended for production, especially for continuous profiling. 5. **Keep the costly or invasive modes on demand**: native-memory profiling, Java method tracing (whose first instrumentation after an attach can deoptimise all compiled methods), and fine intervals. ## Setting and proving the budget State the budget before rollout (for example, **one or two percent of CPU** and no measurable change in p99), then verify it rather than trusting a number from a blog post: run canary instances with and without the agent at the same traffic and compare CPU and latency percentiles. If the budget is exceeded, coarsen intervals or drop modes before dropping the practice. ## Retention and access - **Retention follows the release cadence.** Keeping files for at least two release cycles means every release can be diffed against the one before it. - **Tag files with what produced them**: service, version and instance, so a diff compares like with like. The `%p` (PID) and `%{ENV}` patterns in the file name help. - **Treat profiles as internal data.** Method and class names describe the code's structure; profiles belong where the team's other diagnostics live, not in a public bucket. ## Alternatives a lead should weigh - **Profile a subset of instances** (one per deployment or zone) for a lower total cost and less coverage. - **Rely on the JVM's built-in flight recorder** for always-on recording and keep async-profiler for deep dives; async-profiler can also feed its samples into a JFR recording with `--jfrsync`. - **Profile only on demand**, accepting that the first look at a regression comes after it is reported. The answer differs by fleet: a latency-sensitive tier with frequent releases benefits most from a baseline; a batch tier with ample headroom may need nothing beyond on-demand profiling.

  • Why does wall-clock mode cost more to keep on continuously than CPU mode in a JVM with many threads?
    CPU mode samples only threads that are using CPU, so its cost follows busy cores. Wall-clock mode samples threads whatever their state, so its cost follows the thread count, most of which are idle. async-profiler's source raises wall mode's default interval to five times the CPU default for that reason, and a continuous setup should be coarser still.
  • What breaks if a continuous async-profiler loop writes to a file name without %t?
    Each loop iteration writes the same file, so every new period overwrites the previous one and the history that continuous profiling exists to keep is lost. The documentation says the file name must include a timestamp pattern; `%p` adds the PID, which helps when several JVMs share a directory.

  • Turn on --all everywhere so every question is answered later.
  • A profiler's overhead is a fixed number from its documentation.
  • Attaching later is as accurate as loading the agent at startup.
  • On-demand profiling gives the same evidence as a baseline.
  • The agent's insecure control server can safely face a public network.