skip to content

On a HotSpot JVM running JDK 25, how do you use Native Memory Tracking to find which part of the JVM's native memory keeps growing?

level: middleimportance: must knowfreq 55%

answer

  1. a startup-only switch
  2. baseline once warm, diff later
  3. committed delta, not reserved
  4. thread and class counts in the diff

basics

~10 s

Start the JVM with -XX:NativeMemoryTracking=summary, take jcmd <pid> VM.native_memory baseline once it is warm, then run summary.diff later: the category whose committed figure keeps rising names the growing subsystem.

solid answer

~40 s

Native Memory Tracking is off by default and can only be switched on at startup, with `-XX:NativeMemoryTracking=summary` (or `=detail`); against a JVM started without it, `jcmd <pid> VM.native_memory` just answers `Native memory tracking is not enabled`. Once the service is warm I run `jcmd <pid> VM.native_memory baseline`, then every so often `jcmd <pid> VM.native_memory summary.diff scale=MB`. For each category (Java Heap, Class, Thread, Code, GC, Compiler, Internal, Other, Symbol, Arena Chunk, Metaspace and others) it prints the current reserved and committed figures with a signed change since the baseline, plus thread and class counts. I read the committed deltas, not the reserved ones, and look for a category that keeps climbing across several diffs. If the JVM was started in detail mode, `detail.diff` then names the native call sites behind the growth.

code

bash · 7 lines
bash
java -XX:NativeMemoryTracking=summary -jar service.jar

# after warm-up
jcmd <pid> VM.native_memory baseline

# an hour later, and again after that
jcmd <pid> VM.native_memory summary.diff scale=MB

go deeper

for a junior

Remember that NMT exists, that it is off by default, and that it splits the JVM's own native memory into named categories such as Thread, Code and Class.

for a middle

Explain the full workflow: the startup flag, baseline after warm-up, summary.diff at intervals, reading committed deltas and the thread and class counts, and when detail.diff is worth its cost.

for a senior

Show judgment about what a growing category implies and what NMT cannot see, and compare NMT's committed total with the resident set before trusting it as the whole story.

for a principal

Be ready to argue whether NMT should be on by default across a fleet, given that it cannot be enabled after the fact and adds bookkeeping to every JVM allocation.

## What Native Memory Tracking is **Native Memory Tracking (NMT)** is HotSpot's own accounting of the native memory the JVM allocates for itself: every `malloc` the VM makes through its internal allocator and every region it maps with `mmap`, each tagged with the subsystem that asked for it. It answers one triage question that heap metrics cannot: *when the process is bigger than the heap, which part of the JVM is responsible?* Key facts on JDK 25: - It is **off by default** in product builds (`NativeMemoryTracking` defaults to `off`). - It can only be enabled **at startup**: `-XX:NativeMemoryTracking=summary` or `-XX:NativeMemoryTracking=detail`. Any other value fails startup with `Syntax error, expecting -XX:NativeMemoryTracking=[off|summary|detail]`. - Against a JVM started without it, every `jcmd <pid> VM.native_memory` request prints `Native memory tracking is not enabled`. ## The baseline-and-diff workflow A single report tells you sizes; a leak is a *trend*. The workflow is therefore comparative: 1. Start the service with `-XX:NativeMemoryTracking=summary`. 2. Let it warm up: JIT compilation, class loading and pool start-up all grow native memory early and then settle. 3. Run `jcmd <pid> VM.native_memory baseline`. The JVM answers `Baseline taken`. 4. At regular intervals run `jcmd <pid> VM.native_memory summary.diff scale=MB`. Without a baseline the command prints `No baseline for comparison`. 5. Look for a category whose **committed** figure climbs diff after diff, while you track the resident set of the process alongside. 6. If the JVM runs in detail mode, run `detail.diff` (after a baseline taken in detail mode) to see which native call sites grew. The command takes at most one of `summary`, `detail`, `baseline`, `summary.diff`, `detail.diff` or `statistics` per call, and with none it prints a summary; `scale` picks `KB` (the default), `MB` or `GB`. ## Reading a summary.diff ```console Total: reserved=3104MB +391MB, committed=1772MB +27MB - Thread (reserved=410MB +392MB, committed=31MB +27MB) (threads #405 +390) (stack: reserved=408MB +391MB, committed=29MB +26MB) ``` Each category line carries the current **reserved** and **committed** amounts followed by the signed change since the baseline. **Reserved** is address space set aside; **committed** is memory the JVM has actually made usable, which is far closer to what costs RAM. Read the committed deltas first. Under a category you will see the pieces: `malloc=` for the VM's heap allocations with a count, `mmap:` for mapped regions, `arena=` for arena chunks, and extra lines for some categories. | Category | Growth usually means | |---|---| | Java Heap | the heap was committed further; compare with `-Xmx` | | Class, Metaspace | more class metadata: class loading or class-loader churn; Class adds class counts | | Thread | more threads (the `threads #` line) and their stacks | | Code | the JIT code cache filling | | GC | collector bookkeeping such as remembered sets and marking structures | | Compiler | memory the JIT compilers use while compiling | | Internal, Symbol, Arena Chunk | VM internals, the symbol table, arena-allocated scratch memory | | Other | memory not used by the VM itself, such as `Unsafe.allocateMemory` behind direct buffers | In the example, the heap is flat and 390 new threads appeared, each reserving about 1 MB of stack (the documented Linux x64 default stack size is 1024 KB), while only the touched part of each stack is committed. The evidence names a thread leak; the next step is a thread dump to see which code creates them. ## Detail mode and call sites Summary mode attributes memory to categories. **Detail mode** also records the native call stack of each allocation, so `VM.native_memory detail` lists call sites holding 1 KB or more and the virtual memory map with what reserved each region, and `detail.diff` shows which call sites grew since a detail baseline. It is more expensive than summary mode (a stack walk per tracked allocation and a table of sites), and it too must be chosen at startup. A JVM started in summary mode answers a `detail` or `detail.diff` request with `Detail tracking is not enabled`. ## Pitfalls when reading NMT - **Reserved is not usage.** A JVM reserves large address ranges (the heap at `-Xmx`, the code cache, class space) that can stay largely uncommitted unless `-Xms` or pre-touching says otherwise. - **Committed is not resident.** Committed pages count only once touched; NMT does not report what is in physical RAM. - **NMT sees only the JVM's own allocations.** A JNI library or other native code calling `malloc` directly is invisible, and so is memory the C library keeps after it was freed. If the resident set grows while every NMT category is flat, the cause lies outside NMT. - **One diff is not a trend.** Several diffs against the same post-warm-up baseline separate steady growth from warm-up and caches. - **NMT costs something.** It adds a header to every tracked allocation and a Native Memory Tracking category of its own; its tracking overhead line shows how much.

  • Why take several summary.diff reports rather than one diff an hour after startup?
    Early growth is mostly warm-up: the JIT fills the code cache, classes load, pools start their threads and collector structures size themselves. A single diff mixes that with any leak. Baseline after warm-up, then diff at regular intervals against the same baseline: a category that keeps climbing in a straight line is a suspect, while one that rises with traffic and settles is a cache or a pool reaching its size.
  • What does detail mode add, and what does it cost?
    Detail mode records the native call stack of each allocation site, so `VM.native_memory detail` lists call sites holding 1 KB or more and `detail.diff` shows which sites grew since a detail baseline, along with the virtual memory map. Capturing a stack per allocation costs more CPU and memory than summary mode, and like summary it must be chosen on the command line, so it suits a reproduction instance more than a default.
  • A summary.diff shows the Other category up by several hundred megabytes. Where do you look next?
    In JDK 25 the JVM tags memory obtained through `Unsafe.allocateMemory` as Other, which is the path direct byte buffers and other off-heap allocations by Java code take. So the next step is the direct buffer pool figures exposed through `BufferPoolMXBean` and finding the code that allocates direct buffers or raw off-heap memory without releasing it.

  • You can switch Native Memory Tracking on with jcmd against a running JVM.
  • The reserved figure in an NMT report is memory the process is actually using.
  • One NMT summary taken during the incident is enough to identify a leak.
  • NMT counts every malloc in the process, including calls made by JNI libraries.
  • A JVM started in summary mode can still produce a detail.diff by call site.