On a HotSpot JVM running JDK 25, how do you use Native Memory Tracking to find which part of the JVM's native memory keeps growing?
answer
- a startup-only switch
- baseline once warm, diff later
- committed delta, not reserved
- thread and class counts in the diff
basics
~10 sStart the JVM with -XX:NativeMemoryTracking=summary, take jcmd <pid> VM.native_memory baseline once it is warm, then run summary.diff later: the category whose committed figure keeps rising names the growing subsystem.
solid answer
~40 sNative Memory Tracking is off by default and can only be switched on at startup, with `-XX:NativeMemoryTracking=summary` (or `=detail`); against a JVM started without it, `jcmd <pid> VM.native_memory` just answers `Native memory tracking is not enabled`. Once the service is warm I run `jcmd <pid> VM.native_memory baseline`, then every so often `jcmd <pid> VM.native_memory summary.diff scale=MB`. For each category (Java Heap, Class, Thread, Code, GC, Compiler, Internal, Other, Symbol, Arena Chunk, Metaspace and others) it prints the current reserved and committed figures with a signed change since the baseline, plus thread and class counts. I read the committed deltas, not the reserved ones, and look for a category that keeps climbing across several diffs. If the JVM was started in detail mode, `detail.diff` then names the native call sites behind the growth.
code
bash · 7 linesjava -XX:NativeMemoryTracking=summary -jar service.jar
# after warm-up
jcmd <pid> VM.native_memory baseline
# an hour later, and again after that
jcmd <pid> VM.native_memory summary.diff scale=MBgo deeper
Remember that NMT exists, that it is off by default, and that it splits the JVM's own native memory into named categories such as Thread, Code and Class.
Explain the full workflow: the startup flag, baseline after warm-up, summary.diff at intervals, reading committed deltas and the thread and class counts, and when detail.diff is worth its cost.
Show judgment about what a growing category implies and what NMT cannot see, and compare NMT's committed total with the resident set before trusting it as the whole story.
Be ready to argue whether NMT should be on by default across a fleet, given that it cannot be enabled after the fact and adds bookkeeping to every JVM allocation.
## What Native Memory Tracking is **Native Memory Tracking (NMT)** is HotSpot's own accounting of the native memory the JVM allocates for itself: every `malloc` the VM makes through its internal allocator and every region it maps with `mmap`, each tagged with the subsystem that asked for it. It answers one triage question that heap metrics cannot: *when the process is bigger than the heap, which part of the JVM is responsible?* Key facts on JDK 25: - It is **off by default** in product builds (`NativeMemoryTracking` defaults to `off`). - It can only be enabled **at startup**: `-XX:NativeMemoryTracking=summary` or `-XX:NativeMemoryTracking=detail`. Any other value fails startup with `Syntax error, expecting -XX:NativeMemoryTracking=[off|summary|detail]`. - Against a JVM started without it, every `jcmd <pid> VM.native_memory` request prints `Native memory tracking is not enabled`. ## The baseline-and-diff workflow A single report tells you sizes; a leak is a *trend*. The workflow is therefore comparative: 1. Start the service with `-XX:NativeMemoryTracking=summary`. 2. Let it warm up: JIT compilation, class loading and pool start-up all grow native memory early and then settle. 3. Run `jcmd <pid> VM.native_memory baseline`. The JVM answers `Baseline taken`. 4. At regular intervals run `jcmd <pid> VM.native_memory summary.diff scale=MB`. Without a baseline the command prints `No baseline for comparison`. 5. Look for a category whose **committed** figure climbs diff after diff, while you track the resident set of the process alongside. 6. If the JVM runs in detail mode, run `detail.diff` (after a baseline taken in detail mode) to see which native call sites grew. The command takes at most one of `summary`, `detail`, `baseline`, `summary.diff`, `detail.diff` or `statistics` per call, and with none it prints a summary; `scale` picks `KB` (the default), `MB` or `GB`. ## Reading a summary.diff ```console Total: reserved=3104MB +391MB, committed=1772MB +27MB - Thread (reserved=410MB +392MB, committed=31MB +27MB) (threads #405 +390) (stack: reserved=408MB +391MB, committed=29MB +26MB) ``` Each category line carries the current **reserved** and **committed** amounts followed by the signed change since the baseline. **Reserved** is address space set aside; **committed** is memory the JVM has actually made usable, which is far closer to what costs RAM. Read the committed deltas first. Under a category you will see the pieces: `malloc=` for the VM's heap allocations with a count, `mmap:` for mapped regions, `arena=` for arena chunks, and extra lines for some categories. | Category | Growth usually means | |---|---| | Java Heap | the heap was committed further; compare with `-Xmx` | | Class, Metaspace | more class metadata: class loading or class-loader churn; Class adds class counts | | Thread | more threads (the `threads #` line) and their stacks | | Code | the JIT code cache filling | | GC | collector bookkeeping such as remembered sets and marking structures | | Compiler | memory the JIT compilers use while compiling | | Internal, Symbol, Arena Chunk | VM internals, the symbol table, arena-allocated scratch memory | | Other | memory not used by the VM itself, such as `Unsafe.allocateMemory` behind direct buffers | In the example, the heap is flat and 390 new threads appeared, each reserving about 1 MB of stack (the documented Linux x64 default stack size is 1024 KB), while only the touched part of each stack is committed. The evidence names a thread leak; the next step is a thread dump to see which code creates them. ## Detail mode and call sites Summary mode attributes memory to categories. **Detail mode** also records the native call stack of each allocation, so `VM.native_memory detail` lists call sites holding 1 KB or more and the virtual memory map with what reserved each region, and `detail.diff` shows which call sites grew since a detail baseline. It is more expensive than summary mode (a stack walk per tracked allocation and a table of sites), and it too must be chosen at startup. A JVM started in summary mode answers a `detail` or `detail.diff` request with `Detail tracking is not enabled`. ## Pitfalls when reading NMT - **Reserved is not usage.** A JVM reserves large address ranges (the heap at `-Xmx`, the code cache, class space) that can stay largely uncommitted unless `-Xms` or pre-touching says otherwise. - **Committed is not resident.** Committed pages count only once touched; NMT does not report what is in physical RAM. - **NMT sees only the JVM's own allocations.** A JNI library or other native code calling `malloc` directly is invisible, and so is memory the C library keeps after it was freed. If the resident set grows while every NMT category is flat, the cause lies outside NMT. - **One diff is not a trend.** Several diffs against the same post-warm-up baseline separate steady growth from warm-up and caches. - **NMT costs something.** It adds a header to every tracked allocation and a Native Memory Tracking category of its own; its tracking overhead line shows how much.
- Why take several summary.diff reports rather than one diff an hour after startup?Early growth is mostly warm-up: the JIT fills the code cache, classes load, pools start their threads and collector structures size themselves. A single diff mixes that with any leak. Baseline after warm-up, then diff at regular intervals against the same baseline: a category that keeps climbing in a straight line is a suspect, while one that rises with traffic and settles is a cache or a pool reaching its size.
- What does detail mode add, and what does it cost?Detail mode records the native call stack of each allocation site, so `VM.native_memory detail` lists call sites holding 1 KB or more and `detail.diff` shows which sites grew since a detail baseline, along with the virtual memory map. Capturing a stack per allocation costs more CPU and memory than summary mode, and like summary it must be chosen on the command line, so it suits a reproduction instance more than a default.
- A summary.diff shows the Other category up by several hundred megabytes. Where do you look next?In JDK 25 the JVM tags memory obtained through `Unsafe.allocateMemory` as Other, which is the path direct byte buffers and other off-heap allocations by Java code take. So the next step is the direct buffer pool figures exposed through `BufferPoolMXBean` and finding the code that allocates direct buffers or raw off-heap memory without releasing it.
- You can switch Native Memory Tracking on with jcmd against a running JVM.
- The reserved figure in an NMT report is memory the process is actually using.
- One NMT summary taken during the incident is enough to identify a leak.
- NMT counts every malloc in the process, including calls made by JNI libraries.
- A JVM started in summary mode can still produce a detail.diff by call site.