skip to content

Native Memory Diagnosis

When resident memory runs far past the heap: Native Memory Tracking baselines and diffs, the JIT code cache, thread stacks, and malloc arenas NMT cannot see. A classic container-kill question.

part ofJVM internalsread it →
on this pageshow

questions

5

On a HotSpot JVM running JDK 25, how do you use Native Memory Tracking to find which part of the JVM's native memory keeps growing?

level: middleimportance: must knowfreq 55%

answer

  1. a startup-only switch
  2. baseline once warm, diff later
  3. committed delta, not reserved
  4. thread and class counts in the diff

basics

~10 s

Start the JVM with -XX:NativeMemoryTracking=summary, take jcmd <pid> VM.native_memory baseline once it is warm, then run summary.diff later: the category whose committed figure keeps rising names the growing subsystem.

solid answer

~40 s

Native Memory Tracking is off by default and can only be switched on at startup, with `-XX:NativeMemoryTracking=summary` (or `=detail`); against a JVM started without it, `jcmd <pid> VM.native_memory` just answers `Native memory tracking is not enabled`. Once the service is warm I run `jcmd <pid> VM.native_memory baseline`, then every so often `jcmd <pid> VM.native_memory summary.diff scale=MB`. For each category (Java Heap, Class, Thread, Code, GC, Compiler, Internal, Other, Symbol, Arena Chunk, Metaspace and others) it prints the current reserved and committed figures with a signed change since the baseline, plus thread and class counts. I read the committed deltas, not the reserved ones, and look for a category that keeps climbing across several diffs. If the JVM was started in detail mode, `detail.diff` then names the native call sites behind the growth.

code

bash · 7 lines
bash
java -XX:NativeMemoryTracking=summary -jar service.jar

# after warm-up
jcmd <pid> VM.native_memory baseline

# an hour later, and again after that
jcmd <pid> VM.native_memory summary.diff scale=MB

go deeper

for a junior

Remember that NMT exists, that it is off by default, and that it splits the JVM's own native memory into named categories such as Thread, Code and Class.

for a middle

Explain the full workflow: the startup flag, baseline after warm-up, summary.diff at intervals, reading committed deltas and the thread and class counts, and when detail.diff is worth its cost.

for a senior

Show judgment about what a growing category implies and what NMT cannot see, and compare NMT's committed total with the resident set before trusting it as the whole story.

for a principal

Be ready to argue whether NMT should be on by default across a fleet, given that it cannot be enabled after the fact and adds bookkeeping to every JVM allocation.

## What Native Memory Tracking is **Native Memory Tracking (NMT)** is HotSpot's own accounting of the native memory the JVM allocates for itself: every `malloc` the VM makes through its internal allocator and every region it maps with `mmap`, each tagged with the subsystem that asked for it. It answers one triage question that heap metrics cannot: *when the process is bigger than the heap, which part of the JVM is responsible?* Key facts on JDK 25: - It is **off by default** in product builds (`NativeMemoryTracking` defaults to `off`). - It can only be enabled **at startup**: `-XX:NativeMemoryTracking=summary` or `-XX:NativeMemoryTracking=detail`. Any other value fails startup with `Syntax error, expecting -XX:NativeMemoryTracking=[off|summary|detail]`. - Against a JVM started without it, every `jcmd <pid> VM.native_memory` request prints `Native memory tracking is not enabled`. ## The baseline-and-diff workflow A single report tells you sizes; a leak is a *trend*. The workflow is therefore comparative: 1. Start the service with `-XX:NativeMemoryTracking=summary`. 2. Let it warm up: JIT compilation, class loading and pool start-up all grow native memory early and then settle. 3. Run `jcmd <pid> VM.native_memory baseline`. The JVM answers `Baseline taken`. 4. At regular intervals run `jcmd <pid> VM.native_memory summary.diff scale=MB`. Without a baseline the command prints `No baseline for comparison`. 5. Look for a category whose **committed** figure climbs diff after diff, while you track the resident set of the process alongside. 6. If the JVM runs in detail mode, run `detail.diff` (after a baseline taken in detail mode) to see which native call sites grew. The command takes at most one of `summary`, `detail`, `baseline`, `summary.diff`, `detail.diff` or `statistics` per call, and with none it prints a summary; `scale` picks `KB` (the default), `MB` or `GB`. ## Reading a summary.diff ```console Total: reserved=3104MB +391MB, committed=1772MB +27MB - Thread (reserved=410MB +392MB, committed=31MB +27MB) (threads #405 +390) (stack: reserved=408MB +391MB, committed=29MB +26MB) ``` Each category line carries the current **reserved** and **committed** amounts followed by the signed change since the baseline. **Reserved** is address space set aside; **committed** is memory the JVM has actually made usable, which is far closer to what costs RAM. Read the committed deltas first. Under a category you will see the pieces: `malloc=` for the VM's heap allocations with a count, `mmap:` for mapped regions, `arena=` for arena chunks, and extra lines for some categories. | Category | Growth usually means | |---|---| | Java Heap | the heap was committed further; compare with `-Xmx` | | Class, Metaspace | more class metadata: class loading or class-loader churn; Class adds class counts | | Thread | more threads (the `threads #` line) and their stacks | | Code | the JIT code cache filling | | GC | collector bookkeeping such as remembered sets and marking structures | | Compiler | memory the JIT compilers use while compiling | | Internal, Symbol, Arena Chunk | VM internals, the symbol table, arena-allocated scratch memory | | Other | memory not used by the VM itself, such as `Unsafe.allocateMemory` behind direct buffers | In the example, the heap is flat and 390 new threads appeared, each reserving about 1 MB of stack (the documented Linux x64 default stack size is 1024 KB), while only the touched part of each stack is committed. The evidence names a thread leak; the next step is a thread dump to see which code creates them. ## Detail mode and call sites Summary mode attributes memory to categories. **Detail mode** also records the native call stack of each allocation, so `VM.native_memory detail` lists call sites holding 1 KB or more and the virtual memory map with what reserved each region, and `detail.diff` shows which call sites grew since a detail baseline. It is more expensive than summary mode (a stack walk per tracked allocation and a table of sites), and it too must be chosen at startup. A JVM started in summary mode answers a `detail` or `detail.diff` request with `Detail tracking is not enabled`. ## Pitfalls when reading NMT - **Reserved is not usage.** A JVM reserves large address ranges (the heap at `-Xmx`, the code cache, class space) that can stay largely uncommitted unless `-Xms` or pre-touching says otherwise. - **Committed is not resident.** Committed pages count only once touched; NMT does not report what is in physical RAM. - **NMT sees only the JVM's own allocations.** A JNI library or other native code calling `malloc` directly is invisible, and so is memory the C library keeps after it was freed. If the resident set grows while every NMT category is flat, the cause lies outside NMT. - **One diff is not a trend.** Several diffs against the same post-warm-up baseline separate steady growth from warm-up and caches. - **NMT costs something.** It adds a header to every tracked allocation and a Native Memory Tracking category of its own; its tracking overhead line shows how much.

  • Why take several summary.diff reports rather than one diff an hour after startup?
    Early growth is mostly warm-up: the JIT fills the code cache, classes load, pools start their threads and collector structures size themselves. A single diff mixes that with any leak. Baseline after warm-up, then diff at regular intervals against the same baseline: a category that keeps climbing in a straight line is a suspect, while one that rises with traffic and settles is a cache or a pool reaching its size.
  • What does detail mode add, and what does it cost?
    Detail mode records the native call stack of each allocation site, so `VM.native_memory detail` lists call sites holding 1 KB or more and `detail.diff` shows which sites grew since a detail baseline, along with the virtual memory map. Capturing a stack per allocation costs more CPU and memory than summary mode, and like summary it must be chosen on the command line, so it suits a reproduction instance more than a default.
  • A summary.diff shows the Other category up by several hundred megabytes. Where do you look next?
    In JDK 25 the JVM tags memory obtained through `Unsafe.allocateMemory` as Other, which is the path direct byte buffers and other off-heap allocations by Java code take. So the next step is the direct buffer pool figures exposed through `BufferPoolMXBean` and finding the code that allocates direct buffers or raw off-heap memory without releasing it.

  • You can switch Native Memory Tracking on with jcmd against a running JVM.
  • The reserved figure in an NMT report is memory the process is actually using.
  • One NMT summary taken during the incident is enough to identify a leak.
  • NMT counts every malloc in the process, including calls made by JNI libraries.
  • A JVM started in summary mode can still produce a detail.diff by call site.
open as a page

When a JVM in a 2 GiB container with a 1 GiB max heap gains resident memory for two days until OOM-killed while post-GC heap and all NMT categories stay flat, where is the memory going, and how do you prove it?

level: seniorimportance: must knowfreq 50%

basics

~20 s

Flat NMT with rising RSS points outside the JVM's own allocations: a native library leaking malloc'd memory, or glibc arenas retaining freed memory. VM.info's outstanding-versus-retained line, a native heap trim and async-profiler's nativemem leak profile tell them apart.

open as a page

On a HotSpot JVM, why does Native Memory Tracking's committed total rarely match the process's resident set size, and which way can they differ?

level: middleimportance: should knowfreq 45%

basics

~20 s

NMT committed counts memory the JVM made usable; RSS counts pages actually in RAM. Committed exceeds RSS when committed pages are untouched; RSS exceeds committed when native code, C-library retention or file mappings hold memory NMT never tracked.

open as a page

When a JDK 25 HotSpot service slows after days of uptime and its stderr shows "CodeHeap 'non-profiled nmethods' is full. Compiler has been disabled.", what happened, and how do you confirm and fix it?

level: seniorimportance: should knowfreq 35%

basics

~20 s

The JIT's code cache segment for optimised code filled, so HotSpot stopped compiling and new or recompiled hot methods run slower code. Confirm with jcmd Compiler.codecache and the CodeCacheFull event, then size the cache from max_used with headroom.

open as a page

Since Native Memory Tracking can only be enabled when a HotSpot JVM starts, would you turn on its summary mode by default across a fleet of containerised JVM services, and what would you weigh?

level: principalimportance: should knowfreq 30%

basics

~20 s

Summary-mode NMT usually belongs on by default, after measuring its cost: it cannot be enabled mid-incident, and restarting to add it resets a slow leak. Keep detail mode for reproductions and watch RSS too, since NMT sees only JVM-owned memory.

open as a page