skip to content

On a HotSpot JVM, why does Native Memory Tracking's committed total rarely match the process's resident set size, and which way can they differ?

level: middleimportance: should knowfreq 45%

answer

  1. three different numbers
  2. touched pages only
  3. what NMT never sees
  4. anonymous versus file-backed

basics

~20 s

NMT committed counts memory the JVM made usable; RSS counts pages actually in RAM. Committed exceeds RSS when committed pages are untouched; RSS exceeds committed when native code, C-library retention or file mappings hold memory NMT never tracked.

solid answer

~50 s

They measure different things. **Reserved** is address space the JVM set aside, **committed** is memory it has made usable, and **resident** is the pages physically in RAM right now, which is what a container limit eventually charges. Committed can exceed resident because a page costs RAM only once touched: a heap committed up to `-Xms` without `-XX:+AlwaysPreTouch`, or the unused depth of thread stacks, can stay untouched for a long time. Resident can exceed committed because NMT sees only the JVM's own allocations: `malloc` calls made directly by JNI or other native libraries, memory glibc keeps after it was freed, and file mappings such as the JVM's shared libraries are outside it. So I compare the two as trends, and `jcmd <pid> VM.info` gives the resident set with its anonymous and file split next to them.

code

console · 5 lines
console
Process Memory:
Virtual Size: 5321440K (peak: 5389012K)
Resident Set Size: 1873220K (peak: 1873220K) (anon: 1801344K, file: 71876K, shmem: 0K)
Swapped out: 0K
C-Heap outstanding allocations: 402114K, retained: 318720K

go deeper

for a junior

Recall that reserved, committed and resident are three different numbers, and that only resident pages occupy physical memory.

for a middle

Explain both directions of the gap: untouched committed pages make NMT larger, and allocations outside NMT's view make the resident set larger.

for a senior

Use the gap as evidence: a growing difference between anonymous RSS and NMT committed is what sends you looking outside the JVM's own allocations.

for a principal

Decide which of these numbers a team graphs and alerts on, knowing that only resident memory is what a container limit eventually enforces.

## Three numbers, not one When a JVM's memory is questioned, three figures get confused: - **Reserved**: address space the JVM has set aside so that a region can grow contiguously, for example the whole `-Xmx` range for the heap or the maximum code cache. Reserving costs almost nothing. - **Committed**: the part of a reservation the JVM has made usable, plus everything it allocated with `malloc` (NMT counts a malloc as both reserved and committed). This is what **Native Memory Tracking (NMT)** reports as `committed=` per category and in its `Total:` line. - **Resident (RSS)**: the pages of the process that are in physical RAM right now. This is what memory pressure, and eventually a container limit, is about. NMT never measures residency. It is bookkeeping of the JVM's own requests, so its committed total and the operating system's resident set diverge, and they can diverge **in either direction**. ## When NMT committed is larger than RSS A page costs physical memory only once something writes to it. Committed memory that nobody has touched yet is not resident: - **The Java heap**: with `-Xms` equal to `-Xmx`, the heap is committed at startup, but `-XX:+AlwaysPreTouch` is off by default, so its pages become resident only as the collector first allocates into them. RSS climbs for a while after startup with no leak. - **Thread stacks**: each thread reserves its full stack (1024 KB by default on Linux x64), but only the frames it has used are touched. - **Other committed regions** such as parts of the code cache or class space that were committed ahead of use. This direction is harmless: RSS growth that stops at the committed size is the JVM filling space it already owned. ## When RSS is larger than NMT committed This is the direction that matters during a container kill, because it means memory NMT cannot attribute: 1. **Native code calling the C library directly.** A JNI library, or native code reached through the foreign function API, that calls `malloc` without going through the JVM's allocator is invisible to NMT. 2. **C-library retention.** When freed memory stays inside glibc's malloc arenas instead of going back to the kernel, the pages remain resident while NMT, which has seen the free, no longer counts them. 3. **File-backed mappings.** The JVM binary, its shared libraries and files the application maps into memory are resident pages that are not JVM allocations. 4. **Allocator overhead.** The C library adds its own per-chunk overhead around each allocation, which NMT does not count. | Gap direction | Typical cause | Growing over days? | |---|---|---| | committed > RSS | heap or stacks committed but untouched | no, it closes as pages are touched | | RSS > committed, file-backed | shared libraries, mapped files | rarely | | RSS > committed, anonymous | native mallocs outside NMT, glibc retention | yes, the classic slow climb | ## Comparing them properly The JVM itself prints the resident side. On Linux, `jcmd <pid> VM.info` includes a **Process Memory** section with `Resident Set Size`, its peak, and on kernels 4.5 and later the split into `anon`, `file` and `shmem`. With glibc it adds a line of C-heap **outstanding allocations** and **retained** memory, which speaks directly to point 2 above. ```console Resident Set Size: 1873220K (peak: 1873220K) (anon: 1801344K, file: 71876K, shmem: 0K) C-Heap outstanding allocations: 402114K, retained: 318720K ``` A sound comparison: - Compare **anonymous RSS** with NMT's committed total, because file-backed pages were never NMT's to count. - Compare **trends**, not single values: take both at the same moments, for example alongside each `summary.diff`. - Read a stable gap as overhead and a **growing** gap as evidence that the growth is outside the JVM's own allocations. ## What this means for triage If NMT categories grow in step with RSS, NMT has found the subsystem and its diff is the next piece of evidence. If RSS grows and NMT stays flat, stop reading NMT: the evidence you need comes from the C library and native allocation profiling. The second case is the one that surprises teams, because the JVM's own report looks healthy while the process is killed.

  • Where do you get the resident figure to compare with NMT, without leaving the JVM's own tools?
    `jcmd <pid> VM.info` on Linux prints a Process Memory section with `Resident Set Size` and its peak, split into anon, file and shmem on kernels 4.5 and later, and on glibc a line of C-heap outstanding allocations and retained memory. Comparing anon RSS with NMT committed is a fairer test than total RSS, because file-backed pages are mostly shared libraries and mapped files that NMT never claimed.
  • Why would RSS keep growing after startup even though the heap is committed in full with -Xms equal to -Xmx?
    Committing the heap reserves the right to use it but does not touch the pages. Without `-XX:+AlwaysPreTouch` (off by default), heap pages become resident only as the collector first allocates into them, so RSS can climb for hours toward the committed heap with no leak at all. That growth stops at the committed size; growth beyond it is somewhere else.

  • NMT committed is the same number as the resident set size.
  • If RSS exceeds NMT committed, NMT must be miscounting the heap.
  • Reserved memory in an NMT report counts against a container's memory limit.
  • A heap committed at startup is resident from the first second.
  • Memory freed by native code always returns to the operating system at once.