For a Go process, how does an inuse_space heap profile differ from the RSS the operating system reports?

juniorshould knowfreq 35%

answer

two numbers, two different observers
one counts objects, one counts pages
the profile stops at the last GC
freed spans are still mapped memory
stacks and metadata are invisible to it

basics

~20 s

An inuse_space heap profile counts only live Go heap objects, sampled as of the last completed garbage collection. RSS counts every physical page the kernel has given the whole process, including freed-but-retained heap spans, goroutine stacks and runtime structures.

solid answer

~50 s

They are measured by different observers, which is why they disagree. inuse_space in a Go heap profile is bytes held by heap objects that were still reachable at the most recently completed GC cycle — a sampled view of your program's live data and nothing else. RSS is a kernel number: the physical pages currently mapped into the process. Several layers sit between them that the profile never shows — heap spans the runtime freed but keeps for reuse, goroutine stacks, the allocator's own metadata, anything mapped by cgo, and the binary itself. The profile is also a snapshot as of the last GC, so it can be a whole cycle stale. A 400 MB live heap against a 1.2 GB RSS is therefore normal rather than automatically a leak; runtime.MemStats is what tells you which layer holds the difference.

They are measured by different observers, which is why they disagree. `inuse_space` in a Go heap profile is bytes held by heap objects that were still reachable at the most recently completed GC cycle — a sampled view of your program's live data and nothing else. RSS is a kernel number: the physical pages currently mapped into the process. Several layers sit between them that the profile never shows — heap spans the runtime freed but keeps for reuse, goroutine stacks, the allocator's own metadata, anything mapped by cgo, and the binary itself. The profile is also a snapshot as of the last GC, so it can be a whole cycle stale. A 400 MB live heap against a 1.2 GB RSS is therefore normal rather than automatically a leak; `runtime.MemStats` is what tells you which layer holds the difference.

go deeper

for a junior

Be ready to say in one sentence that a Go heap profile counts live heap objects while RSS counts physical pages the kernel has handed the whole process. Do not claim the two numbers should agree.

for a middle

Explain the layers in between: idle heap spans the runtime retains, goroutine stacks, allocator metadata, and anything mapped outside the Go heap. Add that the profile is a snapshot as of the last completed GC.

for a senior

Show that you would not open an incident on the gap alone. Say which number you read next, runtime.MemStats, and what size or trend in the gap would genuinely worry you.

for a principal

Own what your services' memory dashboards actually plot. Alerting a fleet on RSS with no live-heap signal beside it generates pages that on-call engineers cannot act on.

Two numbers, two observers

When a Go service's memory looks wrong, two numbers usually end up side by side on a screen: the total at the top of a heap profile, and the memory figure on a container or host dashboard. They almost never match, and the first thing to understand is that they are not trying to.

inuse_space is one of the sample indexes of Go's heap profile (the profile served at /debug/pprof/heap, or written with runtime/pprof). It reports, per allocation site, the number of bytes occupied by heap objects that were still reachable at the most recently completed garbage collection cycle. Three qualifications are load-bearing:

Heap objects only. Nothing else the process uses appears here.
Still reachable. Objects the collector proved dead are gone from this view the moment the cycle that found them finished.
As of the last completed GC. It is not a live view. Between cycles the real live set can be much larger than the profile says.

RSS (resident set size) is the kernel's accounting: how many physical page frames are currently mapped into this process's address space. The kernel knows nothing about Go objects. It knows about pages, and it decrements RSS only when something explicitly tells it those pages are no longer wanted.

What lives in the gap

Everything in the following list is inside RSS and outside inuse_space:

Idle heap spans. When the collector frees objects, the memory does not go back to the kernel. It goes back to the Go runtime's page allocator as idle spans, ready to satisfy the next allocation without a syscall. A background scavenger returns some of it over time. Until it does, those pages are resident and empty. This is normally the largest part of the gap right after a memory spike.
Goroutine stacks. Each goroutine has a growable stack, and stack memory is managed separately from the object heap. A service with 50,000 goroutines can hold hundreds of megabytes of stacks that no heap profile mentions.
Runtime metadata. Span records, the per-P allocator caches, the profiling hash buckets and the GC's own bitmaps all cost memory the profile does not attribute to your code.
Anything outside the Go heap. Memory mmap'd directly, allocations made by C code reached through cgo, and the mapped text and data of the binary itself.
Sampling. The heap profile samples rather than recording every allocation, so even for pure heap objects it is a statistical estimate, not a byte-exact ledger.

Why the direction of the gap matters

RSS is essentially always larger than inuse_space, so the interesting question is never is there a gap but is the gap growing, and which layer owns it. A gap that is stable across hours is a service with a working set and a runtime that has sized its arena for it. A gap that grows monotonically over days is worth chasing. A live heap that itself grows monotonically is a different problem entirely — that is a real leak, meaning something is still reachable that should not be.

The practical next step is runtime.ReadMemStats, which fills a runtime.MemStats struct with numbers that are current as of the call (unlike the profile, which is pinned to the last GC). HeapAlloc is the live-object figure; HeapIdle minus HeapReleased is the heap memory the runtime has kept mapped rather than returned; Sys is the total virtual address space the runtime has obtained. Reading those three against RSS tells you within a minute whether you are looking at retained pages, at stacks, at something off-heap, or at an actual leak.

The mistake to avoid

The common error is to treat RSS as the authoritative heap size and open an investigation because it exceeds the profile. The second, subtler error is the reverse: to trust a heap profile that shows a small live set and conclude that memory is fine while the process is steadily approaching its container limit. Both numbers are true; they answer different questions. The heap profile answers "what is my program holding", and RSS answers "what has the operating system committed on my behalf". Diagnosis begins by deciding which of those two questions your symptom is actually about.

## Two numbers, two observers When a Go service's memory looks wrong, two numbers usually end up side by side on a screen: the total at the top of a heap profile, and the memory figure on a container or host dashboard. They almost never match, and the first thing to understand is that they are not trying to. **`inuse_space`** is one of the sample indexes of Go's heap profile (the profile served at `/debug/pprof/heap`, or written with `runtime/pprof`). It reports, per allocation site, the number of bytes occupied by heap objects that were **still reachable at the most recently completed garbage collection cycle**. Three qualifications are load-bearing: - *Heap objects only.* Nothing else the process uses appears here. - *Still reachable.* Objects the collector proved dead are gone from this view the moment the cycle that found them finished. - *As of the last completed GC.* It is not a live view. Between cycles the real live set can be much larger than the profile says. **RSS** (resident set size) is the kernel's accounting: how many physical page frames are currently mapped into this process's address space. The kernel knows nothing about Go objects. It knows about pages, and it decrements RSS only when something explicitly tells it those pages are no longer wanted. ## What lives in the gap Everything in the following list is inside RSS and outside `inuse_space`: 1. **Idle heap spans.** When the collector frees objects, the memory does not go back to the kernel. It goes back to the Go runtime's page allocator as idle spans, ready to satisfy the next allocation without a syscall. A background scavenger returns some of it over time. Until it does, those pages are resident and empty. This is normally the largest part of the gap right after a memory spike. 2. **Goroutine stacks.** Each goroutine has a growable stack, and stack memory is managed separately from the object heap. A service with 50,000 goroutines can hold hundreds of megabytes of stacks that no heap profile mentions. 3. **Runtime metadata.** Span records, the per-P allocator caches, the profiling hash buckets and the GC's own bitmaps all cost memory the profile does not attribute to your code. 4. **Anything outside the Go heap.** Memory `mmap`'d directly, allocations made by C code reached through cgo, and the mapped text and data of the binary itself. 5. **Sampling.** The heap profile samples rather than recording every allocation, so even for pure heap objects it is a statistical estimate, not a byte-exact ledger. ## Why the direction of the gap matters RSS is essentially always larger than `inuse_space`, so the interesting question is never *is there a gap* but *is the gap growing, and which layer owns it*. A gap that is stable across hours is a service with a working set and a runtime that has sized its arena for it. A gap that grows monotonically over days is worth chasing. A live heap that itself grows monotonically is a different problem entirely — that is a real leak, meaning something is still reachable that should not be. The practical next step is `runtime.ReadMemStats`, which fills a `runtime.MemStats` struct with numbers that are current as of the call (unlike the profile, which is pinned to the last GC). `HeapAlloc` is the live-object figure; `HeapIdle` minus `HeapReleased` is the heap memory the runtime has kept mapped rather than returned; `Sys` is the total virtual address space the runtime has obtained. Reading those three against RSS tells you within a minute whether you are looking at retained pages, at stacks, at something off-heap, or at an actual leak. ## The mistake to avoid The common error is to treat RSS as the authoritative heap size and open an investigation because it exceeds the profile. The second, subtler error is the reverse: to trust a heap profile that shows a small live set and conclude that memory is fine while the process is steadily approaching its container limit. Both numbers are true; they answer different questions. The heap profile answers "what is my program holding", and RSS answers "what has the operating system committed on my behalf". Diagnosis begins by deciding which of those two questions your symptom is actually about.

Why can a Go heap profile be stale relative to what the program is doing right now?
Because a heap profile is a snapshot as of the most recently completed garbage collection cycle. Objects allocated since that cycle finished are not reflected in `inuse_space` at all, so a program in the middle of a large allocation burst can look far smaller than it is. `runtime.ReadMemStats`, by contrast, is documented as up to date as of the call itself, which is why the two are usually read together.
Name three things counted in a Go process's RSS that never appear in inuse_space.
Goroutine stacks; idle heap spans the runtime freed but has retained for reuse; and the allocator's own metadata, such as span records and per-P caches. Beyond those there is anything mapped outside the Go heap — cgo allocations, explicit `mmap` regions, and the resident pages of the binary itself.
Is a rising RSS with a flat live heap always a bug?
No. The usual benign cause is a transient spike: the process genuinely needed that memory for a moment, the collector freed the objects, and the runtime kept the pages as idle spans so it can grow again without asking the kernel. It becomes a problem when the retained amount is large relative to the container limit, or when it keeps climbing rather than plateauing.

The heap profile is the inventory of goods still on the shelves. RSS is the size of the warehouse the company is renting. Emptying shelves does not shorten the lease.

saying these in an interview costs you the question

Says RSS should match the heap profile total
Treats any RSS above the live heap as a leak
Thinks goroutine stacks appear in a heap profile
Believes the heap profile is a real-time view
Assumes freed objects are immediately unmapped

For a Go process, how does an inuse_space heap profile differ from the RSS the operating system reports?

§ answer

basics

solid answer

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Two numbers, two observers

What lives in the gap

Why the direction of the gap matters

The mistake to avoid

▾common follow-ups3 questions

▾analogy~10 s

▾red flags5 flags

Two numbers, two observers

What lives in the gap

Why the direction of the gap matters

The mistake to avoid

answer

go deeper