In runtime.MemStats, what do HeapAlloc, HeapIdle, HeapReleased and Sys each measure?
answer
- one field is live objects, three are not
- idle spans are freed but still mapped
- subtract released from idle
- Sys is address space, not resident memory
- reading it stops the world briefly
basics
~20 sHeapAlloc is bytes in live heap objects. HeapIdle is bytes in spans holding no objects. HeapReleased is the part of that already handed back to the OS. Sys is all virtual address space the runtime obtained from the OS.
solid answer
~50 s`HeapAlloc` is bytes of allocated, still-reachable heap objects — the number most people mean by "the heap". `HeapIdle` is bytes sitting in spans with no objects in them: memory the collector reclaimed and the runtime is keeping. `HeapReleased` is how much of that has actually been handed back to the operating system. `Sys` is the total virtual address space the runtime obtained from the OS across heap, stacks and internal structures — an upper bound, not resident memory. The derived number that matters is **`HeapIdle - HeapReleased`**: heap memory the runtime is retaining, mapped and empty, so it can grow again without a syscall. If that difference is large relative to the live heap, there was a recent transient spike. Note that `runtime.ReadMemStats` briefly stops the world, so sample it on an interval, never per request.
code
go · 6 linesvar m runtime.MemStats
runtime.ReadMemStats(&m) // briefly stops the world
retained := m.HeapIdle - m.HeapReleased
fmt.Printf("live=%d idle=%d released=%d retained=%d sys=%d\n",
m.HeapAlloc, m.HeapIdle, m.HeapReleased, retained, m.Sys)go deeper
Know that runtime.MemStats exists, that runtime.ReadMemStats fills it in, and that HeapAlloc is the live-object figure people usually mean when they say the heap.
Define all four fields cleanly and derive HeapIdle minus HeapReleased as the memory the runtime is retaining. Say explicitly that Sys is address space rather than resident memory.
Explain why you sample these on a slow interval rather than per request, since ReadMemStats stops the world, and show how the three-way split of the numbers points at three different fixes.
Decide which two or three of these numbers an entire fleet exports and defend the choice. Exporting all of MemStats gives every engineer a different favourite metric and no shared story during an incident.
## Why these four `runtime.MemStats` has dozens of fields, and reading all of them at once is how people end up with no story. Four of them answer the question "is my process holding memory it is not using, and where is it". ### HeapAlloc Bytes of allocated heap objects that are still reachable. This is the closest thing the runtime has to "my program's data", and it is the number that should track your workload. It rises as you allocate and falls at the end of a GC cycle that proves objects dead. If `HeapAlloc` climbs steadily across hours with a steady workload, you have a genuine leak: something is holding references it should have dropped. ### HeapIdle Bytes in **idle spans** — spans of heap memory with no objects in them at all. A span becomes idle when everything in it dies. Idle memory is not gone: the runtime can hand it back out for new heap allocations, reuse it as goroutine stack memory, or return it to the operating system. Idle memory that has not been returned is still resident, so it still counts against a container limit. ### HeapReleased Bytes of physical memory that were in idle spans and have actually been given back to the OS — and not since reacquired. The runtime's background scavenger does this gradually, on its own schedule. ### Sys The total bytes obtained from the OS: the sum of all the `*Sys` fields, covering the heap, goroutine stacks, span and cache metadata, the profiling hash table and the GC's structures. Critically, `Sys` measures **virtual address space reserved**, not physical memory resident. It is generally an upper bound on RSS, and address space that was reserved but never touched, or that has been released back, is still counted in `Sys`. Treating `Sys` as "what the OS sees" is a common mistake. ## The one derived number to remember ``` retained := HeapIdle - HeapReleased ``` This is heap memory the runtime freed, kept mapped, and has *not* returned — memory that is costing you resident pages while holding nothing. The runtime's own documentation frames it exactly this way: it estimates memory that could be returned but is being retained so the heap can grow without asking the OS again, and if that difference is significantly larger than the live heap it indicates a recent transient spike in live heap size. That sentence is the whole diagnosis for the most common "our memory graph never comes down" report. The service had a peak; the peak sized the arena; the collector reclaimed the objects; the pages stayed. A second useful derivation is `HeapInuse - HeapAlloc`: bytes in spans that *do* hold live objects but are not themselves occupied. Because spans are dedicated to a size class, this is an upper bound on internal fragmentation. It is usually small and usually not your problem, but it explains why the live heap can be a few percent below the memory dedicated to it. ## How to read them `runtime.ReadMemStats(&m)` fills the struct. Two properties of that call matter operationally: - Its numbers are **up to date as of the call**, in contrast with a heap profile, which is a snapshot as of the most recently completed GC cycle. When a profile and MemStats disagree about the live heap, the profile is usually the stale one. - It **stops the world** to take a consistent snapshot. The pause is short, but it is a real pause, and calling it inside a request handler or a tight loop adds latency spikes to the very service you are debugging. Sample it every few seconds from a dedicated goroutine and export the result. ## Reading them against RSS Put the numbers in a row and the shape of the problem is usually obvious: - `HeapAlloc` high and rising → a real leak, in your code's object graph. Go to a heap profile and find the allocation site. - `HeapAlloc` low, `HeapIdle - HeapReleased` high → nothing leaked; the runtime is sitting on pages from an earlier peak. Attack the peak, not the steady state. - `HeapAlloc` low, `HeapIdle - HeapReleased` low, `Sys` low, but RSS still high → the memory is not in the Go heap at all. Look at goroutine stacks, at cgo, at explicit mappings, or at the kernel's own accounting of your process. Each of the three has a different fix, and none of the three is discoverable from the memory graph alone.
- Why is runtime.MemStats.Sys usually larger than the process's RSS?Because `Sys` counts virtual address space the runtime obtained from the operating system, not physical pages currently backing it. Address space that was reserved but never touched, or whose pages were released back to the kernel, still contributes to `Sys`. It is therefore an upper bound on the Go runtime's contribution to RSS — useful as a ceiling, useless as a measurement of resident memory.
- What does a large HeapIdle minus HeapReleased tell you about the recent past?That the live heap spiked recently and has since shrunk. The runtime sized its arena for the peak, the collector reclaimed the objects, and the empty spans were kept mapped so the heap can grow again without a syscall. It is evidence about peak memory, not about a leak, and the fix is to lower the peak rather than to chase the current live set.
- Why should runtime.ReadMemStats not be called on every request?It stops the world to take a consistent snapshot. The pause is brief, but calling it per request turns a diagnostic into a latency source, and under load the stop-the-world cost compounds. Sample it from one goroutine every few seconds and export the fields you care about.
saying these in an interview costs you the question
- Reads Sys as the process's resident memory
- Thinks HeapIdle means memory already given back
- Calls ReadMemStats per request without noticing the pause
- Believes HeapAlloc includes goroutine stacks
- Cannot name a derived number, only individual fields