After a traffic spike, a Go packet server's heap sits at 4 GB with 400 MB live. Why won't it shrink?

seniorshould knowfreq 38%

answer

collecting is not releasing
the unit of release is not the object
nothing ever gets moved to close the gaps
one survivor keeps 8 KB reserved
compare live bytes against reserved-but-unused bytes

basics

~20 s

Freeing objects does not release memory. Only a span with no live objects at all returns to the page allocator, and Go's collector never moves objects, so one survivor pins its whole 8 KB span.

solid answer

~50 s

Sweeping frees object slots, not memory. A span returns to the heap's page allocator only when every slot in it is free, and because Go's collector never relocates objects, a single survivor pins the whole 8 KB span. A spike that filled millions of slots across dozens of size classes leaves a long tail of spans that are 1% live and 100% reserved, and no amount of collecting will consolidate them. Confirm it with runtime/metrics: if /memory/classes/heap/objects:bytes is flat at 400 MB across cycles it is not a leak; a large /memory/classes/heap/unused:bytes is span-level fragmentation, while a large /memory/classes/heap/free:bytes with a small /memory/classes/heap/released:bytes just means the background scavenger has not returned those pages yet. The fixes are on the allocation side -- bound peak concurrency, use uniform buffer sizes, keep long-lived data out of spans dominated by short-lived objects -- plus GOMEMLIMIT so the runtime works to a ceiling.

Sweeping frees object slots, not memory. A span returns to the heap's page allocator only when **every** slot in it is free, and because Go's collector never relocates objects, a single survivor pins the whole 8 KB span. A spike that filled millions of slots across dozens of size classes leaves a long tail of spans that are 1% live and 100% reserved, and no amount of collecting will consolidate them. Confirm it with `runtime/metrics`: if `/memory/classes/heap/objects:bytes` is flat at 400 MB across cycles it is not a leak; a large `/memory/classes/heap/unused:bytes` is span-level fragmentation, while a large `/memory/classes/heap/free:bytes` with a small `/memory/classes/heap/released:bytes` just means the background scavenger has not returned those pages yet. The fixes are on the allocation side -- bound peak concurrency, use uniform buffer sizes, keep long-lived data out of spans dominated by short-lived objects -- plus `GOMEMLIMIT` so the runtime works to a ceiling.

code

go · 10 lines

samples := []metrics.Sample{
	{Name: "/memory/classes/heap/objects:bytes"},
	{Name: "/memory/classes/heap/unused:bytes"},
	{Name: "/memory/classes/heap/free:bytes"},
	{Name: "/memory/classes/heap/released:bytes"},
}
metrics.Read(samples)
for _, s := range samples {
	fmt.Printf("%s %d\n", s.Name, s.Value.Uint64())
}

go deeper

for a junior

Know that freeing objects in Go does not immediately shrink the process: the runtime keeps spans for reuse and returns pages to the OS gradually.

for a middle

Explain the difference between sweeping and releasing -- a span becomes reusable as pages only when every object in it is free, and only then can it be scavenged.

for a senior

Show the diagnosis: compare live bytes against reserved-but-unused bytes to rule out a leak, then choose between bounding the peak, uniform buffer sizes and a memory limit.

for a principal

Decide what the service promises about memory -- a runtime-enforced ceiling, a restart policy, or capacity sized for peak reservation rather than steady-state live data -- and make that promise explicit.

Why a Go heap does not shrink back after a spike

This is one of the most common 3am misreadings in Go services: memory went up during a burst, the burst ended, and memory stayed up. The instinct is "leak". Usually it is not.

What sweeping actually frees

Garbage collection in Go marks live objects, then sweeps spans, marking dead object slots as free in the span's bitmap. Those free slots are immediately reusable for objects of that span's size class. Nothing is handed back to the operating system, and nothing is handed back to the page allocator either. The span is still a span of, say, the 48-byte class, and it still occupies its pages.

A span only goes back to the heap's page allocator when it has zero allocated objects. Then its pages join the free page ranges, and only then can the background scavenger tell the kernel that those pages are no longer needed.

Why one survivor is expensive

Go's collector is non-moving: an object keeps its address for its entire life. There is no compaction phase, no relocation, no defragmentation. The upside is that pointers never need fixing up, interior pointers are legal, and unsafe/cgo interop is straightforward. The downside is exactly this scenario: if a span of 170 slots holds one live object, the other 169 slots are reusable but the 8 KB stays reserved, and no collection cycle can migrate that survivor to consolidate.

Now scale it. During the spike, the service allocated across dozens of size classes at once -- per-datagram buffers, ids, per-connection state -- filling thousands of spans in each class. When the burst drains, survivors are sparse but spread, so a large number of spans are lightly occupied. Reserved memory reflects the peak; live memory reflects the present.

The two different "unused" numbers

runtime/metrics distinguishes them, and reading the wrong one sends you down the wrong path:

/memory/classes/heap/objects:bytes -- memory holding heap objects. Compare it across several cycles. Flat means no leak; monotonically climbing means find the retainer.
/memory/classes/heap/unused:bytes -- memory reserved in spans but not currently holding objects. This is the span-level fragmentation bucket. High here and flat objects is exactly the story above.
/memory/classes/heap/free:bytes -- completely free pages that have not been returned to the OS yet.
/memory/classes/heap/released:bytes -- pages already returned to the OS.

A big free with a small released is not fragmentation at all; it is the scavenger being deliberately gradual. It returns free page runs over time rather than instantly, because unmapping and re-faulting pages is expensive and a service that spiked once will often spike again.

What to do at 3am, and what to do afterwards

At 3am the questions are narrow. Is live memory flat? If yes, this is reserved memory and the process is not on a trajectory to die; the decision is whether the box has headroom until the next deploy. debug.FreeOSMemory forces a collection and an immediate return of free pages, which is a legitimate one-shot lever to buy room -- but it is a stop-the-world hit and calling it periodically is a smell, not a fix.

Afterwards the fixes are all on the allocation side, because the allocator itself is not configurable:

Bound the peak. The heap is sized by the peak, so limiting in-flight work -- a bounded number of concurrent datagram handlers, a queue with a ceiling -- limits the reservation. This is the single most effective change.
Allocate uniform sizes. Buffers that are all the same size land in one class, so their spans fill and drain together instead of leaving a sparse tail in fifteen classes.
Separate lifetimes. Long-lived data allocated in the middle of a burst is what pins spike-era spans. Building long-lived structures up front, or copying retained values into a compact structure you own, lets the burst-era spans empty completely.
Set GOMEMLIMIT. A soft limit makes the collector run more often and the scavenger work harder to stay under it. It does not compact anything and it will not save a process whose live set genuinely exceeds the limit -- then the runtime just collects continuously -- but it converts "grows until the kernel objects" into "works to a ceiling".

The sentence that settles the diagnosis

Reserved memory is a high-water mark of allocation; live memory is a measure of retention. In a non-moving, size-class allocator the first can only come down when whole spans empty. If live is flat and reserved is high, you are looking at the cost of your peak, and the fix is to make the peak smaller -- not to hunt for a leak that is not there.

## Why a Go heap does not shrink back after a spike This is one of the most common 3am misreadings in Go services: memory went up during a burst, the burst ended, and memory stayed up. The instinct is "leak". Usually it is not. ### What sweeping actually frees Garbage collection in Go marks live objects, then sweeps spans, marking dead object slots as free in the span's bitmap. Those free slots are immediately reusable **for objects of that span's size class**. Nothing is handed back to the operating system, and nothing is handed back to the page allocator either. The span is still a span of, say, the 48-byte class, and it still occupies its pages. A span only goes back to the heap's page allocator when it has **zero** allocated objects. Then its pages join the free page ranges, and only then can the background scavenger tell the kernel that those pages are no longer needed. ### Why one survivor is expensive Go's collector is **non-moving**: an object keeps its address for its entire life. There is no compaction phase, no relocation, no defragmentation. The upside is that pointers never need fixing up, interior pointers are legal, and `unsafe`/cgo interop is straightforward. The downside is exactly this scenario: if a span of 170 slots holds one live object, the other 169 slots are reusable but the 8 KB stays reserved, and no collection cycle can migrate that survivor to consolidate. Now scale it. During the spike, the service allocated across dozens of size classes at once -- per-datagram buffers, ids, per-connection state -- filling thousands of spans in each class. When the burst drains, survivors are sparse but *spread*, so a large number of spans are lightly occupied. Reserved memory reflects the peak; live memory reflects the present. ### The two different "unused" numbers `runtime/metrics` distinguishes them, and reading the wrong one sends you down the wrong path: - `/memory/classes/heap/objects:bytes` -- memory holding heap objects. Compare it across several cycles. Flat means no leak; monotonically climbing means find the retainer. - `/memory/classes/heap/unused:bytes` -- memory reserved in spans but not currently holding objects. This is the span-level fragmentation bucket. High here and flat objects is exactly the story above. - `/memory/classes/heap/free:bytes` -- completely free pages that have not been returned to the OS yet. - `/memory/classes/heap/released:bytes` -- pages already returned to the OS. A big `free` with a small `released` is not fragmentation at all; it is the scavenger being deliberately gradual. It returns free page runs over time rather than instantly, because unmapping and re-faulting pages is expensive and a service that spiked once will often spike again. ### What to do at 3am, and what to do afterwards At 3am the questions are narrow. Is live memory flat? If yes, this is reserved memory and the process is not on a trajectory to die; the decision is whether the box has headroom until the next deploy. `debug.FreeOSMemory` forces a collection and an immediate return of free pages, which is a legitimate one-shot lever to buy room -- but it is a stop-the-world hit and calling it periodically is a smell, not a fix. Afterwards the fixes are all on the allocation side, because the allocator itself is not configurable: - **Bound the peak.** The heap is sized by the peak, so limiting in-flight work -- a bounded number of concurrent datagram handlers, a queue with a ceiling -- limits the reservation. This is the single most effective change. - **Allocate uniform sizes.** Buffers that are all the same size land in one class, so their spans fill and drain together instead of leaving a sparse tail in fifteen classes. - **Separate lifetimes.** Long-lived data allocated in the middle of a burst is what pins spike-era spans. Building long-lived structures up front, or copying retained values into a compact structure you own, lets the burst-era spans empty completely. - **Set `GOMEMLIMIT`.** A soft limit makes the collector run more often and the scavenger work harder to stay under it. It does not compact anything and it will not save a process whose live set genuinely exceeds the limit -- then the runtime just collects continuously -- but it converts "grows until the kernel objects" into "works to a ceiling". ### The sentence that settles the diagnosis Reserved memory is a high-water mark of allocation; live memory is a measure of retention. In a non-moving, size-class allocator the first can only come down when whole spans empty. If live is flat and reserved is high, you are looking at the cost of your peak, and the fix is to make the peak smaller -- not to hunt for a leak that is not there.

How would you tell this apart from a genuine leak?
Watch live bytes, not total. Sample `/memory/classes/heap/objects:bytes` and `/gc/heap/objects:objects` across several collection cycles: fragmentation shows a flat live set under a high reservation, while a leak shows the live set climbing cycle after cycle. If live is climbing, the question stops being about the allocator and becomes about what is retaining the objects.
Why can't the collector fix this by moving live objects together?
Go's collector is non-moving by design. Objects keep their addresses for life, which is what makes interior pointers, `unsafe` and cgo interop workable and removes the need for read barriers on every pointer load. The price is that fragmentation cannot be compacted away, so it has to be avoided by allocation shape instead.
What does setting GOMEMLIMIT actually change in this situation?
It gives the runtime a soft ceiling on total memory, so the collector runs more often and the scavenger returns pages more aggressively as you approach it. It does not defragment anything and it is not a hard cap: if the live set alone nears the limit, the process spends its time collecting rather than working. Pair it with a bound on in-flight work.

A car park after a concert. Almost everyone has left, but one car parked on each level means no level can be closed off, so the site still occupies the same ground it did at capacity.

saying these in an interview costs you the question

Declares a leak without checking whether live bytes are flat
Expects freed objects to shrink the process immediately
Believes Go's collector compacts or relocates the heap
Proposes calling debug.FreeOSMemory on a timer as the fix
Blames size-class rounding waste for gigabytes of reservation

After a traffic spike, a Go packet server's heap sits at 4 GB with 400 MB live. Why won't it shrink?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Why a Go heap does not shrink back after a spike

What sweeping actually frees

Why one survivor is expensive

The two different "unused" numbers

What to do at 3am, and what to do afterwards

The sentence that settles the diagnosis

▾common follow-ups3 questions

▾analogy~20 s

▾red flags5 flags

Why a Go heap does not shrink back after a spike

What sweeping actually frees

Why one survivor is expensive

The two different "unused" numbers

What to do at 3am, and what to do afterwards

The sentence that settles the diagnosis

answer

go deeper