`slices.Collect` over an unbounded sequence OOMs a service — how do you confirm that and fix it?

seniorshould knowfreq 34%

answer

the helper is eager, the source is not
cost scales with elements yielded
growth copies, so the peak is worse
live heap, not cumulative allocation
bound it at the call site or upstream

basics

~20 s

slices.Collect drains the whole sequence into one slice, so memory grows with the number of elements yielded and the iterator's laziness is discarded. Confirm with a heap profile: inuse_space dominated by the growing backing array under that call. Fix by streaming or capping instead of collecting.

solid answer

~50 s

slices.Collect is not lazy: it drives the sequence to exhaustion and appends every element into one slice, so its cost is O(total yielded) and the resulting backing array stays live as long as the caller holds it. Over a sequence with no natural end — an in-process read model that keeps yielding cached records — that turns a bounded iterator into an unbounded allocation, and the kernel eventually kills the process. Confirm it rather than guess: take a heap profile from net/http/pprof or go test -memprofile, open it with go tool pprof, and look at inuse_space — you will see growslice under slices.AppendSeq under your collect call site holding most of the live heap. The fix is to stop materialising: range the sequence and write each record straight to the response, or append until a hard row cap and break, or push the limit down into the source that produces the sequence.

`slices.Collect` is not lazy: it drives the sequence to exhaustion and appends every element into one slice, so its cost is O(total yielded) and the resulting backing array stays live as long as the caller holds it. Over a sequence with no natural end — an in-process read model that keeps yielding cached records — that turns a bounded iterator into an unbounded allocation, and the kernel eventually kills the process. Confirm it rather than guess: take a heap profile from `net/http/pprof` or `go test -memprofile`, open it with `go tool pprof`, and look at `inuse_space` — you will see `growslice` under `slices.AppendSeq` under your collect call site holding most of the live heap. The fix is to stop materialising: range the sequence and write each record straight to the response, or append until a hard row cap and break, or push the limit down into the source that produces the sequence.

code

go · 14 lines

// Before: materialises every cached record into one slice.
records := slices.Collect(cache.Records()) // iter.Seq[Record]
render(records)

// After: keep at most one page, pre-sized so the slice never grows.
const maxRows = 5000
rows := make([]Record, 0, maxRows)
for r := range cache.Records() {
	if len(rows) == maxRows {
		break
	}
	rows = append(rows, r)
}
render(rows)

go deeper

for a junior

Know that slices.Collect walks the whole sequence and keeps every element, so it is only safe when you know the sequence is short.

for a middle

Explain why memory scales with the elements yielded and why the peak is higher still, since the slice reallocates and copies as it grows. Name a bounded alternative such as stopping the loop at a row cap.

for a senior

Walk the diagnosis end to end: heap profile, inuse_space, the growslice-under-Collect stack that names the call site, then a fix that removes the materialisation rather than raising a limit. Say why AppendSeq into a pre-sized slice does not cap anything.

for a principal

Own the shape of the boundary: an API that can hand out an unbounded sequence will eventually be collected by somebody, so the limit belongs in the producer's contract, and the cost of that change is weighed against paging every caller.

What the failure actually is

An admin listing is backed by an in-process read model: a map keyed by a composite struct key, holding cached records, exposed to callers as an iter.Seq[Record]. The handler does the obvious thing:

records := slices.Collect(cache.Records())

The iterator was the memory-safe part of the design — it yields one record at a time and holds nothing — and slices.Collect throws that away in a single call. Collect drains the sequence to exhaustion and appends every element into one slice, so:

Live memory is proportional to the number of records yielded, not to what the page will display.
Peak memory is worse than that. The slice grows by reallocating a larger backing array and copying, so at each growth step both the old and the new array are live at once. Add whatever the records themselves point at — strings, nested slices — and the collected slice keeps that whole object graph reachable.
The garbage collector cannot help. Nothing here is garbage; the handler is holding a live reference to every record. RSS climbs, the collector works harder for nothing, and eventually the OOM killer takes the process.

The class of bug: a lazy producer composed with an eager consumer. The laziness only survives if nothing in the chain materialises.

Confirming it, not guessing

Guessing here is expensive because the same symptom — climbing RSS ending in a kill — also comes from leaked goroutines, an unbounded cache, or a connection pool. Get the evidence:

Take a heap profile. In a long-running service, import net/http/pprof and fetch /debug/pprof/heap while RSS is high; in a reproduction, go test -memprofile mem.out -bench ..
Read inuse_space, not alloc_space. go tool pprof -inuse_space shows what is live now, which is the question you are asking. alloc_space shows cumulative allocation since start and will happily blame a hot path that allocates constantly but retains nothing.
Look at the call stack under the growth. The tell-tale shape is runtime.growslice beneath slices.AppendSeq beneath slices.Collect beneath your handler, holding the large majority of the live heap. That names the call site with no ambiguity.
Cross-check the count. If inuse_space for that stack scales with how much the read model holds rather than with concurrent requests, it is a materialisation problem, not a concurrency one.

A GODEBUG=gctrace=1 line will confirm the heap goal climbing without ever coming back down, which is useful corroboration but does not name a call site — the heap profile does.

Fixing it

In rough order of preference:

Do not collect at all. If the records are being rendered or written out, range the sequence and encode each record as it arrives. Peak memory becomes one record plus the output buffer.
Cap what you keep. If a slice really is needed, append with a hard limit and stop:

rows := make([]Record, 0, maxRows)
for r := range cache.Records() {
	if len(rows) == maxRows {
		break
	}
	rows = append(rows, r)
}

Pre-sizing with make(..., 0, maxRows) also removes the growth-and-copy peak.

Push the limit down to the source. Give the read model a method that yields at most N records, or a paging cursor. The best bound is the one applied before the data is produced.
Reuse a buffer with slices.AppendSeq only if you have also bounded the sequence — AppendSeq appends everything the sequence yields, so a pre-sized destination does not cap it. This is the fix that looks right and is not.
Keep less per record. If the records are wide and only three fields are shown, collect a projection struct instead of the full record; the backing array shrinks by whatever you stop retaining.

What to take away

slices.Collect, slices.Sorted and maps.Collect are all eager and all unbounded — they are conveniences for sequences you know are small. Treat every collect call over a sequence whose length you do not control as an allocation of unknown size, and make the bound explicit at the call site or upstream of it. In review, the question to ask about any slices.Collect is simply: what is the largest number of elements this sequence can yield?

## What the failure actually is An admin listing is backed by an in-process read model: a map keyed by a composite struct key, holding cached records, exposed to callers as an `iter.Seq[Record]`. The handler does the obvious thing: ```go records := slices.Collect(cache.Records()) ``` The iterator was the memory-safe part of the design — it yields one record at a time and holds nothing — and `slices.Collect` throws that away in a single call. `Collect` drains the sequence to exhaustion and appends every element into one slice, so: - **Live memory is proportional to the number of records yielded**, not to what the page will display. - **Peak memory is worse than that.** The slice grows by reallocating a larger backing array and copying, so at each growth step both the old and the new array are live at once. Add whatever the records themselves point at — strings, nested slices — and the collected slice keeps that whole object graph reachable. - **The garbage collector cannot help.** Nothing here is garbage; the handler is holding a live reference to every record. RSS climbs, the collector works harder for nothing, and eventually the OOM killer takes the process. The class of bug: a lazy producer composed with an eager consumer. The laziness only survives if nothing in the chain materialises. ## Confirming it, not guessing Guessing here is expensive because the same symptom — climbing RSS ending in a kill — also comes from leaked goroutines, an unbounded cache, or a connection pool. Get the evidence: 1. **Take a heap profile.** In a long-running service, import `net/http/pprof` and fetch `/debug/pprof/heap` while RSS is high; in a reproduction, `go test -memprofile mem.out -bench .`. 2. **Read `inuse_space`, not `alloc_space`.** `go tool pprof -inuse_space` shows what is *live now*, which is the question you are asking. `alloc_space` shows cumulative allocation since start and will happily blame a hot path that allocates constantly but retains nothing. 3. **Look at the call stack under the growth.** The tell-tale shape is `runtime.growslice` beneath `slices.AppendSeq` beneath `slices.Collect` beneath your handler, holding the large majority of the live heap. That names the call site with no ambiguity. 4. **Cross-check the count.** If `inuse_space` for that stack scales with how much the read model holds rather than with concurrent requests, it is a materialisation problem, not a concurrency one. A `GODEBUG=gctrace=1` line will confirm the heap goal climbing without ever coming back down, which is useful corroboration but does not name a call site — the heap profile does. ## Fixing it In rough order of preference: - **Do not collect at all.** If the records are being rendered or written out, range the sequence and encode each record as it arrives. Peak memory becomes one record plus the output buffer. - **Cap what you keep.** If a slice really is needed, append with a hard limit and stop: ```go rows := make([]Record, 0, maxRows) for r := range cache.Records() { if len(rows) == maxRows { break } rows = append(rows, r) } ``` Pre-sizing with `make(..., 0, maxRows)` also removes the growth-and-copy peak. - **Push the limit down to the source.** Give the read model a method that yields at most N records, or a paging cursor. The best bound is the one applied before the data is produced. - **Reuse a buffer with `slices.AppendSeq`** only if you have also bounded the sequence — `AppendSeq` appends everything the sequence yields, so a pre-sized destination does not cap it. This is the fix that looks right and is not. - **Keep less per record.** If the records are wide and only three fields are shown, collect a projection struct instead of the full record; the backing array shrinks by whatever you stop retaining. ## What to take away `slices.Collect`, `slices.Sorted` and `maps.Collect` are all eager and all unbounded — they are conveniences for sequences you know are small. Treat every collect call over a sequence whose length you do not control as an allocation of unknown size, and make the bound explicit at the call site or upstream of it. In review, the question to ask about any `slices.Collect` is simply: what is the largest number of elements this sequence can yield?

Why look at inuse_space rather than alloc_space in the heap profile?
`inuse_space` reports memory that is still live at the moment the profile was taken, which is exactly what an OOM is about. `alloc_space` reports everything allocated since the process started, so a hot path that churns short-lived buffers dominates it while retaining nothing. Diagnosing retention with `alloc_space` sends you after the wrong function.
Would pre-sizing the destination with slices.AppendSeq have capped the memory?
No. `slices.AppendSeq(dst, seq)` drains the whole sequence and appends every element, growing past `dst`'s capacity exactly like `append` does. Pre-sizing only removes the grow-and-copy churn; the bound has to come from stopping the loop or from the source yielding fewer elements.
Is slices.Sorted safe here if you only need the top ten records?
No — it is worse. `slices.Sorted` collects the entire sequence into a slice first and only then sorts it, so it has the same unbounded materialisation plus a full sort. For a top-ten you keep a bounded structure of ten while streaming, or push the ordering and limit down to whatever produces the sequence.
How would you keep this from coming back in review?
Make the bound part of the API rather than a discipline: have the read model expose a method that yields at most N records, or return a paging cursor, so there is no unbounded sequence to collect. Failing that, the review question on every `slices.Collect` is what the maximum element count is, answered in a comment at the call site.

saying these in an interview costs you the question

Says slices.Collect stays lazy until the slice is read
Blames the garbage collector for not freeing live data
Diagnoses retention from alloc_space instead of inuse_space
Pre-sizes the destination and calls the memory bounded
Raises GOMEMLIMIT instead of removing the materialisation

`slices.Collect` over an unbounded sequence OOMs a service — how do you confirm that and fix it?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

What the failure actually is

Confirming it, not guessing

Fixing it

What to take away

▾common follow-ups4 questions

▾red flags5 flags

What the failure actually is

Confirming it, not guessing

Fixing it

What to take away

answer

go deeper