A log shipper keeps each parsed line as a subslice of the 1 MB buffer it was read from, and heap use climbs steadily. Why?

## The mechanism A slice value is a window: a pointer into a backing array, a length and a capacity. Reachability in Go is by allocation, not by window. If any slice, anywhere, points into an array, that entire array is live and the garbage collector will not reclaim any of it. There is no way to free "the part you are not using" — the collector is not compacting, and it has no notion of a partially live allocation. So a log shipper that reads a 1 MB chunk, splits it into lines, and keeps `buf[start:end]` for the lines it cares about has, from the collector's point of view, kept the megabyte. Retained memory grows with the number of *buffers* still referenced, not with the number of *bytes* the program logically holds. One surviving line per buffer is enough to pin the buffer, which is what makes this leak-shaped rather than merely wasteful: a slow-growing set of retained lines drags an unbounded number of buffers along with it. ## Recognising it at 3am The symptom is a heap that grows with retained-object count and never comes back down after a GC cycle, on a program whose own accounting says it is holding very little. Take a heap profile and read the `inuse_space` view — the live bytes, not the cumulative `alloc_space` allocations. The signature of this bug is a mismatch between *where* the memory was allocated and *what* is keeping it alive: the profile blames the read path (the site that allocated the buffers), because that is where the surviving allocations were made, while the code that actually holds the references is the parsing or batching path. A profile of allocation counts looks entirely normal; only the live-bytes view shows a small number of huge, ancient allocations that will not die. Two confirmations are cheap. First, compare the live bytes attributed to buffer allocation against the number of items you believe you are retaining multiplied by their average length; a ratio of hundreds to one is the tell. Second, look at the retaining structure: if your batch, cache or queue holds `[]byte` values you never copied, you have found it without needing any further tooling. ## The fix Copy what you keep, at the moment you decide to keep it: ```go line := make([]byte, end-start) copy(line, buf[start:end]) batch = append(batch, line) ``` `copy` transfers `min(len(dst), len(src))` elements and returns that count, so the destination's *length* must be right — a destination made with `make([]byte, 0, n)` copies nothing. `append([]byte(nil), buf[start:end]...)` is an equivalent one-liner that allocates a fresh array because a nil slice has no capacity. If the value is textual and will not be mutated, converting to `string` copies too, and gives you an immutable value that is cheap to share. After the fix, each retained line owns a small allocation of its own, and every read buffer becomes garbage as soon as parsing of that chunk finishes — which also means the buffers can be reused rather than re-read. ## What does not fix it - **Capping capacity** with a full slice expression, `buf[start:end:end]`. That controls whether a later append can write into the buffer; it changes nothing about reachability. The array is still pointed at, so it is still live. - **Setting the buffer variable to nil** after parsing. The variable is not what keeps the array alive; the surviving subslices are. - **Calling the collector explicitly.** The memory is genuinely reachable; no amount of collection pressure can reclaim it, and forcing collection only burns CPU while the heap stays flat. - **Shrinking the read size.** Smaller buffers reduce the amplification factor but not the shape of the bug, and they cost more syscalls. ## The generalisation worth stating This is the retention half of a rule that has a mutation half. Handing out a subslice of a buffer you own shares two things you may not have intended: the *contents*, which the holder can read and write and which your next write can change under them; and the *lifetime*, which is now the buffer's, not the window's. Any API that returns a view into a reusable buffer should either be documented as valid only until the next call — the convention several stdlib readers follow, where the bytes handed back may be overwritten by the next read and must be copied to be retained — or should copy on the way out. Choosing to copy costs one allocation per retained item and buys you a lifetime you can reason about; on a hot path where nothing is retained, the view is the right call. What you cannot do is return the view and then keep it.