A Go RPC service has p99 spikes and runtime.gcAssistAlloc frames in its CPU profile — what is happening?

### Reading the profile `runtime.gcAssistAlloc` in a CPU profile means one specific thing: the goroutine on that stack was made to perform garbage-collection marking itself, inside the allocator, before its allocation returned. Underneath it you will typically see the marking routines (`runtime.gcAssistAlloc1`, `runtime.gcDrainN`); above it you will see `runtime.mallocgc` and then whatever application function allocated — a decode step, a struct literal, an `append` that grew, a map insert. That stacking is the whole reason this failure is confusing. The assist time is charged to *your* function, so the profile says "decoding is expensive" when decoding did not change. What changed is the price of allocating. ### The mechanism behind the symptom Go's collector marks concurrently, funded by background mark workers capped at roughly a quarter of the process's Ps. When the program allocates faster than those workers can mark, the runtime does not raise the collector's budget; it raises the price of allocation. Each allocating goroutine is charged scan work in proportion to the bytes it allocates and performs that work inline. A high-fan-out read path is the classic generator of this. One inbound request produces many downstream calls, each of which decodes a response into a fresh set of small short-lived structs. The bytes per request are large, the objects are pointer-bearing (so they add scan work as well as heap), and they die almost immediately — meaning they contribute nothing to the live set but everything to the allocation rate the collector is racing. ### Why the tail moves and the mean does not Assists only occur while a mark phase is running, and only while background marking is behind. That is a minority of wall-clock time. A request that arrives outside that window pays nothing; a request that arrives in the middle of it can pay a great deal. On top of that, an assisting goroutine deliberately scans a larger chunk than it owes and banks the surplus as credit, so cost arrives in lumps rather than smoothly. The result is a latency distribution with a normal body and a heavy tail — exactly the shape that generates a customer complaint about p99 while every dashboard of average latency looks healthy. ### Why the endpoint at the top of the profile is often innocent Allocation rate is a property of the whole process, but assists are billed at the allocation site. Whichever code path happens to be allocating when the collector falls behind pays, regardless of who created the backlog. A cheap endpoint co-resident with an expensive one will show assist frames and elevated p99 that it did nothing to earn. The practical consequence: do not optimise the function at the top of the assist profile. Find the allocation *rate* — which call paths produce the most bytes per second across the whole process — and attack that. ### Confirming the diagnosis - Measure the share of the CPU profile that sits under `runtime.gcAssistAlloc`. A few percent is background noise; a double-digit share is the story. - Look at which call paths feed it. Those are the allocation sites paying the bill, not necessarily the ones creating it. - Check whether the elevated latency correlates with mark phases rather than with request content. If two requests with identical inputs differ by an order of magnitude, timing, not logic, is the variable. - Distinguish it from stop-the-world time, which is not attributed to your functions at all and hits every in-flight request in the same short window whether or not it was allocating. - Watch for the silent variant: a goroutine that owes assist work but can find neither credit nor available marking work parks until credit is released. That shows as latency with no matching CPU anywhere. ### What actually fixes it Every durable fix reduces one of the two inputs to the assist charge — bytes allocated, or scan work created. - **Allocate fewer bytes per request.** Reuse buffers the handler already owns across the items it processes, decode into a preallocated slice rather than appending into a fresh one per item, and avoid intermediate copies of payloads that are read once. - **Reduce pointer density.** Assist debt is repaid in *scanning*, and scanning follows pointers. A `[]Item` of pointer-free structs is one object the collector can skip past; a `[]*Item` is one edge per element to walk. Flattening pointer-heavy types lowers the scan work per live byte, which lowers the ratio the collector needs to charge. - **Cap fan-out concurrency.** Letting one request launch an unbounded number of concurrent downstream decodes concentrates allocation into a burst, which is the worst possible shape for a pacer trying to stay ahead. A bounded worker count smooths it. - **Shorten the live set the decode produces.** Objects that die immediately still cost allocation, but objects that survive cost scanning on every subsequent cycle too. What does *not* fix it: adding goroutines (the allocation is the same, now more concurrent), or blaming the decoder's logic and micro-optimising code that was never slow.