A crawler returns errors.Join over one failure per module and a heap profile shows it retaining hundreds of MB. Why?

## Why the error is the leak `errors.Join` allocates a slice sized to the non-nil arguments and stores them. Nothing about that is unusual — but an error value in Go is an interface holding a concrete value, and that concrete value may hold anything. `fmt.Errorf("decode %s: %w", path, err)` retains the formatted string and the wrapped cause. A custom error carrying `Body []byte` retains the body. An error that holds a `*http.Response` retains the response and, if the body is not drained and closed, potentially the connection state as well. Each of those is small on its own; multiplied by the number of failed items in a long crawl, and held for the whole crawl because the joined error accumulates until the function returns, they become the process's dominant allocation. The giveaway in a heap profile taken with `inuse_space` is that the retained bytes are not in the fetch path where you would expect them, but rooted at the slice inside the joined error, still reachable from a live local variable in the crawl function. Nothing has leaked in the C sense; the collector is doing exactly what it should, because your program still holds the reference. This is worse than an equivalent leak elsewhere for two reasons. First, it grows with the *failure* rate, so it appears during precisely the incident you are trying to diagnose — the registry is down, every fetch fails, and the crawler that normally holds twelve errors now holds a hundred thousand. Second, it is invisible in review: the accumulate-then-join shape is the recommended idiom, and nothing in it says "bounded". ## Shrinking the causes The first fix is to make each retained error small and self-describing. An error that a human will read needs an identity and a reason, not evidence: - Wrap with the module path and the operation, not the payload: `fmt.Errorf("fetch %s: %w", path, err)`. - Never capture a large `[]byte`, a decoded document, or a response value in a long-lived error. If you need a sample for debugging, truncate it to a bounded prefix at the point of failure. - Make sure the response body is drained and closed on the failure path, so the error does not keep transport resources alive alongside the message. ## Capping the count The second fix is a ceiling. Keep the first N causes — N in the tens, not thousands — and replace the tail with a single counted summary: ```go const maxKept = 20 if len(errs) > maxKept { extra := fmt.Errorf("and %d more modules failed", len(errs)-maxKept) errs = append(errs[:maxKept:maxKept], extra) } return errors.Join(errs...) ``` If the causes cluster — and during an incident they nearly always do — grouping is better than truncating: count occurrences per classified reason and emit one cause per class, `"142 modules: connection refused"`. That preserves the information a reader actually wants while retaining a constant number of values. Everything you drop should still be *observed*: log or count each failure as it happens, so the error is a summary and the log is the record. An error is a return value, not a storage medium. ## The count that lies A second, quieter defect lives in the same code. Because Join drops nil arguments, a joined error's causes count the failures, not the attempts, and there is no way to recover the attempts from the value. Two anti-patterns follow: - **Splitting the message on newlines to count failures.** A cause whose own message contains a newline inflates the count, and a single cause is textually identical to the bare error. Read the causes properly instead, by asserting for `interface{ Unwrap() []error }`. - **Reporting "n failures" from a value you have capped.** Once you truncate or group, the number of causes is no longer the number of failures, so carry the real counters yourself — attempted, succeeded, failed — and let the error carry only the human-readable summary. ## Doing it without changing behaviour If the crawler's contract is that callers can `errors.Is` the result against specific sentinels, capping must preserve that. Two guards: make sure at least one cause of each *distinct* class survives the cap, so no matchable condition disappears; and keep the wrap chain intact on the causes you retain, using `%w` rather than `%v`, so the sentinels remain reachable. Then confirm with a benchmark run with `-benchmem` over a synthetic all-failing input, and a heap profile of the same run, that the retained bytes are now flat in the number of failures rather than linear in it.