A Go sidecar with GOMAXPROCS=64 runs under a half-core quota and stalls in bursts. How do you confirm the cause?

## The setup A sidecar runs beside every pod in the fleet. It hashes fixed-size chunks of data — CPU-bound, embarrassingly parallel work. Its CPU limit is deliberately small, half a core, because it should be cheap. The nodes have 64 CPUs. The symptom is not a high average: it is bursts. Throughput is fine when measured over a minute, and the p99 is dreadful. ## Step 1: what does the runtime think it has? The first two numbers cost nothing and settle most of the question: - `runtime.NumCPU()` — the logical CPUs the process can see. Here, 64. - `runtime.GOMAXPROCS(0)` — the Ps the runtime chose. Here, also 64. Against a 0.5-CPU grant, that is a 128-fold mismatch between what the runtime will attempt and what the kernel will pay for. Log both at start-up in every service; a value you have to go and measure during an incident is a value nobody checks. If GOMAXPROCS is 64 on a modern toolchain, ask why the container-aware default did not apply. The three usual answers: the `go` directive in `go.mod` still names a pre-1.25 version, so the old default is in force; the code calls `runtime.GOMAXPROCS(runtime.NumCPU())` somewhere in start-up; or the deployment injects a `GOMAXPROCS` environment variable computed from something other than the limit. ## Step 2: confirm the kernel is throttling A large P count is a strong hypothesis, not proof. The cgroup keeps the evidence itself. Under cgroup v2, `cpu.stat` inside the container reports `nr_periods` (how many quota periods have elapsed), `nr_throttled` (how many of those ended with the group frozen) and `throttled_usec` (how long it spent frozen in total). What you are looking for is a ratio, not an absolute: `nr_throttled` close to `nr_periods` means the group is running out of allowance in nearly every period. Sampling those counters over a minute and dividing gives you throttled time per second of wall clock — a number you can put next to the latency spikes and show that they move together. ## Step 3: rule out the other explanations The reason this diagnosis is worth practising is that throttling *looks like* other things: - **Slow code.** A genuine hot spot degrades latency smoothly with load; throttling produces a cliff aligned with the quota period, and the process is idle — not busy — during the stall. - **Lock contention.** Contention shows up as work in progress that is not advancing; throttling shows up as no work in progress at all, with the CPU accounting frozen. - **A dependency being slow.** Throttling stalls hit CPU-bound stretches, including the stretch between an upstream response arriving and it being processed, which makes downstreams look slow when they are not. The distinguishing observation is always the same: the process is not running, and the reason it is not running is that its cgroup has no CPU allowance left in this period. ## Why a large GOMAXPROCS makes it worse rather than better With N Ps, N goroutines run in parallel and the quota drains roughly N times faster. Halving the period's useful work into the first few milliseconds and freezing for the remainder is worse for latency than spreading the same work across the whole period, even though the total CPU consumed is identical. Two secondary costs pile on: - The garbage collector sizes its dedicated mark workers as a fraction of GOMAXPROCS (roughly a quarter), so an inflated value means a large slice of an already tiny allowance goes to collection whenever a cycle runs. - Each P carries per-P runtime state and caches, so an inflated value costs memory in a container that is usually memory-limited too. ## The fix, in order of preference 1. Let the runtime do it — a toolchain and a `go.mod` `go` directive recent enough for the container-aware default, and no code or environment override fighting it. 2. If something must pin the value, derive it from the container's CPU limit rather than from `runtime.NumCPU()`, and log it. 3. Only then consider changing the limit itself — but that is a capacity conversation, and it should follow the runtime being sized correctly, not substitute for it. Verify the same way you diagnosed: after the change, GOMAXPROCS should be a small number, and the throttled-period ratio should fall towards zero with the tail latency.