In a Go service, throughput plateaus while CPU sits near 30% — how do block and mutex profiles find the lock?

## Reading the symptom first "Throughput will not climb, CPU is well under the quota" is a specific signature. It rules out being compute-bound and it rules out the scheduler being saturated. Work is arriving and something is making the goroutines that would do it stop. In Go this looks idle because of how the runtime handles a contended lock: a goroutine that cannot take a `sync.Mutex` spins only very briefly and is then parked, its P handed to something else. A serialised program therefore does not burn CPU the way a spinlock-based one would. Low CPU is evidence *for* contention, not against it. ## The shape that produces it The common cause is a piece of shared state that every request touches. A middleware layer wrapping every handler, holding one in-process limiter and a small struct of counters behind a single mutex, is enough: ```go type stats struct { mu sync.Mutex counts map[string]int } func (s *stats) record(route string) { s.mu.Lock() defer s.mu.Unlock() s.counts[route]++ } ``` The critical section is a map increment — microseconds at worst. It does not matter. If every request passes through it, the whole service is a queue for one lock, and its ceiling is one over the lock's service time regardless of how many cores you add. ## The measurement 1. **Enable both profilers on one instance**, at coarse rates: a mutex fraction in the tens or hundreds, a block rate in the tens of microseconds. Coarse sampling is weighted towards long waits, so it preserves the ranking you care about. 2. **Take a delta.** Both profiles accumulate for the life of the process, so a single capture from a week-old process is dominated by history. Capture, wait a fixed interval under load, capture again, and work with the difference. 3. **Start with the mutex profile.** It fires only on genuine contention and records the **holder's** stack at the release, so the top entry by delay points straight at `record` — the critical section whose holding time cost everyone else. Had you started with the block profile, the same contention would be spread across every handler that happened to be waiting. 4. **Cross-check with the block profile.** This is where you separate the two kinds of waiting the symptom could be: waiting for a lock, or waiting on a channel. A limiter implemented as a token channel, or a handoff to a bounded queue, shows up only in the block profile. If the block profile's request-path stacks are dominated by a channel receive rather than by `Lock`, your ceiling is the limiter's capacity, not a critical section — a completely different fix. 5. **Read count against delay.** Enormous contentions count with modest per-event delay is the "everything goes through one lock" shape. A small count with huge delay is the "someone holds the lock across an RPC or a disk write" shape. ## The block profile's built-in noise Sort a block profile by total delay and the top entries are usually background goroutines parked in a `select`, waiting for work they are supposed to be waiting for. That is not contention and it is why the profile has a reputation for being useless. The discipline is to look only at stacks that lie on the request path and to use the mutex profile as the arbiter for lock questions. An entry in the mutex profile always represents time a goroutine genuinely lost to another goroutine's lock. ## What the two profiles will not tell you - Time in syscalls and network I/O appears in neither. If both profiles are quiet and throughput is still capped, look outside the runtime's synchronisation: a connection pool with a small maximum, a downstream service, or a per-instance concurrency limit. - Contention on locks internal to the runtime or to a library that does not use `sync.Mutex` will not necessarily show as a mutex sample. - Neither profile tells you the fix. They tell you which critical section is expensive; whether the answer is to shard the state, to use an atomic counter, to accumulate per-request and flush periodically, or to remove the shared state entirely, is a design call. ## Closing the loop After the change, repeat the same delta capture under the same load. The mutex profile's top entry should shrink and something else should become the ceiling — usually the next-largest shared thing. Contention work is iterative, and the profiles are how you tell you moved the bottleneck rather than merely moved the code. ## What a strong answer sounds like Name the symptom's meaning (parked, not spinning), enable both profilers deliberately at coarse rates on one instance, diff two captures, use the mutex profile to name the critical section and the block profile to rule channels in or out, and read count and delay together before proposing a fix.