Your Go http.Server's goroutine count climbs linearly under a slow-header probe while RPS stays flat — what is happening and which field fixes it?
answer
- goroutines track connections, not requests
- flat RPS means nothing was ever dispatched
- the goroutine is stuck before the handler exists
- the write deadline never armed for these
- the header phase has its own field
basics
~10 sEach connection the probe opens parks a per-connection goroutine blocked reading request headers that never complete, so nothing is dispatched and RPS stays flat while goroutines accumulate. Set http.Server.ReadHeaderTimeout, and IdleTimeout alongside it.
solid answer
~50 snet/http serves every accepted connection on its own goroutine, and that goroutine blocks reading the request line and headers before any handler exists. A probe that opens connections and trickles header bytes leaves each of those goroutines parked indefinitely, which is why the goroutine count tracks open connections linearly while request throughput does not move at all — no request has completed, so nothing has been served. `WriteTimeout` does not help, because its clock only starts once a request's headers have been read. Handler-side cancellation does not help either: the handler never runs and there is no request context yet. The field that bounds this phase is `ReadHeaderTimeout`, or `ReadTimeout`, which it falls back to. Confirm the diagnosis with the goroutine profile from net/http/pprof, which will show thousands of stacks parked in net/http's connection-serving path rather than in your own code.
go deeper
Know that a Go server runs one goroutine per accepted connection, so a rising goroutine count can mean open connections rather than in-flight work.
Explain why the goroutine is parked before any handler runs, and why the response-side and handler-side controls have no effect on a request that never finished arriving.
Read the two signals together, rule out the handler-leak explanation from the goroutine profile's stacks, and name the header deadline plus the idle deadline as the change you ship.
Insist that a slow-client probe and a goroutine-count assertion live in the load-test suite, so an unbounded server is caught by the pipeline rather than by whoever is on call.
## The shape of the signal Two lines on a dashboard tell the whole story: goroutine count rising in a straight line, request rate flat. Anything that rises with *load* would move both. A count that rises while throughput does not is counting something that is not requests — here, connections that never became requests. ## Why one connection costs one goroutine net/http's accept loop hands each accepted connection to a goroutine dedicated to serving it. That goroutine's first act is to read the request line and headers from the connection. Reading blocks. Until a complete header block arrives, there is no `http.Request`, no route match, no handler, and no request context. The goroutine sits in that read for exactly as long as the read deadline allows — and with no read deadline configured, that is forever. So the resources a stalled connection holds are: a goroutine with its stack, the buffered reader and writer attached to the connection, and a file descriptor. None of it is a leak in the sense of a bug; the runtime is doing precisely what it was configured to do. It is a leak in the sense that matters operationally, because nothing will ever release it. This is cheap to cause and expensive to absorb. A probe — or an unfriendly client — that opens connections and sends one header byte every few seconds keeps them all alive at almost no cost to itself. On a public endpoint exposed directly to untrusted clients, there is no reason to expect anything else. ## Why the obvious fixes do not apply **`WriteTimeout`** is reset whenever a new request's header is read. In this scenario no request's headers have ever finished being read, so no write deadline is ever armed for them. A server that sets only `WriteTimeout` is entirely unprotected here. **`IdleTimeout`** governs the wait for the *next* request on a connection that already completed one. These connections never completed one, so it does not apply to their initial stall — although you want it set anyway, for the phase it does cover. **Handler-side cancellation** — a `context.WithTimeout` at the top of the handler, or checking `r.Context()` — cannot help by construction. The handler never runs. Anything written inside `ServeHTTP` is downstream of the phase that is stuck. **A proxy in front** helps only if every path to the process goes through it. A public endpoint exposed directly does not have that property, and even where a proxy exists it is a mitigation you do not own rather than a bound on your own process. ## The fix Set `ReadHeaderTimeout` to a small number of seconds. Nothing legitimate needs longer to send headers, and this is the only field that bounds a connection before dispatch. Set `IdleTimeout` in the same change, because the two together are what bound a connection outside a request. If `ReadTimeout` is set and `ReadHeaderTimeout` is not, the header phase inherits `ReadTimeout` and you are already protected, just with a much looser number than the phase deserves. When the deadline expires, the read fails and the connection is closed. There is no status code — there is no request to answer — and the goroutine returns and is collected. That is the whole mechanism. ## Confirming it rather than guessing The goroutine profile is the direct evidence. With net/http/pprof registered, the goroutine profile lists every live goroutine grouped by stack; a stalled-connection incident shows thousands of identical stacks sitting in net/http's connection serve and header read path, with none of your own package's frames in them. That distinguishes this from the other reason goroutine counts climb — a handler leaking goroutines per request — whose stacks are unmistakably in your code, and which would climb with request rate rather than independently of it. A useful second signal is open file descriptors or established connections for the process, which should climb in lockstep with the goroutine count. If goroutines climb and connections do not, look at the handler instead. ## The follow-through The interesting part of this answer, for a senior chair, is not naming the field. It is that the probe should exist. A load test that only drives well-behaved fast clients will never surface this, because every request completes and the goroutine count sits flat under any configuration. A slow-header probe belongs in the same suite as the throughput test, with the goroutine count asserted as a bound rather than eyeballed, so the day someone constructs an `http.Server` without timeouts the suite says so rather than production doing.
- Why does a context.WithTimeout inside the handler not fix this?Because the handler is never reached. The goroutine is blocked reading the request headers, and until those are complete there is no request, no route match and no request context. Every handler-side control is downstream of the phase that is stuck.
- How would you tell this apart from a handler leaking a goroutine per request?By the stacks and by the correlation. The goroutine profile here shows thousands of identical stacks inside net/http's connection serving path with none of your frames; a handler leak shows your own package. And a handler leak climbs with request rate, whereas this climbs while request rate is flat.
- The connection is closed when the deadline expires. What status does the client get?None. There was never a complete request to respond to, so the server closes the socket and the client sees a reset. That is the right outcome here, and it is a reminder that these fields are transport-level controls that cannot express an HTTP status.
- What would you add to the load-test suite so this cannot regress?A slow-client probe that opens connections and trickles header bytes, run alongside the throughput test, with the process goroutine count asserted to stay under a bound. A well-behaved load generator will never reproduce the failure, which is why the probe has to be deliberate.
saying these in an interview costs you the question
- Blames the handler despite request rate staying flat
- Reaches for WriteTimeout to bound an incomplete header read
- Proposes a handler-side context timeout for a pre-dispatch stall
- Assumes an upstream proxy already bounds a directly exposed endpoint
- Calls it a runtime bug rather than a missing deadline