A Go service's outbound requests fail with "cannot assign requested address" and the socket table is full of TIME_WAIT entries to one host. How do you diagnose it?

## Reading the symptom Two signals arrive together and they say the same thing from different ends. `connect: cannot assign requested address` (Linux `EADDRNOTAVAIL`) on an outbound dial means the kernel could not find a free local port for the four-tuple. Because a client socket's tuple is (local IP, local port, remote IP, remote port), and here the remote pair is fixed, the usable space is the ephemeral port range — commonly around 28,000 ports. `TIME_WAIT` is the state the **actively closing** side holds for roughly two maximum segment lifetimes so that late duplicate segments cannot be mistaken for a new connection. Tens of thousands of `TIME_WAIT` sockets to one address means your process is opening and closing connections to that address at a very high rate — which is the definition of not pooling. So the diagnosis is not "we need more ports". It is "why is each request getting its own connection?" ## Confirm before you change anything Count the states, so you know the ratio rather than the anecdote: ```text ss -tan state time-wait dst 10.0.4.12 | wc -l ss -tan state established dst 10.0.4.12 | wc -l ``` A healthy pooled client shows a stable, smallish number of established connections and almost no `TIME_WAIT` to that host. A non-pooling client shows the inverse, and the `TIME_WAIT` count tracks your request rate times sixty seconds. Then prove it from inside the program. `net/http/httptrace` gives a per-request hook that reports exactly what the transport did: ```go trace := &httptrace.ClientTrace{ GotConn: func(info httptrace.GotConnInfo) { log.Printf("reused=%v wasIdle=%v idle=%v", info.Reused, info.WasIdle, info.IdleTime) }, } req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace)) ``` If `Reused` is false on every request, connection reuse is broken and the rest of the investigation is a code search. This is worth wiring behind a debug flag permanently; it is the difference between believing you pool and knowing it. ## The Go-side causes, in order 1. **A transport constructed per call.** `&http.Client{Transport: &http.Transport{}}` inside a helper or a handler gives every request a private, empty idle list. This is the classic cause and the one the symptom fits best, especially in an SDK where a `NewClient` helper is called per operation instead of per process. 2. **`Transport.DisableKeepAlives = true`.** Sometimes set during a past debugging session and never removed. It closes every connection after one exchange by design. 3. **`MaxIdleConnsPerHost` at its default of two** while running hundreds of concurrent requests to one host. You keep two warm and close the rest, so most requests still dial. 4. **Connections closed by the peer.** A server or intermediary with a short idle timeout, or one that sends `Connection: close`, will also produce churn — but then it is the *server* that closes and the `TIME_WAIT` sockets accumulate on its side, not yours. That asymmetry is a useful discriminator: `TIME_WAIT` on the client host points at client-side behaviour. 5. **Too many distinct destinations.** Pools are keyed per destination; if you address a downstream by rotating IPs or by unique hostnames, no pool ever warms up. ## The fix One long-lived `*http.Client` per dependency, held in package or struct state, with a cloned transport whose `MaxIdleConnsPerHost` matches sustained concurrency and whose `MaxIdleConns` is raised to match. In an SDK, that means the client is created in the constructor and reused for every call, never per operation. After the change, the same two measurements confirm it: `Reused` becomes true for the overwhelming majority of requests, `TIME_WAIT` collapses, and established connections settle at roughly your concurrency level. ## What not to reach for first Widening `net.ipv4.ip_local_port_range`, enabling `tcp_tw_reuse`, or setting `SO_REUSEADDR`-style options will buy headroom and will hide the defect. They are legitimate as a stopgap during an incident, and they are the right tool when the churn is genuinely unavoidable — but a service that burns 28,000 ports a minute against one host is nearly always a service that forgot to share its client. Fix the pooling, then decide whether the OS tuning is still needed. ## The review lesson This defect is invisible at low traffic: a per-call transport works perfectly in tests and in staging, and only fails when concurrency and duration multiply into port pressure. That makes it a code-review rule rather than a testing rule — no `http.Transport` literal below a constructor — backed by a `Reused` metric so the property is observable in production rather than assumed.