An API gateway holds one shared HTTP connection pool (say, 100 connections) used for calls to a dozen different backend microservices. During an incident, one backend service — call it the 'search' service — starts accepting connections but never responding (its handler threads are all deadlocked). Walk through how this takes down calls to the other eleven, unrelated backend services, and how bulkheading the connection pool per backend would change the outcome.

## Why this one is the dangerous shape of failure This scenario is the canonical bulkhead failure mode playing out at the **connection-pool layer** instead of the thread-pool layer, and walking through the mechanics precisely is worth doing because the 'accepts but never responds' detail is what makes it especially dangerous — it's slower and sneakier than an outright connection refusal. ## The shared pool, step by step Start with the shared pool as configured: 100 total HTTP connections, drawn from and returned to a single pool regardless of which of the twelve backends a given call targets. 1. **Under normal conditions**, connections are checked out briefly (for the duration of a request/response round trip) and returned promptly, so 100 connections comfortably serve bursty traffic across twelve backends because most connections are idle in the pool at any instant, available to whichever backend needs one next — this is precisely the utilization benefit of pooling that makes shared pools attractive in the first place. 2. **When search starts accepting TCP connections** (so the connection-establishment step succeeds and looks healthy from the gateway's point of view) but then never writes a response — its handler threads are deadlocked, so the request just sits — every gateway call to search now holds its checked-out connection for as long as the gateway is willing to wait, which is the read/response timeout configured on the HTTP client (or, worse, no timeout at all, in which case the connection is held forever). 3. **If search calls arrive at, say, 20 per second** and the gateway waits up to 30 seconds for a response before giving up, the steady-state number of connections tied up in stuck search calls approaches `20 x 30 = 600` — vastly more than the entire 100-connection pool — meaning within a few seconds of the deadlock beginning, all 100 connections are checked out to stuck search calls and none are available for anyone else, including the eleven backends that have nothing wrong with them. ## What the eleven healthy backends see At that point, a call to any other backend — say, the inventory service — attempts to check out a connection from the pool and finds none available; it either blocks waiting for one to free up (which won't happen until a search call times out and releases its connection, which given the arrival rate is immediately re-claimed by the next queued search call) or fails immediately with a pool-exhausted error, depending on the client's configuration. From the perspective of a caller of the gateway, or from monitoring dashboards, this looks exactly like a full gateway outage — elevated error rates and timeouts across every route — even though eleven of twelve backends never received a bad request and are operating completely normally. This is precisely why 'one service accepts connections but hangs' is more dangerous than 'one service refuses connections outright': - a **refused** connection fails fast and returns the connection to the pool almost immediately; - whereas an **accepted-but-hung** connection ties up the resource for the full timeout window, maximizing the damage per unit of bad traffic. ## What bulkheading the pool changes Bulkheading the connection pool per backend changes the outcome **by construction, not by detection**. If the 100 connections are instead partitioned — for example, roughly proportional to each backend's expected traffic share, giving search perhaps 8 connections reserved specifically for it and the remaining 92 split among the other eleven backends — then when search deadlocks: - only its own 8 connections can ever be consumed by stuck calls; - the moment those 8 are exhausted, further search calls are rejected immediately (fail-fast) rather than queuing for a connection that will never come; - and every other backend's dedicated slice of the pool remains completely untouched, because there is no shared resource left for search's failure to consume. Inventory, and the other ten backends, continue serving traffic at full capacity throughout the incident. The trade-off, as always, is that in normal times those 8 connections reserved for search sit unused whenever search's actual traffic is light, capacity that a shared pool could have lent to a momentarily busier backend — a real but much smaller cost than a full gateway outage. ## Where the infrastructure already does this This exact pattern — a service that hangs, rather than fails fast, causing shared connection-pool exhaustion that takes down unrelated traffic — is the textbook justification **Netflix** gave for building **Hystrix**, and it remains a standard root-cause pattern in postmortems for API gateways and service meshes that pool connections/threads globally rather than per-upstream. Modern service meshes like **Envoy** address it structurally by maintaining separate connection pools per upstream cluster by default, with per-cluster circuit-breaking limits (max connections, max pending requests, max requests) — which is bulkheading built into the mesh's data plane rather than something application code has to implement itself.