A service calls three downstream dependencies (inventory, pricing, and recommendations) using a shared thread pool for all outbound HTTP calls. Recommendations starts responding slowly. Why can that alone stall inventory and pricing calls too, and what does the bulkhead pattern do about it?

## What the pattern isolates The **bulkhead pattern** isolates the resources a service uses to call one dependency from the resources it uses to call another, so that saturation on one path can't consume capacity the service needs for a different, healthy path. The name comes directly from shipbuilding: a ship's hull is divided into watertight compartments by bulkheads, so a hull breach floods only the compartment it occurred in rather than the entire ship. Applied to software, the "compartments" are typically resource pools assigned per downstream dependency, or per class of work with meaningfully different reliability/latency characteristics: - a dedicated thread pool, - a dedicated HTTP connection pool, - or a semaphore (a simple counter limiting how many concurrent calls are allowed). ## The noisy-neighbor default Without this isolation, a common default configuration - one shared thread pool or connection pool used for every outbound call a service makes - creates what's often called a **"noisy neighbor"** problem: if one dependency starts taking many seconds per call instead of milliseconds, every thread that picks up a call to it gets tied up waiting, and because the pool is shared, those occupied threads are no longer available to handle calls to completely unrelated, perfectly healthy dependencies. Within a short window, all the calling service's outbound capacity can be consumed by one slow dependency, and calls to a fast, healthy cache or database time out purely because there's no free thread left to make them - not because anything is actually wrong with that dependency. ## Thread-pool against semaphore isolation There are two common implementation styles, and the choice between them is a real trade-off. | Style | What it gives | What it costs | |---|---|---| | **Thread-pool isolation** | true isolation; straightforward to enforce hard timeouts and reclaim a stuck thread | additional idle threads consuming memory and stack space, plus context switching | | **Semaphore isolation** | caps concurrency through a permit counter the calling thread acquires and releases | weaker protection against a hung call | **Thread-pool isolation** gives each dependency its own dedicated, bounded pool of worker threads; a call to that dependency executes on one of its own threads, so if those threads all end up blocked waiting on a slow response, only that dependency's pool is exhausted - other pools are untouched. This gives true isolation and, because the threads are genuinely separate and interruptible, makes it straightforward to enforce hard timeouts and reclaim a stuck thread. The cost is overhead: each additional pool means additional idle threads consuming memory and stack space, plus the CPU cost of context switching between the calling thread and the pool's worker thread for every call. **Semaphore isolation** is lighter-weight: instead of routing the call to a separate thread, it simply limits how many concurrent calls to a given dependency are allowed to be in flight at once, using a permit counter that the calling thread itself acquires and releases. This avoids the extra threads and context-switch cost, but it doesn't protect against the calling thread itself getting stuck - if the call genuinely blocks rather than being cleanly cancellable, the calling thread is unavailable for other work regardless of the semaphore, so semaphore isolation caps concurrency but offers weaker protection against a hung call than thread-pool isolation does. ## Blast radius Why this matters beyond the immediate noisy-neighbor scenario is that bulkheads set the **blast radius** of a partial outage. Without isolation, a single degraded dependency - even one that's genuinely a small, non-critical feature like a recommendations widget - can, through pure resource contention, take down completely unrelated critical paths like checkout or authentication if they happen to share the same outbound call infrastructure. This is exactly the kind of failure that's disproportionately painful because the actual root cause looks, from the outside, unrelated to the symptom, which makes it slow to diagnose during an incident. ## Sizing the pools The trade-off on the provisioning side is that bulkheads require you to actually size each pool, and **undersizing** is its own failure mode: if a dependency's dedicated pool is set too small relative to real concurrent demand, calls to that dependency start queuing or getting rejected even during periods when the dependency itself is perfectly healthy and fast - the artificial cap becomes the bottleneck instead of the dependency's true capacity. This shows up in production as a throughput plateau for that dependency's calls that doesn't correlate at all with the dependency's own observed latency, which can be confusing to diagnose if the team doesn't remember the pool size is the limiting factor. ## Why it pairs with a breaker Bulkheads are almost always paired with **circuit breakers** rather than used alone: the bulkhead contains how much damage a struggling dependency can do to the rest of the service while calls are still being attempted, and the breaker stops attempting calls altogether once the dependency's failure rate confirms it's genuinely unhealthy, avoiding wasted effort on a known-bad path. Libraries like `resilience4j` implement both as separate, composable decorators around the same call, and this pairing was one of Netflix's original Hystrix design principles for exactly this reason.