When sizing a per-dependency bulkhead (thread-pool or semaphore) for a downstream API, what inputs do you use to pick the pool size or permit count, and what happens if you get it wrong in either direction?

## Sizing is Little's Law applied to a pool Sizing a bulkhead is fundamentally a capacity-planning exercise grounded in **Little's Law**, which states that the average number of items in a system (here, concurrent in-flight calls) equals the average arrival rate multiplied by the average time each item spends in the system. Applied to a bulkhead: the concurrency you need to provision for a dependency is approximately its expected request rate (calls per second under normal peak load) multiplied by its expected call duration (average or a chosen percentile of latency, in seconds). For example, a dependency that receives 100 requests/second and typically responds in 50ms needs roughly `100 times 0.05`, or 5, concurrent slots just to keep up under steady state — but real systems don't run at a perfectly steady average, so you provision headroom above that baseline to absorb normal variance (bursts, brief latency spikes) without spurious rejections. ## The timeout is the second, coupled input The timeout enforced on the call itself is a second, tightly coupled input. Because concurrency need scales with how long each call holds its slot, a tighter timeout directly reduces the pool size required for the same throughput: - if you cut the enforced timeout from 5 seconds to 1 second, a slot is held for at most a fifth as long in the worst case, so **fewer slots are needed** to sustain the same steady-state throughput; - and just as importantly, the **pool drains faster** once a dependency starts failing, because stuck calls get evicted sooner rather than accumulating. This is why bulkhead sizing and timeout configuration are usually decided together, not independently — a bulkhead sized assuming a 200ms typical latency but paired with a 30-second timeout is dramatically under-protected against a dependency that goes fully unresponsive, because every stuck call now occupies its slot 150x longer than the sizing assumed. ## Getting it wrong in either direction Getting the size wrong has costs on both sides, and they are qualitatively different failures. - **Undersizing** shows up as a self-inflicted availability problem: during genuinely normal peak traffic (not a dependency incident at all), calls get rejected or queue past acceptable latency purely because the pool's cap is below what legitimate demand requires. This is often mistaken for a downstream problem during on-call triage — the symptoms (elevated error rate, timeouts) look identical to a real dependency degradation — until someone checks the downstream's own health metrics and finds it fine, then checks the bulkhead's own saturation/rejection metrics and finds the pool pegged at its cap. The fix is either raising the cap (if there's slack resource capacity to give it) or investigating why demand grew past the sizing assumption. - **Oversizing** is the quieter failure: a bulkhead sized far larger than steady-state need (e.g., 'just set it to 500 to be safe') reduces the isolation benefit, because a large enough pool for one dependency starts to behave like a shared pool in miniature — enough stuck calls can still accumulate to matter, and meanwhile those over-provisioned threads/slots are memory and scheduler overhead that isn't available to size other dependencies' pools generously, or (in a thread-pool bulkhead) is literal idle thread footprint sitting around unused most of the time. ## A sizing workflow that starts from observed data A production-realistic sizing workflow starts from observed data rather than a guess: 1. **Pull** the dependency's actual p50/p95/p99 latency and peak QPS from existing metrics/tracing. 2. **Compute** the Little's-Law baseline. 3. **Add headroom** — a common starting point is 1.5 to 2x the computed steady-state concurrency, tuned down over time based on real rejection-rate telemetry. 4. **Pair it with a timeout** set close to the dependency's own p99 rather than a generous, arbitrary default. After deployment, the pool's utilization and rejection-rate metrics become the feedback loop: - a partition that regularly sits near 100% utilization during ordinary peak traffic needs a bigger cap or investigation into demand growth; - while a partition that never approaches its cap even during the dependency's worst observed latency spikes is a candidate to shrink, freeing resources for a differently-behaved dependency elsewhere in the same service. ## What the libraries recommend A concrete illustration: **Netflix's Hystrix** documentation explicitly recommended computing thread-pool size from measured throughput and 99th-percentile execution time plus a margin, rather than picking a round number, precisely because an under-provisioned command thread pool produces exactly the false-rejection failure mode described above — under normal traffic, not just during a genuine dependency outage. **Resilience4j's** `Bulkhead` and `ThreadPoolBulkhead` configuration follows the same principle: `maxConcurrentCalls` (or core/max pool size plus queue capacity) should be derived from the dependency's real observed concurrency needs and reviewed periodically as traffic patterns shift, not set once and forgotten — a bulkhead sized for last year's traffic is a bulkhead quietly waiting to cause a false outage the next time legitimate demand grows.