Your asynchronous service must call a library that only offers blocking operations, so you plan to run it on a separate worker pool and await the result. How do you decide that pool's size and queue bound, and what problem does offloading not solve?

## What offloading actually buys Moving a blocking call to a separate pool has exactly one effect: the thread that stalls is no longer a shared event-loop thread. The loop submits the work, yields, and continues serving everyone else; the result arrives asynchronously later. That is a genuine and important gain — it removes cross-cutting latency — and it is the *only* gain. The dependency is exactly as slow as before, and your capacity for that dependency is now explicitly capped by the pool rather than implicitly capped by everything. Making the cap explicit is a feature: an implicit limit fails as mysterious global slowness, an explicit one fails as a countable rejection. ## Sizing from queueing theory, not intuition **Little's law** — L = λW — relates the average number of items in any stable system (L), the arrival rate (λ), and the average time each item spends inside (W). It holds for any stable system regardless of distribution, which is what makes it a safe interview tool. Applied to an offload pool, L is the concurrency you must support: - 50 ms call, 400 calls/second → L = 400 × 0.05 = 20 concurrent → ~20 workers. - 2 s call, 10 calls/second → L = 20 as well. Note that *slow and rare* costs the same concurrency as *fast and frequent*. Three adjustments matter in practice: 1. **Variance.** Little's law gives the mean. If the service time distribution has a long tail, sizing to the mean leaves the pool saturated during tail events, and queueing delay grows non-linearly as utilisation approaches 1 (the standard queueing result: waiting time scales roughly with ρ/(1−ρ)). Target 60–75% utilisation, not 100%. 2. **The downstream's real limit.** If the dependency itself can only serve 10 concurrent operations, a 50-worker pool merely relocates the queue into the dependency, where you cannot see or bound it. Size to min(what Little's law says you need, what the dependency can absorb). 3. **Nature of the work.** Waiting work (I/O) can have many more workers than cores because they are parked. CPU-heavy work should have a pool near core count regardless of arrival rate — extra threads only add context switching and lengthen everyone's latency. If both kinds are offloaded, they need separate pools. ## Bounding the queue: a latency decision An unbounded queue is not a safety net, it is a delayed outage: it absorbs overload by converting it into growing latency and memory, so requests are executed long after the client gave up. Work that is already past its deadline still consumes a worker, which pushes fresh work further behind — congestion collapse. Bound the queue by the **latency budget**. Expected wait ≈ queue length × (service time / workers). Given a 200 ms budget, 50 ms service and 20 workers, each queued item adds 2.5 ms, so ~80 items is the point where admission is pointless. Beyond it, reject immediately. Even better than a fixed bound is a deadline check at dequeue time — discard work whose deadline has passed before spending a worker on it — plus adaptive admission control that shrinks the effective limit when observed latency rises. ## Bulkheads: one pool per dependency class A single shared "blocking work" pool couples unrelated dependencies: when one payment provider slows from 50 ms to 5 s, its calls occupy every worker and the unrelated user-profile lookups queue behind them. Separate pools give each dependency an isolation compartment, so failure is contained and its capacity is visible per dependency. The cost is more threads overall and more configuration; the benefit is that a single dependency's degradation stops being a whole-service incident. ## What offloading does not fix - **It is not a speed-up.** Same latency per call; you have only relocated the waiting. - **It does not raise throughput** beyond the dependency's own limit. - **It does not remove the need for timeouts.** Without one, a hung dependency permanently consumes workers until the pool is dead — the pool merely localises the damage. - **It does not replace a circuit breaker.** When a dependency is down, continuing to spend workers and queue slots on doomed calls delays recovery; failing fast frees both. - **It does not cure retries.** Retrying into a saturated pool multiplies load exactly when it is least affordable; budget retries and back off. - **It adds a hand-off cost** — task submission, context switch, result delivery — typically tens of microseconds, which matters only if the offloaded operation is itself very short. Sub-microsecond work should not be offloaded at all. ## Operating it Export four numbers per pool: active workers, queue depth, rejection rate, and time-in-queue percentiles. Queue depth persistently above zero means under-provisioning or a slowed dependency; rejections mean the bound is doing its job; rising time-in-queue with flat service time is the early warning. Alarm on time-in-queue, because it is the one that predicts customer-visible latency before rejections start. ## The short version "Size with Little's law against the dependency's measured service time and arrival rate, capped by what the dependency can actually absorb, targeting well under full utilisation. Bound the queue by the latency budget so overload rejects instead of rotting. One pool per dependency for isolation. And remember offloading only protects the loop — it does not make anything faster, so timeouts, deadline checks and a breaker still apply."