Derive a sizing rule for a worker pool from first principles: how many workers does a purely CPU-bound workload need, and how does that number change when each task spends most of its time waiting on something external?

## The quantities For one task let **S** be service time - the time it actually needs a CPU - and **W** be wait time, the time it is blocked and consuming no CPU. Let **C** be the number of hardware threads and **U** the utilization you are targeting, a number below 1. ## The derivation A worker cycles through S of computing and W of waiting, so at any instant the probability that it wants a CPU is S/(S+W). With N independent workers the expected CPU demand is N x S/(S+W) cores. Setting demand equal to the CPU you intend to consume: ``` N x S/(S+W) = C x U N = C x U x (S+W)/S N = C x U x (1 + W/S) ``` The factor (1 + W/S) is the whole content of the rule: it is how many workers must exist for every one that is currently running. It is sometimes called the blocking coefficient, written 1/(1-b) where b is the blocked fraction W/(S+W) - the same number in different clothing. ## Reading it at the two extremes **W = 0 (compute only).** N = C x U. Sizing at the core count is the answer; a small addition is common because even compute-bound tasks stall on page faults, allocation, or an occasional log write, and a spare runnable worker keeps the core from idling during those stalls. Going far above buys nothing and costs switching and cache residency. **W >> S (mostly waiting).** A task that computes 5 ms and waits 95 ms has W/S = 19, so 8 cores at 80% utilization implies about 8 x 0.8 x 20 = 128 workers. This is why blocking I/O designs end up with pools in the hundreds while compute pools stay in single digits, and why one pool cannot serve both: a single N cannot satisfy two workloads whose W/S differ by two orders of magnitude. ## Relationship to Little's law The rule is the same statement as Little's law applied to the workers. In a stable system the average number of tasks in service equals arrival rate times average residence time, L = lambda x (S+W). Substituting the throughput a saturated CPU can sustain, lambda = C x U / S, gives N = C x U x (S+W)/S again. Recognizing that the formula is a queueing identity rather than folklore is the difference between reciting it and being able to adapt it. ## Where it breaks - **W is not always free.** Waits that involve encryption, deserialization, or busy polling contain hidden CPU time, so the true S is larger than assumed and the formula oversizes. - **Mixed W/S in one pool.** The formula's N is meaningless for a pool where some tasks wait 100 ms and others compute for 100 ms; that is an argument for separate pools, not for averaging. - **The downstream is the real constraint.** If the dependency saturates at 50 concurrent calls, sizing to 128 does not create throughput; it creates a queue on the other side, longer latency, and possibly a collapse there. The pool then acts as a concurrency limit and should be set to the dependency's healthy operating point. - **Memory and connections.** Each worker may hold a connection or a large buffer, so the sustainable N can be capped by those long before CPU or latency arguments bite. - **U is not 1.** Targeting full utilization makes queueing delay explode, so U is usually chosen in the 0.6-0.8 range for latency-sensitive work. ## How to obtain S and W Measure, do not guess: instrument the task to record total residence time and CPU time (inferring W as the difference), or compare process CPU time against wall time under a controlled load. Then use the formula for a starting value and adjust by observing utilization, queue depth and latency at each setting. The formula's real role is to give a defensible starting point and, more importantly, to tell you which direction to move when the numbers disagree.