A service runs its work on a fixed set of worker threads fed by a task queue. What happens when every worker is already busy and new tasks keep arriving, and how does that show up to the callers submitting work?
answer
- capacity = workers / service time
- queue absorbs bursts, not sustained overload
- latency ~ rho/(1-rho) hockey stick
- Little's law L = lambda x W
- head-of-line blocking starves cheap tasks
basics
~20 sNew tasks wait in the queue instead of running. Throughput stays flat at what the workers can serve, so latency climbs as the backlog grows. An unbounded queue eats memory; a bounded one eventually refuses work or blocks the submitter.
solid answer
~40 sA pool has a fixed service capacity: worker count divided by average task duration. Below that arrival rate, tasks start almost immediately. Once every worker is busy, extra tasks only queue - throughput does not rise, so each task's latency becomes queue wait plus run time, and the wait keeps growing while arrivals exceed capacity. Callers see latency climbing steadily rather than a clean failure, then timeouts, then retries that add more arrivals and make it worse. With an unbounded queue the backlog and memory grow until the process dies; with a bounded queue submissions start being refused or blocked once it fills. The second effect is starvation: cheap tasks queued behind expensive ones wait for work they have nothing to do with, so unrelated workloads sharing one pool degrade together.
code
text · 6 linesarrivals: 30 tasks/sec capacity: 2 / 0.1s = 20 tasks/sec
t=1s queue=10 wait~0.5s
t=2s queue=20 wait~1.0s
t=5s queue=50 wait~2.5s <- grows by 10/sec forever
throughput stays pinned at 20/sec the whole timego deeper
Say plainly that extra tasks queue, latency grows, and throughput does not - and that an unbounded queue is a memory risk.
Add the arithmetic: capacity is workers divided by service time, and waiting time explodes as utilization approaches 1.
Bring in diagnosis (throughput plateau versus falling throughput, Little's law), retry amplification, and head-of-line starvation of unrelated work.
Frame it as an admission-control problem: decide what load you promise to serve, enforce it with backpressure and deadlines, and isolate workload classes so saturation in one does not consume the whole service.
## What a worker pool is A worker pool is a fixed set of long-lived threads plus a queue of pending tasks. Submitting work does not run it - it enqueues a description of the work. A free worker takes the next task, runs it to completion, then loops. The pool decouples the rate at which work is *submitted* from the rate at which it can be *served*, and caps how many things run at once. ## The capacity arithmetic Let `c` be the worker count and `S` the mean time a worker spends on one task (including any time blocked waiting on a network call, because a blocked worker is still occupied). The pool can complete at most `c / S` tasks per second. With 8 workers and 50 ms tasks that is 160 tasks/sec, no matter how big the queue is. Utilization is `rho = lambda * S / c` for arrival rate `lambda`. Standard queueing results say mean waiting time grows roughly in proportion to `rho / (1 - rho)`: at 50% utilization the wait is on the order of one service time, at 90% it is several, and as `rho` approaches 1 it grows without bound. This is why a pool feels fine right up until it suddenly does not - the latency curve is a hockey stick, not a ramp. If `lambda > c / S`, the system is not slow, it is *unstable*: the backlog grows by `lambda - c/S` items every second forever. No amount of queueing fixes that; only reducing arrivals, reducing `S`, or adding workers does. ## What saturation looks like from outside - Latency rises while throughput plateaus - the classic signature of a queue, as opposed to a slow dependency (where throughput would fall with concurrency held constant). - Little's law (`L = lambda * W`) lets you sanity-check: measured in-flight count equals arrival rate times observed latency. A large `L` with all workers busy means the excess is sitting in the queue. - Callers time out, retry, and increase `lambda` - a retry storm that turns a temporary overload into a self-sustaining one. - Memory grows if the queue is unbounded, because the backlog is retained task objects and their captured arguments. ## Starvation Saturation is about the aggregate; starvation is about a particular task never getting a turn. On a FIFO queue, a burst of long tasks head-of-line-blocks every short task behind it. On a shared pool, a slow dependency's tasks occupy all workers and unrelated fast work - health checks, cache refreshes, an admin endpoint - is starved even though it needs milliseconds of CPU. Priority queues, separate pools per workload class, or admission limits per class are the usual remedies; how a full queue rejects work is a separate concern from whether the pool is saturated at all. ## Responses Short term: shed load or apply backpressure so `lambda` falls, and put deadlines on the work so tasks that nobody is waiting for anymore stop consuming workers. Medium term: reduce `S` (cache, batch, fix the slow dependency) or raise `c`. Structurally: stop mixing workload classes on one pool. Note that a queue is a shock absorber for bursts, not extra capacity - if the average arrival rate exceeds service capacity, the queue only converts a fast failure into a slow one.
- How would you tell a saturated pool apart from a pool whose tasks simply became slower?Compare throughput with in-flight count. If throughput has plateaued at the old ceiling while queue depth and latency grow, arrivals are outrunning capacity. If throughput has fallen while worker count is unchanged, per-task service time went up - usually a slow dependency - and the queue growth is a consequence, not the cause. Little's law ties the two together: a rise in observed latency with constant arrival rate must show up as more work in the system.
- Does adding a bigger queue help a pool that is persistently overloaded?No. A queue only buffers bursts, smoothing arrival variance against a service rate that is on average high enough. If the mean arrival rate exceeds the mean service rate, the backlog grows without limit, so a bigger queue only means callers wait longer before failing and the process holds more memory. The fixes are fewer arrivals, faster tasks, or more workers.
A supermarket with a fixed number of tills. Once every till is busy, more shoppers only make the line longer - the checkout rate is unchanged, and one shopper with a full trolley delays everyone behind them.
saying these in an interview costs you the question
- Claiming a larger queue increases the pool's throughput
- Assuming a full queue means the CPU is busy, when workers may all be blocked on I/O
- Treating rising latency as a dependency problem without checking queue wait time
- Ignoring that retries on timeout increase the arrival rate and deepen the overload
- Believing one shared pool is fine because 'the tasks are all small'