An email-digest sender's ThreadPoolExecutor backs up at a 1,200-send-per-minute peak; why does raising max_workers not fix it?

### The number that actually sets pool width For an I/O-bound pool, worker count is not a property of the machine at all — it is a property of the traffic. Little's law gives it directly: the concurrency you need is arrival rate multiplied by service time. A digest sender at a 1,200-send-per-minute peak is taking 20 arrivals per second; if a send completes in 200 ms, then 20 × 0.2 = 4 sends need to be in flight on average. Twenty workers do nothing for that load except sit idle. If service time triples to 600 ms, the requirement rises to 12, and a pool of 4 begins to fall behind — the same pool, the same code, a different answer, because the input changed. This is why `os.process_cpu_count()` is the wrong input here. Cores bound how much *computation* can happen at once; they say nothing about how many sockets can be open waiting for a remote reply. A pool sized from cores on an I/O workload is either wildly too small on a one-CPU allocation or wildly too large on a wide host. ### Why adding workers stops helping Every I/O pool sits in front of something with its own ceiling: a connection pool with a fixed size, an endpoint enforcing a rate limit, a name resolver, a disk. Once that ceiling is reached, extra workers do not increase completed work per second; they increase the number of places work is waiting. Latency per item gets worse because each in-flight request now shares the same bottleneck with more competitors, and error rates often get worse too, because a saturated remote starts refusing or timing out. The classic shape is a throughput curve that flattens and then bends downward as the pool grows — and past that point, worker count is a knob that only adds memory and scheduler work. Adding threads has a floor cost as well: each is an OS thread with a real stack reservation, and for pure-Python work they contend for the GIL. But the decisive fact is that once the downstream saturates, the pool's width has stopped being the limit. ### The unbounded queue is the real hazard The more damaging half of this failure is invisible from the worker count. The work queue inside a `ThreadPoolExecutor` is unbounded. `submit()` therefore *never* blocks: it appends and returns a `Future` immediately. A sender that loops over a recipient list submits all of them in a fraction of a second, and the queue absorbs every one — each holding a reference to its arguments, and each `Future` holding its eventual result alive. Three consequences follow. First, memory grows with the backlog rather than with the concurrency, and a burst that lasts a few minutes can be the thing that ends the process. Second, latency measured at the send call is meaningless: nearly all the elapsed time is queue wait, which no timing around the actual I/O will show. Third, and most subtly, the system has lost backpressure — the producer never learns that the consumer is behind, so it cannot slow the intake, shed low-priority digests, or fail fast. A backlog of queued work also ages badly: if a batch is rejected downstream because of an encoding mismatch in the payload and each item retries, the retries land behind an already deep queue, and the effective delay per item compounds. ### What to do instead Bound admission. The smallest correct change is a counting semaphore sized to the number of items you are willing to have outstanding: acquire it on the submitting thread before `submit()` and release it in the task's `finally`. Because `acquire()` blocks, the producer now inherits the consumer's rate, which is exactly what backpressure means. The alternative shape is your own bounded `queue.Queue` with a fixed set of consumer threads, where `put()` blocks when full; that also gives you a natural place to drop or prioritise when the queue is at its limit, which a semaphore does not. Then measure the right things. Worker count and CPU utilisation both look healthy while a pool drowns. The two numbers that tell the truth are how long an item waits between being submitted and starting, and how deep the backlog is; a rising queue-wait time with idle workers means the downstream is the limit, while a rising queue-wait time with all workers busy and the downstream unsaturated means the pool is genuinely too narrow. Only in that second case does raising the worker count help — and even then, only up to the concurrency the dependency has agreed to serve.