A geocoding batch using asyncio.to_thread() starts timing out unrelated connections - why?

## The shape of the incident A batch job geocodes addresses through a synchronous client, correctly offloaded with `await asyncio.to_thread(client.geocode, addr)`, and fans out across a 340-case regression pack. The batch itself completes. What breaks is everything *else* in the process: unrelated outbound calls start failing at connect time, health checks flap, and the latency histogram fills with values that look impossible against wall-clock timestamps - which is why the first hypothesis is usually a clock-skew artefact in the timing logs. It is not. It is one saturated thread pool. ## One pool, shared by everything `asyncio.to_thread()` takes no executor argument. It submits to the running loop's **default executor**, created lazily as a `concurrent.futures.ThreadPoolExecutor` the first time anything needs it. Every `to_thread` call anywhere in the process shares it, and its worker cap is the pool's own default: `min(32, CPUs + 4)`, with the CPU count coming from `os.process_cpu_count()` since 3.13 - so a container pinned to two CPUs gets six workers, not thirty-two. Submissions beyond the cap sit in an unbounded queue, which is why saturation shows up as latency rather than as an error. The part that turns a slow batch into an outage is that the loop uses the same pool for its own work. Address resolution (`getaddrinfo`) is a blocking C call, so the default event loop performs it in the default executor. With every worker parked in the geocoding client, a new connection cannot resolve its host; it waits behind the batch and eventually reports a connect timeout. The failure lands on components that have nothing to do with the batch, which is exactly what makes it hard to attribute. ## Isolation is the fix Give the noisy dependency its own bounded pool and route it through the loop's `run_in_executor()`: ```python loop = asyncio.get_running_loop() pool = ThreadPoolExecutor(max_workers=8, thread_name_prefix="geocode") await loop.run_in_executor(pool, functools.partial(geocode, addr, retries=3)) ``` Three properties come with that. The default pool stays free for name resolution and for short offloads elsewhere. The thread name prefix makes stack dumps and thread listings legible during the next incident. And the pool size becomes an explicit capacity decision per dependency instead of an accident of CPU count. `run_in_executor()` accepts positional arguments only, so keyword arguments go through `functools.partial`. It is also the door to a `ProcessPoolExecutor` when the offloaded work is CPU-bound, at the cost of picklable arguments and no lambdas or closures. ## Bounding, and who cleans up A pool cap limits concurrency but not the queue: submitting all 340 cases at once still allocates 340 futures and their argument graphs immediately. Bound the fan-out on the async side too - an `asyncio.Semaphore` around each `run_in_executor` await, or batching the input - so memory tracks in-flight work rather than total work. Cleanup is yours. `asyncio.run()` shuts down the *default* executor before returning, waiting for its threads; it knows nothing about pools you created. Use the executor as a context manager, or call `shutdown(wait=True, cancel_futures=True)` explicitly on the way out. Note that a synchronous `shutdown(wait=True)` called from the loop thread blocks the loop while it drains, so at shutdown time do it after the loop's work is done, or offload the shutdown itself. ## Sizing the dedicated pool For blocking I/O, worker count is a concurrency budget, not a CPU budget: the right number is roughly the target in-flight requests, capped by what the downstream dependency and its connection pool tolerate. Oversizing turns your outage into the dependency's. Undersizing shows up as queueing delay in *your* pool, which is at least attributable, because the wait is now behind a named prefix instead of behind name resolution. ## The diagnostic move to remember When unrelated connections start timing out while a blocking batch runs, dump thread names and stacks. If you see the whole default pool parked in one library's frames, you have found it: the coincidence is not clock skew, it is a shared, bounded, invisible resource with two very different classes of user on it.