skip to content

How do you decide between asyncio and a thread pool for 50,000 idle connections, and what does the choice cost the codebase?

level: principalimportance: should knowfreq 38%

answer

  1. Connection-bound means cost per waiter
  2. Thread stacks versus small task objects
  3. One blocking call stalls everything
  4. Async spreads through every caller
  5. One loop is still one core

basics

~20 s

At that concurrency the deciding number is cost per waiter: an operating-system thread reserves a large stack, an asyncio task is a small object. Asyncio wins on capacity, but only if nothing on the path blocks.

solid answer

~50 s

Fifty thousand mostly-idle connections is a connection-bound problem, so the metric is cost per waiter, not throughput. A thread per connection reserves a stack — commonly megabytes of address space each, tunable with `threading.stack_size` — plus a kernel scheduler entry, and the machine falls over long before the work does. An `asyncio` task waiting on a socket is a small Python object and a frame, so one loop holds tens of thousands comfortably. The catch is total: a single blocking call anywhere on the request path stalls the entire loop, so every driver, client and serializer must be non-blocking or be pushed through `asyncio.to_thread`, and each of those bridges is a signal that the fit is wrong. The codebase cost is the real decision — async spreads through every caller, changes how you test, profile and debug, and constrains library choice for years.

code

python · 15 lines
python
import asyncio
import time


async def idle_connection():
    await asyncio.sleep(1.0)


async def main():
    start = time.perf_counter()
    await asyncio.gather(*(idle_connection() for _ in range(50_000)))
    print(f"50,000 waiters finished in {time.perf_counter() - start:.2f}s")


asyncio.run(main())

go deeper

for a junior

Know the shape of the answer: threads are expensive per connection because each reserves a stack, while asyncio holds many waiting connections cheaply in one loop.

for a middle

Explain the mechanics behind that — stack and scheduler cost per thread versus a task object and a selector registration — and why any blocking call inside a coroutine stalls every other connection.

for a senior

Bring the operating judgment: measure the real concurrent peak and the CPU per connection, run a loop per core, detect loop lag in production, and use a bounded thread bridge for the blocking dependencies you cannot replace.

for a principal

Own the multi-year cost. Async colours every caller, constrains library choice, changes testing and debugging practice, and needs a deliberate boundary and migration plan — and sometimes the right call is a bigger machine and synchronous code.

## Name the class first Fifty thousand mostly-idle connections is not a throughput problem. Almost nothing is computed; almost everything is waiting. That makes it connection-bound, and connection-bound work is decided by *cost per waiter*, which is a memory and scheduling question, not a CPU one. Getting this framing out first is most of a strong answer, because it explains why the usual CPU-bound reasoning about the interpreter lock is irrelevant here. ## The arithmetic A thread per connection reserves a stack. The default is typically measured in megabytes of virtual address space per thread — `threading.stack_size` lets you shrink it, at the risk of overflow in deep call chains — plus a kernel task structure and a place in the scheduler's run queues. Fifty thousand of those is not a configuration you tune; it is a design you replace. An `asyncio` task parked on a socket read is a Python object, a coroutine frame, and a registration in the selector. Two or three orders of magnitude cheaper per waiter is the whole reason asyncio exists, and it is why every high-connection-count Python service is built on it. An intermediate option deserves naming: a thread pool that is *not* one thread per connection, sitting behind a mechanism that only hands a thread to a connection with actual work. That works well up to a few thousand concurrent connections and keeps the code entirely synchronous. If your real ceiling is two thousand connections rather than fifty thousand, taking the async cost may be a mistake. ## The cost the interview is really probing Asyncio's price is not performance, it is reach. The event loop runs one thing at a time; anything that blocks — a synchronous database driver, a CPU-heavy serialization step, a filesystem read, a DNS lookup through a blocking resolver — freezes every other connection for its duration. Consequences that a lead must own: - **Library constraint.** Every dependency on the request path must have a non-blocking implementation, or be wrapped with `asyncio.to_thread` / `loop.run_in_executor`, which reintroduces threads and their cost. A handful of bridges is normal; a codebase full of them means the model was the wrong choice. - **Function colouring.** `async def` propagates up the call chain. Shared utility code either forks into two variants or forces callers to become async. That is a repository-wide, multi-quarter migration in a mature service, not a module-level change. - **Observability and debugging.** Stack traces are per-task rather than per-thread, latency regressions appear as loop-lag rather than as a slow function, and a blocking call shows up as *everything* being slow. Teams need new instruments: loop-lag measurement, slow-callback detection, per-task diagnostics. - **Testing and skills.** Async tests, cancellation semantics and timeout behaviour are their own body of knowledge. Cancellation in particular changes how cleanup must be written. ## CPU still needs an answer One loop uses one core. Connection-bound services still need to serialize, compress and validate, so the deployment shape is normally a loop per core — several processes each running an event loop — with the small amount of genuinely heavy work offloaded to a worker pool. If per-connection CPU is not trivial, do that arithmetic explicitly: fifty thousand connections at even one millisecond of CPU each per second is fifty CPU-seconds per second, and no concurrency model rescues you from that. ## How I would decide Measure the real ceiling: peak simultaneous connections, bytes and CPU per connection per second, and how much of the request path currently sits inside blocking libraries. If the peak is a few thousand and the code is synchronous, keep threads and bound the pool. If the peak is tens of thousands, asyncio is the only model that fits on sane hardware, and the decision to make is the migration shape: async at the edge with a synchronous core behind a bounded thread bridge, or a full conversion. State the cost honestly, pick the boundary deliberately, and revisit it with the same measurements a year later.

  • Where would you draw the async boundary in an existing synchronous service?
    At the connection edge. Make the accept-and-parse layer async so the connection count is cheap, and keep the business core synchronous behind a bounded thread bridge until it is worth converting. That contains the colouring problem to one layer and gives you the capacity win immediately, at the cost of a hop per request that you should measure rather than assume is free.
  • What would make you keep threads despite a five-figure connection target?
    A dependency on the request path with no non-blocking implementation, a team with no async experience and a near deadline, or a measurement showing the real concurrent peak is far below the nominal connection count because connections are short-lived. Capacity you can buy with a slightly larger machine is cheaper than a migration you cannot finish.
  • How do you detect that a blocking call has crept onto the event loop in production?
    Instrument loop lag: schedule a callback at a fixed interval and record how late it actually runs. A rising tail there, correlated with all endpoints slowing together rather than one, is the signature. Development-time slow-callback logging catches most cases earlier, and code review should treat any synchronous client call inside a coroutine as a defect.

Threads are hotel rooms reserved for each guest whether or not they are in; asyncio is a coat-check ticket. Fine until one guest blocks the counter and nobody else can be served.

saying these in an interview costs you the question

  • Chooses asyncio for speed rather than for cost per waiter
  • Assumes a thread per connection scales to tens of thousands
  • Ignores that one blocking call freezes the whole event loop
  • Treats an async migration as a module-level change
  • Forgets that one event loop still uses one CPU core
  • Wraps everything in to_thread and calls the service async

context

resolving links…