How does logging.handlers.QueueHandler with QueueListener keep slow I/O off the calling thread?

### The problem it solves A `logging.Handler` does its work synchronously, on the thread that called `logger.info(...)`, while holding the handler's lock. If that work is a write to a file on a slow disk, a rotation, a DNS lookup or a socket connect, the thread doing real work waits for it — and every other thread that wants the same handler queues behind the lock. In an asyncio program the consequence is sharper still: a blocking handler stalls the event loop, and every task on it, for the duration of the write. `logging.handlers.QueueHandler` and `logging.handlers.QueueListener` split the handler in two. The logger-side half does the cheapest possible thing and returns; a background thread does the expensive half. ### The wiring ```python import logging import logging.handlers import queue log_queue = queue.Queue(maxsize=10_000) sink = logging.handlers.RotatingFileHandler( "etl-export.log", maxBytes=10_000_000, backupCount=5, encoding="utf-8" ) listener = logging.handlers.QueueListener(log_queue, sink, respect_handler_level=True) listener.start() logger = logging.getLogger("etl.export") logger.addHandler(logging.handlers.QueueHandler(log_queue)) ``` The real handler is attached to the **listener**, never to the logger — attaching it to both writes every line twice. `respect_handler_level=True` makes the listener honour each sink's own level; without it the listener hands every dequeued record to every handler regardless of level. ### What prepare() does, and why it matters `QueueHandler.emit()` is three lines: `self.enqueue(self.prepare(record))`, wrapped in a `try` whose `except` calls `handleError(record)`. `prepare()` formats the record with the handler's formatter, copies the `logging.LogRecord`, writes the merged text into `msg` and `message`, and sets `args`, `exc_info`, `exc_text` and `stack_info` to `None`. Two things follow. **Good:** the record becomes picklable, so the same design works across processes with a `multiprocessing.Queue`, and a traceback is already rendered into text before the record travels. **Less obvious:** the `%`-style argument merging happens on the calling thread, not the listener thread. The queue moves the *I/O* off the hot path, not the formatting. It also means the listener's handlers see a record whose arguments are already gone, so a formatter downstream can only re-arrange the fields around an already-rendered message. ### The queue is a policy decision `enqueue()` uses `queue.Queue.put_nowait`. On a bounded queue that raises `queue.Full` the moment the listener falls behind; `emit()` catches it and calls `handleError()`, which drops the record and — while `logging.raiseExceptions` is true — prints a traceback to `sys.stderr`. So: * **Bounded queue:** logging degrades by losing records under pressure, and memory stays flat. Usually the right choice for a service, provided you know it is happening; a small `enqueue()` override that counts drops turns silent loss into a metric. * **Unbounded queue:** no record is ever dropped, and a listener that cannot keep up turns your log backlog into unbounded memory growth. This is not backpressure; it is a slow leak with a deadline. * **Blocking put:** overriding `enqueue()` to use a blocking `put` with a timeout gives real backpressure, at the cost of making a logging call able to stall application work — which is exactly what the queue was introduced to prevent. Choose it deliberately, not by accident. ### Shutdown is not optional `QueueListener.start()` creates a daemon thread. At interpreter exit a daemon thread is simply abandoned, so any records still sitting in the queue are lost, and `logging.shutdown()` does not help: it flushes handlers, and the queue is not a handler's buffer. Call `QueueListener.stop()` on your shutdown path — it enqueues a sentinel and joins the thread, so everything already queued is written first. Registering it with `atexit.register` or using the listener as a context manager, which its `__enter__`/`__exit__` support allows, both work. Order matters: stop the listener *after* the components that log, or you lose exactly the shutdown messages you most want. ### Where it fits with the other handlers This is also the standard answer to multiple processes needing one rotating file: workers attach a `QueueHandler` over a `multiprocessing.Queue`, one listener in the parent owns the single rotating handler, and only that one process ever renames a file. The queue buys three things at once — the disk write leaves the worker, the rotation gets a single owner, and the ordering of the merged file is decided in one place. In an asyncio service the same wiring is what keeps a file write out of the event loop: the coroutine calling `logger.info(...)` performs a `put_nowait` and continues, and the write happens on the listener's thread. Nothing about the pair is asyncio-specific — there is no async handler protocol here — which is exactly why it is the standard answer for that case too.