skip to content

Asyncio Synchronization and Queues

asyncio ships its own Lock, Event, Semaphore, Condition and Queue that suspend the task instead of the thread. The trap is reaching for threading.Lock in a coroutine and stalling the whole event loop.

part ofPythonoverview, primer and where to startread it →
on this pageshow

questions

4

Why use asyncio.Lock instead of threading.Lock inside a coroutine?

level: juniorimportance: must knowfreq 58%

answer

  1. Two lock families, one shared thread
  2. One suspends a task, one blocks a thread
  3. Control only changes hands at await
  4. Not thread-safe: bridge through the loop
  5. async with asyncio.Lock()

basics

~20 s

asyncio.Lock suspends only the awaiting task, so the event loop keeps running everything else. threading.Lock blocks the entire thread the loop runs on, freezing every task on that loop — including the one that would release the lock.

solid answer

~40 s

`threading.Lock.acquire()` blocks the OS thread. A coroutine runs on the event loop's thread, so blocking there stops the loop itself: no other task runs, no I/O callback fires, and if the holder needs an `await` to finish, the program deadlocks rather than merely slowing down. `asyncio.Lock.acquire()` is a coroutine — a contended acquire parks the task on a FIFO waiter queue and yields to the loop, which runs other tasks until `release()` wakes the first waiter. Use it as `async with lock:`; a plain `with` raises TypeError because there is no `__enter__`. A lock is still needed in single-threaded async: tasks interleave at every `await`, so a read-modify-write spanning one can be torn. The asyncio primitives are not thread-safe — signal them from another thread via the running loop's `call_soon_threadsafe`.

code

python · 19 lines
python
import asyncio

lock = asyncio.Lock()
balance = 0

async def deposit(n):
    global balance
    async with lock:
        current = balance
        await asyncio.sleep(0)   # another task may run here
        balance = current + n

async def main():
    async with asyncio.TaskGroup() as tg:
        for _ in range(100):
            tg.create_task(deposit(1))
    print(balance)   # 100; without the lock it is 1

asyncio.run(main())

go deeper

for a junior

Be ready to say that asyncio has its own Lock and that async with is how you take it. Know that blocking calls of any kind inside a coroutine stall the whole event loop, not just your own task.

for a middle

Explain the mechanics: a contended asyncio.Lock parks the task on a waiter future and yields to the loop, while threading.Lock parks the OS thread the loop lives on. Be able to show the lost-update bug a lock across an await prevents.

for a senior

Demonstrate production judgment: keep locked spans free of I/O, bridge threads with call_soon_threadsafe or run_coroutine_threadsafe, and push genuinely blocking work to asyncio.to_thread instead of guarding it with the wrong primitive.

for a principal

Own the boundary question: which state actually needs mutual exclusion versus which should be owned by a single task or funnelled through a queue. Locks in async code are usually a design smell about shared mutable state, not a scaling tool.

## Three families, overlapping names Python ships `threading.Lock`, `multiprocessing.Lock` and `asyncio.Lock`. They are unrelated objects with different blocking semantics, and the right one depends on ***what you need to suspend***: an OS thread, a process, or a single asyncio task. ## What `threading.Lock` does in a coroutine `acquire()` blocks the calling OS thread until the lock is free. A coroutine runs on the thread that runs the event loop, so **blocking there blocks the loop**: no other task is resumed, no I/O callback fires, no timer expires. If the task holding the lock needs to be resumed by that same loop in order to release it, nothing can ever release it — you have converted contention into a hard deadlock. The nastiest property of this bug is that an *uncontended* `threading.Lock` acquires instantly and behaves perfectly, so it sails through single-threaded tests and only bites under real concurrency. ## What `asyncio.Lock` does instead `acquire()` is a coroutine. 1. If the lock is free it flips an internal flag and returns immediately. 2. If it is held, the task creates a future, appends it to the lock's waiter deque and awaits it — awaiting hands control back to the event loop, which runs other tasks. 3. `release()` clears the flag and completes the first waiter's future, so waiters are served **first-in-first-out**; a newly arriving task can take a free lock immediately only when the waiter queue is empty, so it cannot barge past queued tasks. The thread is never blocked; exactly one task is suspended. ## Why a lock at all when one task runs at a time? This is the follow-up that separates people who have written asyncio from people who have read about it. Concurrency in asyncio is **cooperative**: control changes hands *only* at an `await`. So any code with no `await` in it is already atomic with respect to other tasks and needs no lock. But the moment a critical section spans an `await` — read a balance, await a write, store balance + n — another task can run in the gap, observe the stale value, and produce a **lost update**. `async with lock:` makes the whole section atomic with respect to other tasks on that loop, and the `async with` form guarantees the release even if the body raises or the task is cancelled. ## Not thread-safe — the second half of the rule The asyncio primitives assume every touch happens on the loop's own thread; they do no locking of their own. Setting an `asyncio.Event` directly from a worker thread is a data race and the waiting task may never be woken, because waking it means scheduling a callback on the loop. The supported bridges are: - the running loop's `call_soon_threadsafe` (schedule a plain callable, e.g. `event.set`); - `asyncio.run_coroutine_threadsafe` (submit a coroutine and get a `concurrent.futures.Future` back). In the other direction, when you must call blocking code from a coroutine, push it onto a worker thread with `asyncio.to_thread` — and inside *that* thread, `threading.Lock` is the correct primitive again. ## The rest of the family - **`asyncio.Event`** is a one-shot flag with `set`, `clear`, `wait` and `is_set`; note that `set()` completes every waiter's future immediately, so a `clear()` issued before those tasks are actually resumed does not un-wake them — they still return from `wait()` and must re-check the condition themselves. - **`asyncio.Semaphore`** is a permit counter for bounding concurrency. - **`asyncio.BoundedSemaphore`** additionally raises `ValueError` if you release more permits than it started with. - **`asyncio.Condition`** bundles a lock with `wait`/`wait_for`/`notify`/`notify_all` for the cases where a bare event is too coarse. All of them are `async with`-aware where that makes sense, and all of them are single-loop, single-thread objects. ## Finding the mistake Because a blocking call in a coroutine produces latency rather than an exception, asyncio gives you a detector: run with **debug mode** on — `asyncio.run(main(), debug=True)`, the loop's `set_debug` method, or the `PYTHONASYNCIODEBUG` environment variable — and the loop logs a warning naming any callback or task step that occupied the thread longer than its slow-callback threshold. That is how a `threading.Lock` acquire, a synchronous file read or a CPU-bound loop inside a coroutine announces itself. The fix is the same in every case: use the asyncio primitive if you are coordinating tasks, and push genuinely blocking work off the loop with `asyncio.to_thread`. ## Version notes Python 3.10 removed the `loop` parameter from every asyncio primitive. On 3.14 they bind to the running loop lazily, on first use, so constructing one at module import time is fine — but the object belongs to whichever loop first touches it, and reusing it under a second `asyncio.run()` raises `RuntimeError`. Prefer creating them inside the coroutine that owns them.

  • If one event loop runs one task at a time, when is an asyncio.Lock genuinely necessary?
    Only when a critical section contains an `await`. Between two awaits the code is already atomic with respect to other tasks on that loop, so a lock adds nothing. But an `await` is a yield point: another task can run there and observe half-applied state, which is how read-modify-write sequences lose updates. Lock the span that crosses the await, and keep that span as short as possible so you are not serialising I/O behind it.
  • How do you set an asyncio.Event from a worker thread without corrupting the loop's state?
    Capture the loop with `asyncio.get_running_loop()` while on the loop thread, then from the worker call `loop.call_soon_threadsafe(event.set)`. That queues the mutation onto the loop thread and wakes the selector, so the waiting task is actually resumed. To run a whole coroutine from a foreign thread, use `asyncio.run_coroutine_threadsafe`, which returns a `concurrent.futures.Future`. Calling `event.set()` directly from the thread is a race and the waiter may never wake.
  • Is asyncio.Lock fair, or can a fresh task jump the queue?
    It is FIFO-fair. Waiters are appended to a deque and `release()` completes the first pending waiter's future. A brand-new caller takes the lock without waiting only when the waiter queue is empty, so it cannot overtake tasks already queued. That predictability is worth knowing when a hot lock could otherwise starve a slow task — but it also means a long-held lock serialises everything behind it in arrival order.

A threading.Lock in a coroutine is a receptionist who takes a nap while waiting for a phone call: nobody else in the building gets served, including the one person who could wake her.

saying these in an interview costs you the question

  • "asyncio is single-threaded, so locks are never needed"
  • "threading.Lock is fine in a coroutine, it's all one thread anyway"
  • Writing `with lock:` instead of `async with lock:` on an asyncio.Lock
  • Believing asyncio.Lock is thread-safe and interchangeable with threading.Lock
  • Thinking a task can be preempted anywhere, not just at await points
  • Calling event.set() on an asyncio.Event directly from a worker thread

context

open as a page

How does asyncio.Semaphore bound a fan-out of hundreds of coroutines?

level: middleimportance: must knowfreq 55%

basics

~20 s

asyncio.Semaphore(n) holds n permits. async with sem: takes one and suspends the task when none are left, so at most n coroutines are inside the guarded block at once; waiters are released in order as permits come back.

open as a page

asyncio.Queue with no maxsize grows without bound in a streaming job — how do you apply backpressure?

level: seniorimportance: should knowfreq 40%

basics

~20 s

An asyncio.Queue defaults to maxsize=0, meaning unbounded, so a fast producer buffers the whole stream in memory. Give it a maxsize: once full, await queue.put(...) suspends the producer until a consumer drains an item, which is backpressure.

open as a page

What does asyncio.Queue.join() wait for, and what must consumers call?

level: middleimportance: nice to knowfreq 32%

basics

~10 s

join() waits until the queue's unfinished-items counter reaches zero. Every put increments it, and only a consumer calling task_done() decrements it. Consumers that get items but never call task_done() leave join() waiting forever.

open as a page