How do you confirm a live Python process is deadlocked on two threading locks?

### The symptom A lock-order deadlock in Python is a silent hang. There is no exception, no log line, and no `RuntimeError`: the threads involved are blocked inside a C-level lock acquisition, consuming no CPU. Process CPU drops to whatever the unaffected threads are doing, memory stays flat, and any queue in front of the service grows without bound. The signature that distinguishes it from an infinite loop is exactly that CPU floor — a spinning bug burns a core, a deadlock burns nothing. ### Getting stacks out of a live process The problem is that the usual debugging reflex — read the traceback — has nothing to read. You need the stacks of *every* thread, on demand, from a process that will not respond. **Armed in advance, the cheapest option.** `faulthandler` is in the standard library and costs nothing while idle: ```python import faulthandler faulthandler.dump_traceback_later(30, repeat=True, exit=False) ``` If the process fails to cancel that timer within 30 seconds, it prints every thread's stack to standard error and keeps doing so. Placed in a watchdog loop that cancels and re-arms after each successful cycle, it turns a hang into a stack dump you can read in the logs. **On demand, via a signal.** `faulthandler.register(signal.SIGUSR1, all_threads=True)` makes the process dump all thread stacks whenever you send it that signal. That is the one line worth adding to every long-lived Python service before it ever hangs. **From inside, without faulthandler.** `sys._current_frames()` returns a mapping of thread id to the frame each thread is currently executing; pair it with `traceback.format_stack` and `threading.enumerate()` to print named stacks from a watchdog thread that is itself not blocked. **Nothing armed at all.** CPython 3.14 ships PEP 768 remote debugging: `sys.remote_exec` injects a script into an already-running interpreter, so you can attach to a process that was started with no instrumentation and have it print its own thread stacks. An out-of-process stack sampler is the other classic route when you cannot modify the target. ### Reading the dump You are looking for two or more threads whose innermost frame is a lock acquisition — a `with some_lock:` line, or an explicit `.acquire()` — at *different* call sites. Then trace outward: each blocked thread's stack shows which locks it already holds, because those are the `with` blocks it is currently inside. When thread A holds lock 1 and is waiting for lock 2, while thread B holds lock 2 and is waiting for lock 1, you have a lock-order inversion and the diagnosis is complete. If instead one thread is blocked on `Queue.get` and everything else is idle, the bug is a lost producer, not a deadlock. ### Why it happened The most common causes in Python code are: - **Two locks taken in opposite orders** by two code paths that were written months apart, often because one of them acquires the second lock inside a helper the author did not read. - **Calling unknown code while holding a lock** — a callback, a `__del__`, a logging handler, a property that lazily initialises something. That foreign code takes its own lock and the ordering is now out of your hands. - **Re-entering the same non-reentrant lock** on one thread, which is a self-deadlock and shows as a single thread parked in `acquire` with the same lock further up its own stack. Note what does *not* fix an ordering inversion: switching both locks to a reentrant lock. Reentrancy only lets one thread re-acquire a lock it already owns; two threads waiting on each other are unaffected. Candidates offer this fix often and it is a reliable signal. ### Preventing it 1. **One lock.** The cheapest fix is to notice that two locks were protecting one invariant and collapse them. Fewer locks, fewer orderings. 2. **A fixed global order.** Write down the order in which locks may be acquired and follow it everywhere. Where the objects are dynamic, sort them by a stable key — `for lock in sorted(locks, key=id)` — so any two code paths acquire the same pair in the same sequence. 3. **Never hold a lock across foreign code.** Copy the data you need, release, then call out. This also keeps critical sections short, which is worth doing anyway. 4. **Acquire with a timeout.** `lock.acquire(timeout=5)` returning `False` lets you log a clear "suspected lock-order inversion" with the current stack rather than hanging. A loud failure at 3 a.m. beats a silent one. 5. **Prefer designs with no lock pair at all.** Handing work to one owner thread through a `queue.Queue` removes the second lock entirely; confinement removes both. ### Why this is a differentiator rather than a gate Most Python services never deadlock, because most of them use one lock or none. Knowing the `faulthandler` incantations is not something a strong candidate should be marked down for missing — but a candidate who has actually run into a hung process, and can describe how they got stacks out of it, is showing production experience that is hard to fake.