Workers forked with os.fork() from an ETL exporter hang on their first log call. Why?

### The symptom The exporter runs on a schedule, forks a handful of worker processes to push batches to the warehouse, and at the 1,200-jobs-per-minute peak some children stop producing output entirely. They are not slow — they never emit their first log line. The parent stays healthy, the children hold their memory, consume no CPU, and sit there until something kills them. Rerun the same job off-peak and everything works, which is what sends teams looking for a data problem instead of a runtime one. A colleague's first theory is usually a shared mutable default — a list or dict created once at function definition and quietly accumulating across calls. It is a good instinct and it is wrong here: a shared default corrupts *results*, it does not stop a process before its first statement executes, and it would misbehave identically off-peak. ### The mechanism `os.fork()` copies the parent's address space but gives the child only one thread of control: the caller. Every other thread of the parent is absent from the child while all of its memory arrives intact and mid-flight. The exporter, like most services, runs background threads — a metrics reporter, a health check, a log shipper. Those threads log, and every emitted record takes the lock inside a `logging.Handler` so two threads cannot interleave a line. If the fork lands while such a thread is inside that critical section, the child receives the handler lock **already acquired**, and the thread that would have released it does not exist there. The child's first `logging` call blocks on `acquire()` and never returns. That explains every part of the symptom: - **Load-dependent:** the busier the parent's threads, the larger the fraction of wall-clock time they spend holding that lock, and the higher the chance a fork lands inside the window. At 1,200 jobs a minute the window is hit regularly; at ten a minute, almost never. - **Only some children:** each fork is an independent roll of the dice. - **Parent unaffected:** in the parent the holder thread is alive and releases normally. - **No traceback and no CPU:** it is a blocking lock acquisition, not an exception and not a loop. The same story can be told with the import machinery's per-module locks, a buffered writer's lock, a connection pool's internal lock, or the memory allocator's — logging is merely the most common because background threads log constantly. ### Confirming it rather than guessing Get a stack out of a stuck child; do not reason from the outside. - Install `faulthandler.register(signal.SIGUSR1)` in the worker entry point, then send that signal to a hung child and read the traceback it dumps. A post-fork deadlock parks in an `acquire` frame with no I/O beneath it. A slow warehouse write parks in a socket send or receive. - On **Python 3.14**, remote debugging (PEP 768) lets you attach to a running process without preparing it in advance, via `sys.remote_exec`. - Corroborate from the OS: the process is sleeping, its CPU time is flat, and it has no open connection to the warehouse — a slow write would have one. Also check whether the `DeprecationWarning` was there all along. Since **Python 3.12** `os.fork()` raises one when the process is multi-threaded, and services routinely discard it because `DeprecationWarning` is silent by default outside `__main__`. ### The fix Not a longer timeout, and not a retry — a deadlocked child never recovers, so a retry only converts a hang into a slower hang. 1. **Stop forking a threaded process.** On Python 3.14 the `multiprocessing` default is already `forkserver` on Linux and other Unix (macOS and Windows use `spawn`), which forks workers from a small single-threaded helper started early rather than from your threaded exporter. If the code calls `os.fork()` directly or asks for `fork` explicitly, that is the line to change. 2. **Or fork before the threads exist.** Build the worker pool during startup, ahead of the metrics and log-shipping threads. This shrinks the window rather than closing it, and it is fragile: whoever adds a thread earlier next quarter reopens the hole. 3. **Or exec immediately in the child**, replacing the address space so the inherited locks are discarded with it. Then make the class of bug visible so it cannot return quietly: run tests and staging with `-X dev` or `PYTHONDEVMODE=1`, and `logging.captureWarnings(True)` so the fork warning reaches your logs. ### What this question is really testing Whether you diagnose from evidence — a real stack out of a real stuck process — instead of from the most recently-read blog post, and whether you know that a load-dependent hang with no traceback in a freshly forked child has a specific, well-understood cause rather than being "flaky".