Why does a tracer installed with sys.settrace see nothing from a webhook receiver's worker threads?

## The state is per-thread, and nothing warns you The trace hook lives in thread state, not in the interpreter's global state. `sys.settrace(func)` arms the thread that executes it. Every other thread - already running or created a microsecond later - has its own empty slot. There is no exception, no warning, and no log line; the tracer just reports a suspiciously small part of the program. A webhook receiver makes this vivid. The process installs instrumentation during startup, on the main thread, then hands each inbound delivery to a worker from a pool. All the interesting code - signature verification, payload parsing, the handler that occasionally swallows an exception - runs on threads the tracer never touched. What you get back is a trace of the accept loop. ## The fixes, in order of preference **1. `threading.settrace(func)` before the workers exist.** The `threading` module keeps a hook that every thread it starts installs for itself, via `sys.settrace`, before running its target. Set it before the pool is created and every worker arrives instrumented. This covers a thread pool executor's workers too, because those are ordinary `threading.Thread` objects. The matching `threading.setprofile` does the same for the profiling hook. **2. `threading.settrace_all_threads(func)`, added in 3.12.** Same idea, but it also installs into Python threads that are already running - the right tool when the pool was created before you decided to instrument. `threading.setprofile_all_threads` is its profiling counterpart. **3. Install inside the worker.** Calling `sys.settrace(func)` as the first statement of the worker function always works and is explicit. It is the only option for threads that were not created through the `threading` module at all - a thread spawned by a C extension, or by the low-level thread module, never consults the `threading` hook. ```python def worker(job): sys.settrace(tracer) try: handle(job) finally: sys.settrace(None) ``` ## The second cause: the tracer uninstalled itself If the callback raises, the exception propagates at the traced location **and** the interpreter unsets the hook for that thread, exactly as if `sys.settrace(None)` had been called. Now combine that with a handler that catches broadly - the classic `except Exception: log_and_continue()` around a webhook payload. The tracer's error is swallowed as if it were a bad payload, tracing stops for that worker, and the worker keeps serving traffic. The symptom is identical to "tracing was never installed", which is why the diagnostic is to call `sys.gettrace()` *inside a worker* and log what comes back: `None` means either never armed or armed and then killed, and only a guarded callback tells the two apart. Wrap the callback body in `try`/`except`, report to a side channel, and always return the callback itself. ## Deciding whether to do this at all Even when it is wired correctly, per-line tracing on request-handling threads is a production hazard. Every executed line becomes a Python-level call, and instrumented code loses the specialized fast paths the adaptive interpreter would otherwise take. Against a 92nd-percentile latency budget of a few tens of milliseconds, a per-line hook on the handler path can consume the budget by itself. Sensible production use looks like: instrument a narrow slice of frames by filtering at the `'call'` event; arm one worker rather than the pool; arm it for a bounded window and disarm in a `finally`; or move measurement to a coarser hook entirely. ## The other thing to check first Before blaming thread scope, confirm nobody else took the slot. There is exactly one trace function per thread, and the last caller wins silently. A debugger attached to the process, or a coverage tool started by the test harness, will have displaced yours - or yours theirs, which is the more embarrassing outcome in a live process. `sys.gettrace()` at the moment of installation tells you what you are about to overwrite. ## What good looks like in the answer A senior answer names three things without prompting: the hook is per-thread state; `threading.settrace` (and, since 3.12, `settrace_all_threads`) is how you extend it to workers; and a callback that raises silently disarms itself. The candidate who only says "you have to set it in each thread" has the mechanism but not the failure modes. ## A checklist for the empty-output case When instrumentation reports less than you expected, work through it in this order rather than rewriting the tracer. First, *which thread armed it* - log the thread name at install time and again from inside a worker. Second, *is it still armed* - `sys.gettrace()` from the worker, since a callback that raised has already removed itself. Third, *did someone else take the slot* - a debugger or a coverage run installs into the same single per-thread hook and the last writer wins silently. Fourth, *is the global handler filtering too aggressively* - returning `None` at the `'call'` event for a frame you actually wanted produces exactly the same symptom as never being armed. Only after those four does it make sense to suspect the events themselves. ## Cleaning up Whatever the mechanism, disarm deliberately. `threading.settrace(None)` stops future threads picking the hook up, and `sys.settrace(None)` clears the current thread, but neither reaches back into workers that are already running and armed; those keep tracing until code on those threads clears their own hook. In a long-lived receiver that means a debugging session left half-disarmed can go on costing latency for hours, which is the argument for a bounded window and a `finally` around every arming.