How would you wire faulthandler into a service whose workers die and hang silently?

## The two silences A warehouse pick-list builder that occasionally disappears is showing you two different failures wearing the same face. - One is a **fatal signal**: an extension dereferences a bad pointer, the process dies with status 139, and the restart looks like a blip. - The other is a **hang**: the worker takes the job, never returns it, and eventually something upstream gives up - an intermittent timeout where the 92nd-percentile build sits far inside its budget and one build in a few thousand never finishes at all. Neither produces a traceback by default, and the fix is to make each of them produce one before you go looking for a cause. ## Enable it by environment `PYTHONFAULTHANDLER=1` belongs in the environment the service starts with, alongside its other runtime variables. The reason to prefer it over a `faulthandler.enable()` call is **coverage**: the environment variable is honoured while the interpreter starts, so a crash during import - which is where extension-module initialisation happens, and a common place for the ugly ones - is still reported. A call in your own startup code cannot cover anything that runs before it. `-X faulthandler` on the command line is equivalent when you control the command line rather than the environment. ## Then verify it Environment variables are lost: - by deployment refactors, - by a base image change, - or by someone copying a run command. One line at boot - assert `faulthandler.is_enabled()`, or log its value with the rest of your startup banner - converts that into a visible fact instead of a surprise you discover only when you needed the dump and did not get it. ## Make sure the output survives The dump goes to a **raw file descriptor**, by default the one behind `sys.stderr`. That is the right default when the supervisor captures stderr and ships it somewhere with enough retention to still hold the moments before a restart; it is the wrong default if stderr goes to a terminal nobody is attached to, or if the log pipeline only ingests the structured stream. If you redirect with `enable(file=...)`, remember that faulthandler keeps the underlying descriptor, not the Python object: the file must stay open for the whole life of the process, and a rotation scheme that closes and reopens the log will leave the handler writing into a file nobody can find. A dedicated append-mode file opened once at startup and never rotated by the application is the boring, correct choice. And because the handler cannot go through the logging system - it must be safe to run in a signal context, so no locks and no allocation - the dump: - will not be JSON, - will not carry your request identifier, - and will interleave with anything else writing to that stream. Plan for a plain-text crash channel rather than trying to make it look like an application log. ## Cover the hang with a watchdog The crash half is handled once the handler is installed; the hang half needs `faulthandler.dump_traceback_later()`. Arm it around the unit of work with a timeout comfortably above the healthy tail - if the slow end of a build sits well inside the budget, set the watchdog above that tail so a dump always means something pathological - and cancel it in a `finally` so it only fires on an overrun. - **`repeat=True`** is worth having when the goal is to distinguish stuck from slow: two identical dumps a few seconds apart say the frames are not moving. - **`exit=True`** turns the watchdog into a hard deadline that kills the worker after dumping; that is a real option when a wedged worker holds a job that another worker could retry, and a bad one when the process holds unflushed state, because the kill skips `finally` blocks and `atexit` handlers. ## Know the boundary faulthandler is **evidence, not protection**. - It does not keep a crashing process alive, - it does not interrupt a stuck call unless you asked it to kill the process, - and it shows Python frames only - a hang inside compiled code appears as the Python call site that entered it. It also cannot report a death that delivers no signal it can handle, such as a hard kill from the platform, so a silent disappearance with no dump and no exit signal points at the environment rather than at the interpreter. Finally, the dump is a stack, not a diagnosis: it tells you which call site was live, and the work of deciding why is still yours. ## The cost Installed handlers cost nothing per bytecode; an armed watchdog costs one sleeping thread. Compared with losing every crash and every hang, that is not a tradeoff worth deliberating - the defensible default for a long-running service is on.