A clinical-lab result loader dies with SIGSEGV inside a compiled extension - how do you capture a core file and read a Python stack from it?

## The situation A loader that ingests result files runs for about 45 seconds of cold start - warming an in-memory index before it touches the first record - and then dies. No traceback, nothing in the application log, status 139. Everything Python-level has already told you all it can. The only remaining witness is the process image at the moment of the fault, which is what a core file is. ## Step 1: make a core possible The most common reason there is no core file is that there was never going to be one. The soft limit `RLIMIT_CORE` defaults to 0 on many systems, so the kernel writes nothing. Raise it in whatever starts the process (`ulimit -c unlimited`) or, if you cannot change the launcher, at the top of the program itself with `resource.setrlimit(resource.RLIMIT_CORE, ...)` - raising the soft limit up to the hard limit needs no privileges. Then find out where cores go. On Linux the destination is the kernel-wide `/proc/sys/kernel/core_pattern`, and on most modern distributions it is not a filename at all but a pipe into a collector daemon, so looking in the working directory finds nothing while the core is sitting in the collector's store. On macOS cores land in `/cores`, and the limit still has to be raised. This is a host setting, not a per-process one, so it is worth confirming rather than assuming. Other reasons a core goes missing: the filesystem is full or read-only, the process changed its user id, or the container the process runs in has no visibility of where the host kernel wrote the file. ## Step 2: keep the exact binaries A core file is only meaningful together with the binaries that produced it. Open it with a *different* build of the interpreter, or against an extension that has since been rebuilt, and the debugger will map addresses to the wrong symbols and quietly show you a fictional stack. Capture the interpreter path, its exact version and build, and the extension binaries alongside the core - in an environment that redeploys frequently, the binaries can be gone within the hour. ## Step 3: debug symbols Symbols are the difference between a usable backtrace and a column of `??`. You need them in two places: for the interpreter itself, which usually means installing the matching debuginfo package or running a build that was not stripped, and for the extension, which means a binary compiled with `-g` and not stripped. Wheels are routinely stripped for size, so the practical move is often to rebuild the extension locally with debug information and reproduce against that. Also worth knowing: the interpreter's own compiler optimisations can inline frames, so a release build's C backtrace is approximate even with symbols present. A debug build of the interpreter additionally turns on internal assertions, which frequently catch the real error - a bad reference count, an object used after free - earlier and much closer to its cause than the eventual segfault does. ## Step 4: get Python frames out of C frames The C backtrace shows the interpreter's evaluation machinery, which is true but not useful on its own: you want to know which Python function was executing. CPython ships a gdb helper script installed next to the interpreter for exactly this. When gdb loads it, you gain commands that walk the interpreter's C stack, recognise its evaluation frames, and print the Python-level call stack from them - a Python backtrace, the local variables of a chosen Python frame, and movement up and down the Python stack rather than the C one. If gdb refuses to load the helper, it is nearly always the auto-load safe-path setting, and the helper can be sourced explicitly instead. If the Python backtrace comes out empty while the C one is fine, the usual cause is missing interpreter symbols: the helper needs to read the interpreter's internal structures by name. ## Step 5: read the two stacks together The pair is what tells the story. The Python stack says which call your code made; the C stack says what the extension did with it. A fault in an extension entered directly from the Python frame you see is a straightforward argument or lifetime problem. A fault deep in allocation or garbage-collection machinery, with no obvious relationship to the Python frame, is the signature of memory corrupted earlier by someone else - the crash site is not the bug site, and you should be looking at what changed rather than at the frame you landed in. ## A cause worth suspecting in this scenario When a crash starts after a deployment that reused a cached build directory, suspect a stale compiled extension. Extension filenames carry an ABI tag - `importlib.machinery.EXTENSION_SUFFIXES` shows the shape the current interpreter accepts - and a binary compiled against a different interpreter minor version or a different C library will happily load in some setups and then fault on the first call into it. Rebuild from a clean tree before you spend a day in a debugger. ## What core files do not cover A core is a corpse. For a process that is still alive but misbehaving, 3.14 added a remote-debugging entry point (`sys.remote_exec`, PEP 768) that injects a script into a running interpreter at a safe point - a different tool for a different failure. Reach for a core when the process is already gone.