How do you capture post-mortem state from a headless nightly ad-auction bidder that crashes once a week?

### Why the interactive answer does not apply A nightly bidder runs under a scheduler with no controlling terminal. `pdb.post_mortem()` there reads from a standard input that is not a tty: at best the job hangs holding a slot until something reaps it, at worst it dies on end-of-input with a confusing secondary error and you lose the original failure. Any interactive debugger in an unattended path is an outage waiting to happen. The technique that survives is the same idea applied without a human: extract from the traceback, at the moment of the crash, everything the human would have typed at the prompt. ### The scenario decides what you need Take the concrete failure: the bidder assumes the bid stream arrives in priority order, and with an 83% cache-hit rate the cached path preserves that order almost always. The 17% that miss the cache occasionally interleave, and one interleaving trips an index error deep in the settlement code. Once a week, in the middle of the night. Re-running does not reproduce it, because the input that produced the ordering is gone. The only thing that will ever answer 'why did this fail' is the state that existed when it failed - the ordering as the raising frame saw it. That is a capture problem, not a debugging-session problem. ### The capture `traceback.TracebackException.from_exception(exc, capture_locals=True)` walks the exception's frames and, for each one, records the `repr()` of every local variable at that moment. Its `format()` produces the familiar traceback with those values inlined under each frame: ``` File "bidder.py", line 5, in apply_bid return bids[index] * factor bids = [12, 30] factor = 0.83 index = 5 ``` That is a post-mortem inspection performed by the program on your behalf. It is deliberately eager: the `repr()` is taken at capture time, while the frames are still referenced, so the artefact is safe to keep after the exception is dropped. ### Where to hook it Wrapping the entry point in `try/except/raise` is the simplest and keeps the failure semantics intact. `sys.excepthook` catches anything that escapes the main thread even from code you did not wrap; `threading.excepthook` is its per-thread counterpart, and forgetting it is why 'the job logged nothing' happens when the crash was in a worker. Whichever you use, emit the artefact and then let the exception do what it was going to do, so exit status and alerting still work. ### The costs, which an interviewer is listening for **Disclosure.** Frame locals are where credentials, tokens, personal data and raw request bodies live. Capturing them writes all of it to wherever the log goes. Realistic mitigations: capture to a restricted artefact store rather than the general log stream, redact by name, or capture only for specific exception types. **Size.** `repr()` of a large container can be enormous, and one crash can produce megabytes. Cap the depth, and be aware that a custom `__repr__` runs your code during failure handling - it can itself raise, or be slow. **Retention.** As long as you hold the exception, its traceback holds the frames, which hold everything they reference. Capturing into a long-lived structure ('keep the last 50 failures') quietly retains 50 whole stacks. Format, emit, drop. ### What else the artefact needs A capture that is not findable is not useful. For a job that fails once a week, the capture must trigger an alert of its own, carry the run identifier and timestamp, and record enough about the inputs - which batch, which shard, which offsets - that you can reconstruct the conditions rather than just admire the values. It is worth writing the artefact before doing anything else in the handler, because cleanup code that runs first can itself fail and lose the original evidence. ### Then reproduce The captured locals normally tell you the input shape that broke; you write a test that constructs it, run that locally, and at that point the interactive tools are appropriate again - `pdb.post_mortem()` on the reproduction, with a terminal, is fine. The production side stays capture-only. If you want an interactive escape hatch for a developer-run invocation, gate it on both an explicit environment switch and a tty check, so no scheduled run can ever open a prompt.