A Python video-metadata extractor's piped log stops short when the process is killed. Why, and how do you fix it?

## The symptom, precisely An extraction worker walks a queue of media files, prints one line per file it has indexed, and its stdout is captured by whatever supervises it. The process is killed, and the log ends one file short of what the downstream store actually contains. The obvious reading, that the last write never happened, is wrong, and chasing it leads you into the wrong code: people start auditing the loop bounds for an off-by-one when the loop was correct and the *log* is the thing that is short. ## Why the tail is missing Captured stdout is not a terminal, so at interpreter startup CPython built `sys.stdout` block buffered: writes accumulate in an `io.BufferedWriter` of `io.DEFAULT_BUFFER_SIZE` bytes, 128 KiB on Python 3.14 and only reach the descriptor when the buffer fills, when something flushes, or when the interpreter flushes the standard streams during normal shutdown. That last clause is the whole story. Normal shutdown happens on a clean return, on `sys.exit()`, and even on an uncaught exception. It does **not** happen when the process is killed by a signal whose default disposition terminates it, and it does not happen through `os._exit()`. `signal.SIGKILL` cannot be handled at all. `signal.SIGTERM` can, but if you never installed a handler its default action ends the process on the spot, so the polite stop signal that an orchestrator sends before the hard one is enough to lose your buffer. Whatever was in that block window is discarded with the process's memory. The amount you lose is not fixed. It is however many lines happened to be in the buffer, which depends on line length: against Python 3.14's 128 KiB block, 90-byte lines mean well over a thousand lost, while 900-byte structured records mean about a hundred and fifty. That variability is why the same bug looks like a different bug on different days. ## The diagnostic that separates the two hypotheses One comparison settles it. `sys.stderr` is line buffered even when it is not a terminal, so anything the program wrote to stderr around the same instant is present. If stderr shows the worker still going after the last stdout line, the work happened and the log lost it. Two more confirmations are cheap: `sys.stdout.isatty()` and `sys.stdout.line_buffering` printed at startup tell you which branch the stream took, and reproducing the run interactively makes the symptom vanish entirely, which is itself the proof, because the only thing that changed is that stdout is now a terminal. ## Fixing it at the producer Buffering belongs to the writing process, so every real fix is on that side. 1. **Line buffering, set in code.** `sys.stdout.reconfigure(line_buffering=True)` early in startup. Each completed line leaves immediately, multi-write `print()` calls still coalesce, and lines never tear in half. This is usually the right default for a log-emitting worker. 2. **Unbuffered, set from outside.** `-u` on the interpreter, or `PYTHONUNBUFFERED=1` in the environment. Use it when you do not control the entry point, which in a container is common. It costs a syscall per write. 3. **Per-call flush.** `print(line, flush=True)` on the statements that matter. Right for a few latency-sensitive markers, wrong as a blanket habit. A fourth measure is complementary rather than an alternative: install a `signal.SIGTERM` handler that shuts down cleanly. That turns a graceful stop into a normal exit, which flushes, and gives you a place to finish the file in flight. It does nothing for `signal.SIGKILL`, so it is not a substitute for buffering discipline. ```python import signal import sys sys.stdout.reconfigure(line_buffering=True) signal.signal(signal.SIGTERM, lambda *_: sys.exit(0)) ``` ## The judgement part The reason interviewers like this scenario is that it separates two habits. One candidate turns on unbuffered output everywhere and moves on. The other asks what the log is for. If it is the audit trail that tells you which media files were processed, buffering is a correctness problem in the observability path and line buffering is mandatory; if it is chatty progress output, the default is fine and the syscalls are not worth paying. In a pipeline made of many services this is a per-service call, not a fleet-wide reflex, though a fleet-wide default of line-buffered stdout is a defensible standard precisely because the failure mode is silent: nothing errors, the log is simply, occasionally, short. And the corollary worth saying out loud: a log with buffering in front of it is not a durable record of progress. If another stage must know exactly how far the extractor got, that state belongs in a store the worker writes to explicitly, not in lines it hoped would reach a collector.