skip to content

A Python ad-auction bidder is SIGKILLed after a 45-second cold start and its log file ends mid-line — why, and what do you change?

level: seniorimportance: should knowfreq 40%

answer

  1. the cut lands at a buffer boundary
  2. one signal runs no user code
  3. with-blocks and atexit never got a chance
  4. the grace period before escalation is the fix
  5. flush is not the same as durability

basics

~10 s

The tail of the output was still in the process's userspace buffer. SIGKILL cannot be caught, so no flush, with-block exit or atexit handler runs, and the buffer dies with the process.

solid answer

~50 s

The bidder's writes accumulate in the file object's byte buffer and only reach the kernel when it fills or something flushes. On CPython 3.14 that buffer is 128 KiB by default, so a 45-second cold start can produce a lot of unwritten text. `SIGKILL` cannot be caught or handled: the kernel tears the process down immediately, so nothing flushes — not a `with` block's exit, not an `atexit` handler, not interpreter shutdown. Whatever the kernel already had is on disk; the rest is gone, and because the cut falls at a buffer boundary rather than a line boundary the file ends mid-line and looks truncated. The fix is to stop holding output: open the log with `buffering=1`, or use `logging` with a `StreamHandler`, which flushes each record. Then handle `SIGTERM` to flush and exit during the orchestrator's grace period, so the escalation to `SIGKILL` never arrives with data pending.

code

python · 13 lines
python
import signal
import sys


def drain_and_exit(signum, frame):
    print(f"shutting down on signal {signum}", flush=True)
    sys.stdout.flush()
    sys.exit(0)


signal.signal(signal.SIGTERM, drain_and_exit)
sys.stdout.reconfigure(line_buffering=True)
print("handler installed before slow startup work")

go deeper

for a junior

Recall that written text can still be sitting in a buffer and that a killed process loses it. Knowing the file is cut where the last write syscall ended, not where the code stopped, is the core idea.

for a middle

Explain which cleanup paths do and do not run: with-block exit, atexit and interpreter shutdown all need the process to keep running, and SIGKILL cannot be caught. Then name the per-line flushing options.

for a senior

Show the layered response you would actually ship: flush diagnostics per record, drain on SIGTERM inside the grace period, install the handler before slow startup, and attack the cold start that keeps triggering the kill.

for a principal

Own the loss policy across services: how much buffered diagnostic output the organisation is willing to lose, how that is enforced by defaults in the base image and logging setup, and how it is kept distinct from data-durability guarantees.

## Reading the symptom correctly A file that ends in the middle of a line is telling you where the last successful `write` system call stopped, not where the program stopped. Python's file objects batch bytes in a userspace buffer and hand them to the kernel in chunks. Those chunks have nothing to do with line boundaries, so the last thing the kernel received almost always ends mid-record. If the process had exited normally, the closing flush would have completed the line. A mid-line ending is therefore a signature of an abrupt death with data still buffered — not of a disk problem, not of a corrupted write. The missing volume tells you the same story. With the default policy on CPython 3.14 the buffer is `io.DEFAULT_BUFFER_SIZE`, 128 KiB — up from 8 KiB in older releases. A bidder emitting startup progress for 45 seconds can easily lose everything it wrote, because it never produced 128 KiB and so never triggered a single flush. ## Why nothing rescued it Engineers usually assume some cleanup path would have saved the tail. Under `SIGKILL` none of them exist: * **`with` blocks.** `__exit__` closes the file and flushes it, but it only runs if the frame unwinds. `SIGKILL` never returns control to Python. * **`atexit` handlers.** They run during normal interpreter shutdown, which is skipped entirely. * **`__del__` on the file object.** Same problem; no garbage collection happens in a process that no longer exists. * **Signal handlers.** `SIGKILL` is one of the two signals a process cannot catch, block or ignore. `signal.signal(signal.SIGKILL, ...)` is not a design you can fall back on — the kernel simply removes the process. The same loss happens for a much more ordinary reason: `os._exit()` also skips all cleanup, which is why a child process that ends that way after `fork` can lose buffered output too. What survives is exactly what the kernel already accepted. Once a `write` returns, the data is in the page cache and other processes can read it whether or not the writer lives; that is why flushing is the boundary that matters here. ## The fixes, in the order to apply them **1. Flush per record where the output is diagnostic.** Open the file with `buffering=1` in text mode so each newline flushes, or route it through `logging`: a `logging.StreamHandler` flushes after every record it emits, which is why properly logged services do not have this bug and services that `print()` do. For the container's stdout, `PYTHONUNBUFFERED=1` or `sys.stdout.reconfigure(line_buffering=True)` does the same job. **2. Use the grace period.** Orchestrators do not open with `SIGKILL`. They send `SIGTERM`, wait a configured grace period, and escalate only if the process is still alive. That window is the real fix: install a `SIGTERM` handler that stops accepting work, flushes the streams and exits. If the bidder's cold start makes it slow to become responsive, make sure the handler is installed before the slow initialisation begins, not after — otherwise the very window you are trying to protect is the window with no handler. **3. Attack the 45-second cold start itself.** A process that takes 45 seconds to be useful will be killed by liveness checks, deploy timeouts and autoscaler churn on a regular basis, and every one of those kills is another chance to lose state. Shrinking startup work, or reporting readiness separately from liveness so the platform stops treating a slow start as a hang, removes the recurring cause rather than the symptom. ## The costs you are trading Flushing per line means one system call per line. For a bidder writing a few hundred lines a second that is invisible; for a hot path writing tens of thousands, it is real, and the answer is to flush on an interval or at checkpoints rather than on every record, accepting a bounded loss window instead of an unbounded one. Be explicit about that window: "we can lose up to one second of diagnostics" is an engineering decision; "we lose whatever happened to be buffered" is not. ## The boundary worth naming Flushing gets bytes out of your process and into the kernel, which is what protects you from a killed process. It does not protect you from a machine losing power, because the data may still be in the page cache; forcing it onto the storage device is a further step and a different guarantee. Interviewers listen for a candidate who keeps those two apart, because conflating them produces both unnecessary syncing on hot paths and false confidence about what survives a crash. ## What a strong answer sounds like Diagnose from the mid-line cut, name the userspace buffer and its 128 KiB default, state plainly that `SIGKILL` runs no Python code at all, then give a layered fix: flush per record for diagnostics, catch `SIGTERM` and drain within the grace period, and reduce the cold start that keeps inviting the kill — with an explicit note on the syscall cost of the flushing policy you chose.

  • Would an atexit handler that flushes the file have saved the tail?
    No. atexit callbacks run during normal interpreter shutdown, and SIGKILL skips all of it — the kernel removes the process without returning control to Python. An atexit flush helps for ordinary exits and unhandled exceptions, where the streams are flushed anyway, and does nothing for the case you are actually debugging.
  • Where does the grace period come in, and what must you get right?
    Orchestrators send SIGTERM first and escalate to SIGKILL only after a timeout, so the fix is a SIGTERM handler that stops taking work, flushes and exits inside that window. Two details matter: install the handler before the slow startup work rather than after it, and make sure the drain finishes well inside the configured grace period.
  • How would you keep the flushing cost bounded on a high-volume writer?
    Do not flush per record. Flush on an interval or at checkpoints, so the syscall rate is bounded by time rather than by throughput, and state the loss window explicitly — for example, at most one second of diagnostics. Alternatively split the streams: flush the low-volume lifecycle events per line and leave the high-volume records block-buffered.
  • Once the writes are flushed, is the data safe from a machine failure?
    Not necessarily. Flushing moves bytes from the process's buffer into the kernel, which protects them from the process dying. They can still be in the page cache when the machine loses power, so surviving that requires forcing the file's data to the storage device — a separate, more expensive guarantee, and one to reach for deliberately rather than by habit.

saying these in an interview costs you the question

  • Blames disk corruption for a file that ends mid-line
  • Proposes catching SIGKILL with a signal handler
  • Assumes a with-block always flushes, whatever kills the process
  • Treats flushing as a durability guarantee against power loss
  • Flushes every record on a hot path without costing the syscalls
  • Ignores the SIGTERM grace period that precedes the kill

context

resolving links…