Why do a Python process's stdout and stderr lines land out of order in one redirected file?

## The symptom A program prints progress to stdout and raises. Both streams are redirected into the same file. The file shows the traceback first, then a block of progress lines that were printed before the exception, sometimes with a line torn across the boundary. In a terminal, the same program looks perfectly ordered. Nothing is corrupt and nothing is racing; two buffers with different flush policies are emptying at different times into one destination. ## Two streams, two policies At interpreter startup CPython decides buffering per stream: - `sys.stdout` on a non-terminal is **block buffered**. Bytes accumulate to `io.DEFAULT_BUFFER_SIZE`, which is 131072 on Python 3.14 and was 8192 through 3.13, and only then hit the descriptor. - `sys.stderr` is **line buffered even when it is not a terminal**. Since Python 3.9 this is unconditional; before that, a redirected stderr was block buffered too, and the interleaving problem looked different, and worse, because both tails could arrive late. The file itself is faithful. Both descriptors point at the same open file description when the shell merges them, so writes append in the order they are issued and never interleave mid-write for reasonably sized writes. The order you see *is* the order of the `write(2)` calls. It just is not the order of the `print()` calls, because block buffering decoupled the two. ## Why it is worth understanding rather than papering over The practical damage is diagnostic. Log forensics leans on adjacency: you want the three lines before the error, and the timestamps in the file are the collector's timestamps, not the program's. With stdout lagging by up to a block, adjacency is a lie. Worse, a single stdout write can be split across two `write(2)` calls when the buffer fills mid-line, so a stderr line can appear inside a stdout line. That is the torn line people mistake for corruption. The second lesson is about a common wrong fix. Sending both to the same file is often *chosen* to get one ordered narrative. It cannot deliver that as long as the two streams flush on different schedules, and adding timestamps at the collector does not repair it either, because the collector times the arrival, not the event. ## Fixes, from best to blunt 1. **Make stdout line buffered.** `sys.stdout.reconfigure(line_buffering=True)` once at startup. Both streams then flush per completed line and the interleaving matches causality. Cheap and precise. 2. **Unbuffer both from outside.** `-u` or `PYTHONUNBUFFERED=1`, when you do not control the entry point. Correct order, more syscalls, and partial writes can still interleave since flushing happens per write rather than per line. 3. **Write everything to one stream.** If the narrative matters more than the separation, send all of it to `sys.stderr`, which is line buffered anyway, or all of it to a line-buffered stdout. One buffer cannot reorder against itself. The thing that does *not* fix it is anything on the reading side. A collector cannot pull out bytes the process has not written, and reordering after the fact requires timestamps the program itself stamped. ## Checking the claim in ten seconds ```python import sys print("out", sys.stdout.line_buffering) print("err", sys.stderr.line_buffering, file=sys.stderr) ``` Run it twice, once bare and once with the output going anywhere that is not a terminal. In a terminal both report line buffering. Redirected, stdout reports `False` and stderr still reports `True`, which is the entire explanation in two lines of output. ## The asymmetry is deliberate It is tempting to call the stderr rule an inconsistency. It is a design choice: stderr carries diagnostics, and a diagnostic that is still sitting in a buffer when the process dies has failed at its one job, so it is worth a syscall per line unconditionally. stdout carries the program's actual output, which may be a gigabyte piped into another tool, and there throughput wins. The interview answer is not "Python is inconsistent", it is "the two streams are optimised for different jobs, and merging them into one file exposes that".