Why does Python's print() output appear instantly in a terminal but arrive late when stdout is piped?

## The stack under `sys.stdout` `sys.stdout` is not the file descriptor. It is an `io.TextIOWrapper` — the text layer that encodes `str` to bytes — wrapping a buffered binary writer, which wraps the raw descriptor. `sys.stdout.buffer` is that binary layer. Two of the wrapper's attributes describe the policy: `line_buffering` (flush on every newline) and `write_through` (pass characters straight down to the binary layer instead of holding them in the text layer). The binary layer has its own block buffer underneath both. ## The policy is chosen once, at startup, from `isatty()` When the interpreter initializes it inspects each standard stream: * **stdout attached to a terminal** → line-buffered. Interactive users need to see output as it is produced, and a terminal is slow enough that per-line syscalls do not matter. * **stdout attached to anything else** — a pipe, a file, a socket, a log collector → block-buffered. Bulk output is the expected case; the buffer amortizes syscalls across many lines. * **stderr** → line-buffered whether or not it is a terminal. This changed in **Python 3.9**; before that, redirected stderr was fully buffered. Nothing re-evaluates this later. The decision is made from what the descriptor looked like at startup, so a program cannot "become" interactive by noticing a terminal halfway through. ## What that means in practice Run a script in a shell and the output looks immediate. Pipe the same script and it can look hung, then emit everything at once at the end. Nothing is broken — the lines are in the process's own memory, waiting. Three things drain that buffer: it fills, someone calls `flush()`, or the interpreter finalizes normally and flushes the standard streams during shutdown. The third case is the one people rely on without realising, and it is the one that fails. Normal shutdown covers a clean return from `main`, `sys.exit()`, and even an uncaught exception — the traceback is printed and finalization still flushes. It does **not** cover a hard kill from outside the process, an out-of-memory kill, `os._exit()`, or a crash inside a native extension. In those cases the buffered text is simply gone: it was never handed to the kernel. ## The asymmetry that confuses people Because stderr is line-buffered and stdout is not, a redirected run produces its two streams at different rates. A program that prints progress to stdout and a traceback to stderr will show the traceback promptly and the progress lines later, in one blob, at shutdown. If a collector merges the two by arrival time, the story it tells is wrong — the breadcrumbs appear to come *after* the failure they preceded. Recognising that as a buffering artefact rather than a logic bug is exactly what the interviewer is looking for. ## Changing the policy From outside, before the process starts: * `python -u` — forces stdout and stderr unbuffered. Since Python 3.7 this also makes the text layer unbuffered, so `sys.stdout.write_through` becomes `True`. It has no effect on stdin. * `PYTHONUNBUFFERED=1` — identical effect, set in the environment, which is what you use when you do not control the command line. From inside, as early as possible: * `sys.stdout.reconfigure(line_buffering=True)` — flush at every newline while keeping partial writes cheap. Available on `io.TextIOWrapper` since Python 3.7. * `print(..., flush=True)` — one call only. What you cannot do is change the decision for output that was already produced before your code ran, or for bytes a native extension writes directly to the descriptor without going through `sys.stdout`. ## Costs and the sensible default Unbuffered means at least one `write` syscall per write call, and the text layer stops coalescing, so a chatty loop pays for it. Line buffering is the usual middle ground: latency of one line, one syscall per line. Full block buffering is right for bulk data — a program piping megabytes into another process should keep it. A useful rule of thumb: if a human or a supervisor is reading the output as it is produced, make the stream low-latency and accept the syscalls; if the output is a data stream being consumed in bulk, leave the block buffer alone and flush at the points that matter. ## Checking, rather than guessing `sys.stdout.isatty()`, `sys.stdout.line_buffering` and `sys.stdout.write_through` tell you exactly what you have at runtime, and printing them under a pipe versus a terminal is a two-line experiment that settles most arguments about this.