In a bash script whose output other programs consume, why should progress and warning messages go to standard error while only results go to standard output?

## Two streams, two audiences Every process starts with descriptor 1 (standard output) and descriptor 2 (standard error) open, and by universal convention they carry different things: - **stdout — the product.** What the command was asked to produce, in a form another program can consume. - **stderr — the commentary.** Progress, warnings, errors, anything a human reads to understand what happened. The convention only pays off if you honour it, because callers make plans based on it. `ls`, `grep`, `curl` and `jq` all do; a script that does not is a bad citizen in every pipeline it appears in. ## What breaks when diagnostics land on stdout Every way a caller consumes a script picks up the pollution. **Command substitution.** `$(...)` captures stdout only: ```bash # script.sh prints "Fetching..." and then the value, both to stdout value=$(./script.sh) # value is "Fetching...\n42", not "42" ``` The assignment succeeds and looks fine; the bug shows up later as an arithmetic error or a nonsense comparison. **Pipelines.** `./script.sh | jq '.id'` dies on the first non-JSON word. `./script.sh | wc -l` reports a count that silently includes the progress lines. **File output.** `./script.sh > report.csv` produces a CSV with a banner line in it, which the next tool either rejects or, worse, parses. Move the same messages to descriptor 2 and all of those callers work unchanged, while the messages remain on the terminal where a human can see them. ## The mechanics in a script The idiomatic form is a one-line helper, so that being correct is easier than being wrong: ```bash log() { printf '%s %s\n' "$(date +%FT%T)" "$*" >&2; } die() { log "FATAL: $*"; exit 1; } log "starting run" printf '%s\n' "$result" # the product, on stdout ``` Prefer `printf` over `echo` for anything with backslashes or a leading `-`, since `echo`'s handling of those varies between shells and builds. ## The controls this hands the caller Once the split is honest, the caller gets orthogonal switches: ```bash ./script.sh 2>/dev/null # keep the data, silence the chatter ./script.sh >/dev/null # keep the chatter, drop the data ./script.sh >data 2>run.log # separate both for later inspection ./script.sh >run.log 2>&1 # merge both for a human-read log ``` None of that is available if everything is on one stream; the caller is reduced to `grep -v` on the messages they hope to remove, which breaks the moment a message changes. ## Buffering, and why a merged log can look out of order When a script's two streams are merged into one file, entries sometimes appear in an order that contradicts the code. The reason is buffering in the C runtime that most external tools use: stdio makes stdout **line-buffered when it is a terminal but block-buffered when it is a file or a pipe**, while stderr is unbuffered. A tool's stdout can therefore sit in a 4 KB buffer, flushed only when full or at exit, while its stderr lines hit the file immediately. This catches people diagnosing a hung job: the log shows errors but none of the normal progress, so the job looks stuck at a much earlier point than it is. It is a buffering artefact, not a stall. GNU coreutils ships `stdbuf` to change a child's buffering (`stdbuf -oL mytool`), and many tools have their own flag — `grep --line-buffered`, `python -u`. Bash's own builtins are not affected; the issue belongs to the external programs a script calls. ## When the data *is* log-like The convention still holds for a script whose product is human-oriented text — a report, a rendered table. That report is the product, so it goes to stdout, and the script's own commentary about producing it goes to stderr. Ask "would a caller want to capture this?", not "is this prose?". Also worth saying: neither stream is how a script reports success. That is the exit status, and a script that prints `ERROR:` to stderr while exiting 0 has broken a different contract that every caller relies on just as heavily.