A Go tool's progress counter shows in a terminal but never when piped into another command. Why?

seniorshould knowfreq 34%

answer

which descriptor did the shell replace?
run it bare, then piped into cat
the counter became the next command's input
diagnostics belong on the other stream
stderr stays on the terminal

basics

~20 s

The counter is being written to os.Stdout. When the output is piped, the shell replaces descriptor 1, so those lines become the next command's input rather than reaching the screen. Human-facing progress belongs on os.Stderr.

solid answer

~50 s

Run the same command twice - bare, then piped into cat - and compare. If the piped run shows the counter mixed into the data, the tool is printing it to os.Stdout: the shell replaced descriptor 1 with a pipe, so the counter is now the downstream command's input, invisible to the user and corrupting whatever parses it. The fix is to move every human-facing message to os.Stderr, which the shell leaves pointed at the terminal and which Go never buffers. If instead the counter appears nowhere at all, either the tool holds it in its own bufio.Writer over stdout and only spills at buffer boundaries, or it deliberately suppressed the display because a Stat on os.Stdout found no os.ModeCharDevice. Gate the animated form on os.Stderr being a terminal, keep a plain periodic line otherwise, and keep stdout carrying nothing but data.

Run the same command twice - bare, then piped into `cat` - and compare. If the piped run shows the counter mixed into the data, the tool is printing it to `os.Stdout`: the shell replaced descriptor 1 with a pipe, so the counter is now the downstream command's input, invisible to the user and corrupting whatever parses it. The fix is to move every human-facing message to `os.Stderr`, which the shell leaves pointed at the terminal and which Go never buffers. If instead the counter appears nowhere at all, either the tool holds it in its own `bufio.Writer` over stdout and only spills at buffer boundaries, or it deliberately suppressed the display because a `Stat` on `os.Stdout` found no `os.ModeCharDevice`. Gate the animated form on `os.Stderr` being a terminal, keep a plain periodic line otherwise, and keep stdout carrying nothing but data.

code

go · 16 lines

fi, err := os.Stderr.Stat()
animate := err == nil && fi.Mode()&os.ModeCharDevice != 0

out := bufio.NewWriter(os.Stdout)
defer out.Flush()

for i, rec := range records {
	fmt.Fprintln(out, rec)
	if i%1000 == 0 {
		if animate {
			fmt.Fprintf(os.Stderr, "\r%d/%d", i, len(records))
		} else {
			fmt.Fprintf(os.Stderr, "processed %d\n", i)
		}
	}
}

go deeper

for a junior

Recall that piping or redirecting replaces stdout only, so anything the user must see while the tool runs has to be printed to os.Stderr.

for a middle

Explain how the shell rewires descriptor 1, why Go never line-buffers stdout for a terminal, and how a bufio.Writer over stdout delays output until it spills.

for a senior

Walk the diagnosis out loud: run the command bare and piped into cat, compare what appears and when, then decide which stream each message belongs on and where a flush is owed.

for a principal

Own the output contract for tools other teams script around: stdout is data with a stable shape, stderr is human, progress is decorative and suppressible, and every automatic guess has a flag that beats it.

The two-run diagnostic

Before theorising, run the identical command twice:

text

$ tool big.csv > /dev/null
$ tool big.csv | cat > /dev/null

What differs between those runs tells you which of three causes you have. Only descriptor 1 changed - stderr is the terminal in both.

Cause 1: the counter is on stdout

By far the most common. Somebody wrote fmt.Printf("\rprocessed %d\n", n) because that is the shortest thing to type, and fmt.Printf means os.Stdout. In a terminal it appears, tangled up with the results but visible, so nobody notices. The moment a user writes tool | jq . the counter is fed to the downstream program as data. Two failures at once: the human sees nothing, and the consumer receives lines it cannot parse. With tool > out.csv the counter ends up inside the data file.

The fix is not conditional printing; it is the right stream. fmt.Fprintf(os.Stderr, ...) for anything a human reads. Stderr survives > and | untouched, and Go never buffers it.

Cause 2: your own buffer is holding it

If the tool wraps stdout - out := bufio.NewWriter(os.Stdout) - then everything written through out, counter included, sits in a 4 KB buffer until it spills. Output arrives in bursts rather than as it is produced, and the tail arrives only at the final Flush.

A useful thing to know here: this is not terminal-dependent in Go. Unlike C stdio, Go does not line-buffer to a terminal and block-buffer to a pipe, so a buffered writer behaves identically in both runs. If the behaviour genuinely differs between the two runs, buffering in your program is not the whole story - which is what makes the two-run comparison diagnostic rather than decorative.

Cause 3: you suppressed it on purpose

Many tools already stat their output and skip the animation when it is not a terminal. That is correct behaviour, but the check is often written against the wrong stream: os.Stdout.Stat() when the display goes to os.Stderr. Then tool | other suppresses progress even though stderr is still the terminal and the user is sitting there watching. Stat the stream you decorate.

The shape that works

stdout: data only. Nothing else, ever. Its format is a contract other people's scripts depend on.
stderr: everything human. Errors, warnings, progress, verbose logging. Unbuffered, so it is on screen when written.
Decoration is conditional, information is not. When stderr is a character device, use \r redraws and colour. When it is not, emit one plain line every N records or every N seconds so a log stays readable.
Flags win. --quiet and an explicit progress flag override the automatic decision in both directions.
Flush before you redraw. If a human is watching both streams on one terminal and stdout is buffered, flush the stdout writer before writing a progress redraw, or the two appear out of order.

The pipeline citizenship you also inherit

A tool that is piped will eventually be piped into something that exits early - tool | head -5. The write to descriptor 1 then fails with EPIPE. Go handles descriptors 1 and 2 specially here: rather than surfacing an error, the runtime raises SIGPIPE and the program dies from the signal, which is the Unix-correct behaviour for a filter. On any other descriptor the EPIPE is returned to your code as an error instead. So do not build retry logic around a failed stdout write; quiet death is the contract.

Why this is a senior question

The bug is trivial once seen, and it is the mark of somebody who has shipped a tool other people script around. The judgment being tested is whether you treat stdout as an interface with a contract - stable, machine-readable, nothing but data - and stderr as the channel for everything else, rather than treating both as "printing".

## The two-run diagnostic Before theorising, run the identical command twice: ```text $ tool big.csv > /dev/null $ tool big.csv | cat > /dev/null ``` What differs between those runs tells you which of three causes you have. Only descriptor 1 changed - stderr is the terminal in both. ## Cause 1: the counter is on stdout By far the most common. Somebody wrote `fmt.Printf("\rprocessed %d\n", n)` because that is the shortest thing to type, and `fmt.Printf` means `os.Stdout`. In a terminal it appears, tangled up with the results but visible, so nobody notices. The moment a user writes `tool | jq .` the counter is fed to the downstream program as data. Two failures at once: the human sees nothing, and the consumer receives lines it cannot parse. With `tool > out.csv` the counter ends up inside the data file. The fix is not conditional printing; it is the right stream. `fmt.Fprintf(os.Stderr, ...)` for anything a human reads. Stderr survives `>` and `|` untouched, and Go never buffers it. ## Cause 2: your own buffer is holding it If the tool wraps stdout - `out := bufio.NewWriter(os.Stdout)` - then everything written through `out`, counter included, sits in a 4 KB buffer until it spills. Output arrives in bursts rather than as it is produced, and the tail arrives only at the final `Flush`. A useful thing to know here: this is *not* terminal-dependent in Go. Unlike C stdio, Go does not line-buffer to a terminal and block-buffer to a pipe, so a buffered writer behaves identically in both runs. If the behaviour genuinely differs between the two runs, buffering in your program is not the whole story - which is what makes the two-run comparison diagnostic rather than decorative. ## Cause 3: you suppressed it on purpose Many tools already stat their output and skip the animation when it is not a terminal. That is correct behaviour, but the check is often written against the wrong stream: `os.Stdout.Stat()` when the display goes to `os.Stderr`. Then `tool | other` suppresses progress even though stderr is still the terminal and the user is sitting there watching. Stat the stream you decorate. ## The shape that works - **stdout: data only.** Nothing else, ever. Its format is a contract other people's scripts depend on. - **stderr: everything human.** Errors, warnings, progress, verbose logging. Unbuffered, so it is on screen when written. - **Decoration is conditional, information is not.** When stderr is a character device, use `\r` redraws and colour. When it is not, emit one plain line every N records or every N seconds so a log stays readable. - **Flags win.** `--quiet` and an explicit progress flag override the automatic decision in both directions. - **Flush before you redraw.** If a human is watching both streams on one terminal and stdout is buffered, flush the stdout writer before writing a progress redraw, or the two appear out of order. ## The pipeline citizenship you also inherit A tool that is piped will eventually be piped into something that exits early - `tool | head -5`. The write to descriptor 1 then fails with `EPIPE`. Go handles descriptors 1 and 2 specially here: rather than surfacing an error, the runtime raises `SIGPIPE` and the program dies from the signal, which is the Unix-correct behaviour for a filter. On any other descriptor the `EPIPE` is returned to your code as an error instead. So do not build retry logic around a failed stdout write; quiet death is the contract. ## Why this is a senior question The bug is trivial once seen, and it is the mark of somebody who has shipped a tool other people script around. The judgment being tested is whether you treat stdout as an interface with a contract - stable, machine-readable, nothing but data - and stderr as the channel for everything else, rather than treating both as "printing".

What happens to a Go filter when the downstream command exits early, as in `tool | head -5`?
The write to descriptor 1 fails with EPIPE, and Go treats descriptors 1 and 2 specially: the runtime raises SIGPIPE and the program exits from that signal rather than returning an error. That is the Unix-correct behaviour for a filter. On any other descriptor the EPIPE is delivered to your code as an ordinary write error, so only stdout and stderr behave this way.
How do you keep buffered stdout output and a live progress display from appearing out of order?
Put them on different streams and flush deliberately: data through the `bufio.Writer` on `os.Stdout`, progress straight to unbuffered `os.Stderr`. If a human is watching both on one terminal, flush the stdout writer before each progress redraw, otherwise buffered data appears after progress that logically preceded it. Never write some records through the wrapper and others directly to `os.Stdout`.
A user redirects stderr to a log file and it fills with carriage returns and escape codes. What went wrong?
The animated form is being emitted unconditionally. The stream choice is right - progress belongs on stderr - but the rendering has to be conditional: `Stat` `os.Stderr`, and when `os.ModeCharDevice` is absent print one plain line per interval with no `\r` and no colour. A quiet flag should be able to silence it entirely regardless of what detection concludes.

saying these in an interview costs you the question

Sends progress or status lines to os.Stdout
Assumes Go flushes stdout automatically when it is a terminal
Runs the terminal check on os.Stdout while decorating os.Stderr
Wraps os.Stderr in a bufio.Writer and rarely flushes it
Adds retry logic after a stdout write fails on a broken pipe
Suppresses progress entirely instead of degrading it to plain lines

A Go tool's progress counter shows in a terminal but never when piped into another command. Why?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

The two-run diagnostic

Cause 1: the counter is on stdout

Cause 2: your own buffer is holding it

Cause 3: you suppressed it on purpose

The shape that works

The pipeline citizenship you also inherit

Why this is a senior question

▾common follow-ups3 questions

▾red flags6 flags

The two-run diagnostic

Cause 1: the counter is on stdout

Cause 2: your own buffer is holding it

Cause 3: you suppressed it on purpose

The shape that works

The pipeline citizenship you also inherit

Why this is a senior question

answer

go deeper