skip to content

What does GOTRACEBACK=all change about the output of a crashing Go program?

level: juniorimportance: must knowfreq 50%

answer

  1. the default is deliberately stingy
  2. one stack, not all of them
  3. an environment variable, not a rebuild
  4. single is the default, all is the fix

basics

~20 s

By default a Go crash prints a stack for the goroutine that panicked. GOTRACEBACK=all makes the runtime print a stack for every user-created goroutine instead. It is an environment variable, so it costs a restart, not a rebuild.

solid answer

~50 s

The Go runtime reads GOTRACEBACK at start-up to decide how much it prints when the program dies from an unrecovered panic or a fatal runtime error. The default level, `single`, prints the failure message, then one stack for the current goroutine with runtime-internal frames elided, and exits with status 2. `all` adds a stack for every user-created goroutine; `system` goes further and also shows run-time function frames and the goroutines the runtime created for itself; `none` suppresses the stacks but still prints the failure message. Because it is an environment variable read by the process, you turn it up by editing wherever the service's environment is defined and restarting - no code change and no new build. You reach for it when the crash report names a goroutine that is obviously the victim rather than the culprit.

code

text · 25 lines
text
$ ./agent
panic: send on closed channel

goroutine 21 [running]:
main.(*uploader).flush(...)
	/opt/agent/upload.go:88
...

$ GOTRACEBACK=all ./agent
panic: send on closed channel

goroutine 21 [running]:
main.(*uploader).flush(...)
	/opt/agent/upload.go:88
...

goroutine 34 [chan receive]:
main.(*collector).drain(...)
	/opt/agent/collect.go:52
...

goroutine 8 [select]:
main.(*agent).run(...)
	/opt/agent/main.go:117
...

go deeper

for a junior

Know that the default crash output covers only the goroutine that panicked, and that GOTRACEBACK=all is how you ask for the rest. Be ready to say it is an environment variable set before the process starts.

for a middle

Explain what each level adds - none, single, all, system - and that the runtime reads the variable at start-up, so the change costs a restart rather than a build. Note that the failure message still prints at none.

for a senior

Show judgment about when to raise it: a crash whose stack names an innocent goroutine, or a failure you cannot reproduce. Talk about what full dumps cost on a service with tens of thousands of goroutines and whether your log pipeline will keep the whole report.

for a principal

Own the default posture: which services ship with all in their environment, how much crash output the logging path can absorb without truncating it, and whether reports carrying function arguments and file paths may leave the machine they were produced on.

## What GOTRACEBACK is `GOTRACEBACK` is an environment variable read by the Go runtime when the process starts. It controls **how much the runtime prints when the program fails** - that is, when a panic reaches the top of a goroutine without being recovered, or when the runtime itself hits a condition it cannot continue from. It has no effect on a panic that some frame recovers: a recovered panic prints nothing at all unless your own code logs it. The levels, from quietest to loudest: | level | what you get | |---|---| | `none` | the failure message only - no goroutine stacks | | `single` | **the default**: the failure message plus one stack, for the goroutine that failed, with run-time internal frames elided | | `all` | the same, plus a stack for every **user-created** goroutine | | `system` | the same as `all`, plus run-time function frames and the goroutines the runtime creates for itself | | `crash` | the same as `system`, and then the process aborts in an operating-system-specific way instead of exiting | ## Why the default hurts Go's crash report is written top-down from the failing goroutine, so it answers "where was this goroutine when it died" and nothing else. That is exactly right when the bug is local - a slice index out of range, a nil pointer dereference on a struct field. It is useless for the large class of Go bugs where the failing goroutine is the **victim**: it sent on a channel that some other goroutine closed, it read a value some other goroutine was concurrently writing, it woke up on a context that a caller you cannot see cancelled. In those cases the stack you need belongs to a goroutine the default level never prints, and the report you are staring at names only the innocent party. Raising the level to `all` prints one block per user goroutine, each headed by a goroutine number and a state, so the goroutine that closed the channel or is holding the thing everyone waits on is in the same report. ## Why `all` is not the default Because dumps are not free. A busy Go service can hold tens of thousands of goroutines, and a stack for each of them is megabytes of text written synchronously to standard error while the process is on its way down. That output has to go somewhere: a log pipeline with a per-line or per-message limit will truncate it, a supervisor that captures a fixed-size buffer will keep the tail and lose the head, and a slow terminal will simply take a long time. `single` keeps the common case short and readable, and leaves the expensive setting as something you opt into. This matters most on software you do not operate. An agent shipped to customer-owned hardware, restarted by whatever supervisor that machine runs, gives you exactly one artefact after a crash: whatever the supervisor captured from standard error. If that report is one goroutine deep because nobody set GOTRACEBACK, the investigation is over before it starts, and the next data point is the next crash - possibly days away. ## Setting it It is process environment, so anywhere the process's environment is defined works: a service unit file, a container's environment block, an init script, or simply prefixing the command: ``` GOTRACEBACK=all ./agent ``` The value is read at start-up, so an already-running process keeps whatever it started with; you restart to change it. A program can also set the level from inside itself, which is the route to take when you cannot touch the environment on the target machine at all. ## The shape of the output Whatever the level, the report opens with the failure - a `panic:` line carrying the panic value, or a fatal runtime error message - then a blank line, then one block per goroutine, each starting with a header naming the goroutine number and its state. `none` keeps the first part and drops the rest. When the program is run under `go run`, the tool adds a final `exit status 2` line, which is the exit code the runtime uses for a program that died this way. ## What to say in an interview Name the default (`single`), name the fix (`all`), say it is an environment variable read at start-up so it needs a restart rather than a build, and be able to explain why the loud setting is not the default: on a service with tens of thousands of goroutines, the dump is both enormous and something the log pipeline may not survive.

  • Does raising GOTRACEBACK require rebuilding the Go binary?
    No. The runtime reads GOTRACEBACK from the process environment when it starts, so you set it wherever that environment is defined and restart the service. The binary is untouched, which is what makes it usable on a machine where shipping a new build takes days.
  • What does GOTRACEBACK=system show that all does not?
    Everything `all` prints, plus frames for run-time functions that are normally elided and the goroutines the runtime creates for itself - the background mark workers, the scavenger, the finalizer goroutine. Use it when you suspect the runtime or cgo rather than your own code; otherwise it is noise around the frames you actually want.
  • Why is single, rather than all, the default level?
    Because a service can hold tens of thousands of goroutines and a stack for each is megabytes written synchronously to standard error as the process dies. Most panics are local to the failing goroutine, so the cheap report is usually the right one, and the expensive one is opt-in.

It is a crash report that shows only the vehicle that stopped: useful when it hit a wall by itself, useless when something else pushed it.

saying these in an interview costs you the question

  • Thinks GOTRACEBACK decides whether the program crashes at all
  • Believes raising it needs a code change or a new build
  • Says GOTRACEBACK=none hides the panic message as well as the stacks
  • Assumes the default report already covers every goroutine
  • Expects a recovered panic to still produce a runtime crash dump