How do you make a Go binary always dump every goroutine on a panic without GOTRACEBACK?

## The problem this solves GOTRACEBACK is an environment variable, and setting an environment variable assumes you can reach the place where the process is started. For an agent shipped to customer-operated hardware - launched by whatever supervisor that machine runs, on a box you will never get a shell on - you cannot. The only artefact of a crash is whatever the supervisor captured from standard error before restarting the process, and if the binary shipped with the default level, that artefact is one goroutine deep. The stack that mattered was never printed, and the next chance to learn anything is the next crash. So the crash posture has to be a property of the binary, decided at build time, not of the machine. ## `debug.SetTraceback` ```go debug.SetTraceback("all") ``` One call, taking the same level strings as the environment variable: `none`, `single`, `all`, `system`, `crash`. From that point on the runtime prints at that level when the program fails. The rule that makes it safe to ship: **if the level requested is lower than the one the environment already asked for, the call is ignored.** Code can raise the verbosity above the environment's, never below it. Two consequences follow, and both are worth stating in an interview: - A library or a framework you import cannot quietly turn your crash output down. - An operator who sets `GOTRACEBACK=system` on a machine they do control still gets `system`, even though your `main` asks for `all`. Your baked-in level is a floor, not a ceiling. Put the call first in `main`, before any goroutine is started and before any work that can fail, because it only governs failures that happen after it runs. ## `debug.SetCrashOutput` Raising the level is only half the problem. The report still goes to standard error, and on a machine you do not operate you have no guarantee anyone is keeping standard error. `debug.SetCrashOutput(f, debug.CrashOptions{})` registers **one additional file** that the runtime writes the whole crash report to, *in addition to* standard error - it is not a redirect. Its semantics are worth knowing precisely: - The runtime **duplicates** the file descriptor, so you may close your own `*os.File` as soon as the call returns. - There is exactly one such destination; a later call replaces the earlier one, and passing `nil` disables it. - It has to be set up **before** the crash, which means opening the file during start-up while everything still works. For a shipped agent that already buffers telemetry records on local disk between uploads, the natural destination is a small append-only file next to that buffer, uploaded with the next batch. That turns "the customer says it restarted overnight" into a report you can read at your desk. ```go func main() { debug.SetTraceback("all") if f, err := os.OpenFile("/var/lib/agent/crash.log", os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0o600); err == nil { if err := debug.SetCrashOutput(f, debug.CrashOptions{}); err != nil { log.Printf("crash capture disabled: %v", err) } f.Close() // the runtime duplicated the descriptor } run() } ``` ## What this still will not capture Be explicit about the limits, because a senior answer that promises total coverage is wrong: - **Recovered panics.** If a frame recovers, the runtime prints nothing - there is no failure. Whatever you want recorded, your own code has to record. - **Kills from outside.** A SIGKILL, an out-of-memory kill by the kernel, a power cut or a watchdog reset produce no Go output at all, because the runtime never runs again. - **Failures before your call.** Anything that dies during package initialisation, before `main` reaches `SetTraceback`, prints at whatever the environment says. - **Growth.** The runtime appends the whole report; it does not rotate or bound the file. If crashes are frequent, that file is your problem to cap. ## Choosing the level to ship `all` is almost always the right baked-in level for an agent: every user goroutine, no run-time noise, and a report small enough to upload. `system` is for a support build handed to one customer chasing a suspected runtime or cgo problem - it roughly doubles the noise for frames most readers cannot use. `crash` is a different decision altogether, because it aborts the process to produce a core file, and on hardware you do not own you will never collect it.