When is GOTRACEBACK=crash worth running in production, and what does it cost?

## What `crash` actually does `GOTRACEBACK=crash` is `system` plus one extra step. The runtime prints the full report - every user goroutine, run-time function frames, the goroutines the runtime created for itself - and then, rather than calling exit with the usual failure status, it crashes the process in an operating-system-specific way. On Unix that means raising SIGABRT, whose default action is to terminate the process and let the kernel write a core. That last part is the whole point, and it is also where most of the misunderstanding lives: **Go does not write the core file.** All Go does is die in a way that asks the operating system to. Whether a file appears is entirely the host's decision. ## What you get for it A core is a snapshot of the process's memory at the moment it died. With the matching binary you can inspect not just the stacks - which the traceback already gave you - but the actual values: the contents of a buffer, what was in a map, which struct field was unexpectedly zero, how many records were queued. For a class of bugs that the stacks alone cannot settle - memory corruption, a suspected runtime or cgo problem, a value that is wrong in a way nobody can reproduce - that is the difference between a theory and an answer. ## What it costs **It may silently produce nothing.** The core size limit for the process may be zero, the kernel's core pattern may point at a handler that drops it, the destination may not be writable. The Go traceback still prints in full, which is exactly why people believe the setting worked while no core was ever written. Verify on the target host, not on your laptop. **Size.** A core is roughly resident memory. On an agent sharing a small local disk with buffered telemetry records waiting to be uploaded, a multi-gigabyte file dropped at the worst possible moment can fill the disk and take out the very buffer you need - a diagnostic that destroys evidence. **Data exposure.** A core is an image of everything the process held: credentials, tokens, decrypted customer data, request bodies. It is not debug output, it is a copy of production memory sitting outside every control you normally apply to that data. Where it may be written, who may read it, how it travels and when it is deleted are questions to answer *before* you enable it, not after somebody attaches one to a ticket. **Termination semantics.** The process dies by signal rather than exiting with the usual failure status. Supervisors that distinguish the two may back off restarts, alert differently, or classify it as a crash loop. **Retrieval.** On customer-operated hardware you rarely have a way to pull a large binary file off the machine, and you may not be permitted to. A setting whose only artefact you cannot collect is not a diagnostic. ## When it is right - On hardware **you** own - a canary instance, a soak-test fleet, one node deliberately configured for it. - For a bug you are **actively chasing**, where stacks have already failed to settle the question, most often a suspected runtime, memory-corruption or cgo problem. - With the host prepared in advance: a checked core limit, a known destination with room, and a documented handling and deletion path for the file. ## When it is wrong As a fleet-wide default "because it is the most detailed level". You will pay disk, restart-behaviour and data-handling costs on every host, mostly for cores nobody opens. ## What to run instead For a service, ship `all`: every user goroutine, no run-time noise, output small enough for a log pipeline to keep whole. Raise to `system` when the suspicion moves below your code. Add a second destination for fatal output, from inside the program, so the report survives a supervisor that discards standard error - that combination answers the majority of production crashes without ever producing a memory image. ## What to say in an interview Name the mechanism (SIGABRT, and the OS writes the core, not Go), name the prerequisite (host core configuration, or you get nothing), and name the two costs that make it a decision rather than a setting: a file the size of your memory on the host's disk, and that file containing everything the process knew.