Reading a Traceback

## What the runtime prints When a Go program panics, or the runtime hits a fatal error, it writes a **traceback** to standard error and exits. The traceback has a fixed shape: 1. the panic message (`panic: send on closed channel`, `panic: runtime error: index out of range [3] with length 3`); 2. for a memory fault, a `[signal SIGSEGV: ...]` line describing the fault; 3. a goroutine header — `goroutine 23 [running]:` — giving the goroutine's id and the state the runtime recorded for it; 4. the **frames**, innermost call first; 5. optionally a `created by ...` line saying which function ran the `go` statement that started this goroutine. ## One frame is two lines ``` main.(*Consumer).ack(0xc0000ac000, 0xc0000b4030) /srv/worker/consume.go:57 +0x64 ``` **Line one — the function.** The name is fully qualified: package first, then the receiver in parentheses for a method (`main.(*Consumer).ack` is method `ack` with receiver type `*Consumer`), then the function. Compiler-generated names appear here too: `main.run.func1` is the first anonymous function literal written inside `run`, and function literals nested inside that one gain further suffixes. Generic code shows the instantiated shape in brackets. The values in parentheses are raw machine words scraped from the frame's argument area, printed in hex — they are a hint about the arguments, not a decoded list of them. **Line two — where.** Indented with a tab: the source file path and line, then `+0x64`, the byte offset of the program counter from the start of that function's machine code. The path is the path as it was compiled, which is why a binary built elsewhere can print a directory that does not exist on your machine. ## The order, and which line actually broke Frames are printed **innermost first**. The top frame is the function that was executing when the panic happened, and its file:line is the statement that panicked. Every frame below it is a caller, and for those the runtime resolves the *return address* back to a source line — so the line you see is the **call site**, the call that had not yet returned. A middle frame saying `consume.go:31` does not mean line 31 is broken; it means the call written on line 31 is still in progress. Reading a middle frame as "the bug" is the most common misreading of a traceback. The bottom frame is the goroutine's entry point: `main.main()` for the main goroutine, or the function passed to `go` for any other. For a non-main goroutine the stack is followed by ``` created by main.(*Consumer).run in goroutine 1 /srv/worker/consume.go:29 +0x9a ``` which names the function containing the `go` statement, the file and line of that statement, and — in recent Go — the id of the goroutine that executed it. That line is how you get from "some goroutine is stuck" to "this line spawned it". ## What is hidden At the default traceback level the runtime **elides its own frames**, so the first frame you see is your code even when the panic was raised deep inside the runtime (an index check, a nil map write). Raising the traceback level un-hides them, at the cost of a much longer dump. Very deep stacks — runaway recursion is the usual cause — are truncated, and the runtime says so with a line reading `...additional frames elided...`. Inlining also changes what you read. The compiler inlines small functions, but the traceback still prints the inlined function as its own frame line so the call chain reads correctly. The consequence is that a printed frame does not always correspond to a physical stack frame, and the `+0x` offset on such a line is an offset inside the enclosing function that absorbed it. When a line contains several calls, or optimisation has blurred line attribution, the offset is the precise thing: `go tool objdump` on the binary maps it back to an instruction. ## Reading one in practice Work top down. Read the panic message first — it usually names the operation (`send on closed channel`, `assignment to entry in nil map`). Then take the top frame's file:line and open it; that is the statement. Then walk down until you reach a frame in code you own, which tells you how control got there. Finally, if the goroutine is not `main`, read the `created by` line to learn who started it and from where — for a background goroutine that is often the only pointer back to the code that owns the bug, because nothing in the stack above it mentions the caller that set the work up. Two practical prerequisites make all of this work: the binary and the source must be the same revision, or the file:line lands on the wrong line; and the crash output must be captured somewhere, because a traceback that scrolled past in a container's stdout is worth nothing.