A Go service stalls for four seconds twice a day; how do you get an execution trace of it?

## Why the obvious approach fails The standard capture paths are all *prospective*: `runtime/trace.Start` records from now, `go test -trace` records a run you launch, `/debug/pprof/trace?seconds=5` records the next five seconds. Every one of them requires you to be present, and pointed at the right instance, before the interesting thing happens. A stall that lasts four seconds and occurs twice a day gives you a window you cannot hit by hand, on one instance out of many, and by the time an alert reaches a human the evidence is gone. The general answer to "the event is over before I can attach" is a **flight recorder**: keep recording continuously into a bounded circular buffer, throw the old data away, and only persist when something tells you the recent past was interesting. ## The flight recorder in runtime/trace `runtime/trace.FlightRecorder` does exactly that. You construct it with a config, start it, and leave it running for the life of the process: ```go fr := trace.NewFlightRecorder(trace.FlightRecorderConfig{ MinAge: 5 * time.Second, MaxBytes: 32 << 20, }) if err := fr.Start(); err != nil { log.Fatal(err) } defer fr.Stop() ``` - **MinAge** asks the recorder to retain at least that much recent history, so you know the dump will cover the run-up to the trigger and not just the last few milliseconds. - **MaxBytes** caps the memory it will hold. The two are a request, not a contract: on a very busy process the byte cap binds first and you get a shorter window than the age you asked for. That tension is the whole tuning decision — a gateway holding tens of thousands of connections generates events fast, so retaining five seconds of it costs real memory. When the trigger fires, `fr.WriteTo(w)` writes the retained window out in the ordinary execution-trace format. Save it to a file (name it with the timestamp and the instance) and open it later with `go tool trace`. Recording continues, so a second incident later in the day can be dumped too — though you should rate-limit dumps so a flapping trigger cannot fill a disk. ## The trigger is the part you have to design A flight recorder with no trigger is just overhead. In-process detection is what makes it work, because only the process knows in real time that something took too long. Practical triggers: - A watchdog goroutine that ticks on a short interval and measures its own lateness — if a tick meant for 100 ms arrives 3 s late, the whole process was stalled. - A latency guard around the service's own work: if a handler or a broadcast round exceeds its budget by a wide margin, dump. - A health probe that is failing locally before the load balancer notices. Make the trigger cheap, make it fire on an *outlier* rather than a mild regression, and give it a cooldown. And dump from a goroutine that is not itself part of the stalled path if you can. ## What you do with the dump You now hold a trace whose window straddles the incident — the seconds before, the stall itself, and the recovery. That is what makes it worth the effort: the cause of a stall is almost always in the run-up, and the run-up is exactly what an after-the-fact capture misses. Open it with `go tool trace`, look at the shape of the window first, and work outwards from the moment the timeline goes quiet. ## Alternatives, and when they are better - **Continuous short captures on a loop**, keeping only the ones whose window overlapped an anomaly, is a poor substitute: it is more expensive, writes constantly, and still misses events that straddle a boundary. - **Trigger a capture from outside** — an alert that curls the trace endpoint — is reasonable when the incident lasts minutes, not seconds. Four seconds is far too short for a human or even an alerting pipeline in the loop. - **Cheaper signals first.** Before you ship a flight recorder, check whether a counter, a lightweight runtime metric, or a periodic runtime tracer already tells you enough. A trace is the heavy artillery; reach for it when you need to know exactly what was and was not running. ## The costs, stated plainly The tracer is on permanently, so you carry its CPU cost continuously rather than for five seconds — small in recent Go, but not zero, and highest on precisely the busiest process. You hold the buffer's memory for the life of the process. And you have added a code path that writes a file on a production host, which needs a size cap, a rate limit and a known destination. Those are all acceptable prices for catching a twice-daily four-second stall that has otherwise resisted every attempt at capture; they are not acceptable prices for idle curiosity, and saying so is part of the answer.