Where must the deferred recover live when a controller runs each reconcile in its own goroutine?

## Why placement is the whole question Recovery in Go is scoped to a single goroutine's stack. `recover` returns a value only when it is called by a function that the **panicking goroutine itself** deferred. A controller that spawns one goroutine per reconcile therefore has as many potential crash sites as it has in-flight goroutines, and the loop that started them protects none of them. The consequence is severe and specific to Go: an unrecovered panic in *any* goroutine terminates the **whole process**. It does not quietly kill one worker the way an uncaught exception on a worker thread does in several other runtimes. One malformed object in one reconcile can take down every other reconcile in flight. ## The wrong shape ```go func (c *Controller) start(k Key) { defer func() { if r := recover(); r != nil { /* never fires */ } }() go c.reconcile(k) } ``` Two separate things are wrong. The deferred closure belongs to `start`'s stack, and `start` returns immediately — long before the child goroutine does any work. And even if it were still running, the recovery would be on the wrong goroutine's defer chain. ## The right shape ```go func (c *Controller) startReconcile(k Key) { go func() { defer func() { if r := recover(); r != nil { c.results <- fmt.Errorf("reconcile %v: recovered panic: %v", k, r) } }() c.results <- c.reconcile(k) }() } ``` The boundary is the first statement of the function the `go` statement runs. Everything the goroutine does afterwards is inside it. ## Getting the failure back out A goroutine's return values are discarded — `go f()` has nowhere to put them — so the recovery handler must route the failure deliberately. The usual choices: - **A channel.** Send the wrapped error on the same channel the goroutine would have sent a normal result on. The reconcile loop receives one value per item either way and can requeue the failed key. - **Shared state under a lock.** Record the failure against the item's key in the status map, with the mutex held. Never write a shared variable from the handler without synchronisation; that swaps a panic for a data race. - **A callback supplied by the starter.** Useful for a library where the caller decides what a failed unit of work means. Whichever you choose, include the key of the item that failed. A recovered panic with no identity is nearly useless when a thousand objects are reconciling and one of them is poisonous. ## Making it structural rather than a habit The failure mode here is a missed `go` statement, not a misunderstood one. Every developer on the team knows the rule; someone still writes `go c.reconcile(k)` in a hurry. Two practices help: 1. **One spawn helper.** Give the controller a single `func (c *Controller) spawn(k Key, f func() error)` that installs the boundary and publishes the result, and require all concurrency to go through it. Now the boundary exists in exactly one place and can be reviewed once. 2. **Review rule.** A bare `go` keyword in a long-lived service is a review comment by default. It is far easier to spot in a diff than to debug after a 3am process restart. ## Panics the boundary cannot catch Be honest about the limits when you describe this. A per-goroutine recovery stops panics raised on that goroutine's stack. It does nothing about a goroutine that a library starts internally, and it does nothing about runtime failures that are not panics at all — concurrent map writes, for instance, abort the program without giving deferred functions a chance to run. The boundary is a containment mechanism for bugs in *your* reconcile path, not a general guarantee that the process survives. ## Summary The deferred recovery goes at the top of the function that `go` runs, once per spawn, with the failure routed explicitly back to the loop that cares. The starter's own defers are on the wrong stack and will never fire, and the price of getting it wrong is the entire process rather than one unit of work.