skip to content

A Go daemon panics during startup inside an init() function and logs nothing. How do you diagnose it?

level: seniorimportance: should knowfreq 38%

answer

  1. the logger was not built yet
  2. look somewhere other than the log pipeline
  3. the process never reached its entry point
  4. flags are still unparsed at that moment
  5. constructors can return an error; this cannot

basics

~20 s

Read the raw stack trace on standard error: initialization runs before main() configured logging, so nothing reached the log pipeline. The stack names the failing package. Then move failable work into a constructor that returns an error.

solid answer

~50 s

The first thing to understand is why there is no log line: initialization runs before `main.main`, so whatever logger, flag parsing and config loading main() performs has not happened. The runtime writes the panic message and the goroutine stack to standard error and exits non-zero, and if your supervisor only captures the application's structured log stream, that output is dropped -- so capture raw stderr first. The stack trace identifies the failing package and the initialization frame inside it. Typical causes are a package-level variable or init() that reads an environment variable, opens a file, dials a dependency, or parses something at import time. Reproduce it by building or testing the smallest package that transitively imports the suspect one. The fix is structural: replace the import-time work with an exported constructor returning `(T, error)` that main() calls after logging and configuration exist.

code

go · 9 lines
go
var endpoint = mustEnv("METRICS_ENDPOINT")

func mustEnv(key string) string {
	v := os.Getenv(key)
	if v == "" {
		panic("metrics: " + key + " not set")
	}
	return v
}

go deeper

for a junior

Remember that initialization finishes before main() starts, so a crash there produces no application log line -- the evidence is the stack trace the runtime writes to standard error before the process exits.

for a middle

Explain the mechanics: init() cannot return an error, panicking is its only failure signal, recover from main is impossible, and the same sequence runs inside test binaries so a small go test reproduces it.

for a senior

Show the triage path end to end: capture raw stderr, read the stack for the failing package, name the likely import-time cause, reproduce with the smallest package that imports it, then propose the structural fix rather than a retry.

for a principal

Frame it as an operability rule: work that can fail must not run before the process can report failure, and you should be able to say what enforcement -- review standard, analyzer, startup checklist -- keeps that true across teams.

### Why the logs are empty Go initializes every package -- variables first, then init() functions, imports before importers -- before it calls `main.main`. Everything a well-built daemon does to become observable happens *inside* main(): parsing flags, reading config, constructing the logger, wiring a handler that ships records off-box. A panic during initialization happens strictly before all of that. There is no logger yet, so there is no log line, and there never will be one. What there is: the runtime prints `panic: <message>` followed by the panicking goroutine's stack trace to **standard error**, and terminates the process with a non-zero exit status. If the process is supervised by something that only forwards the structured log stream, or that starts collecting after a readiness signal that never arrives, that output can vanish entirely -- which is exactly why the incident looks like "it died silently". ### Step one: capture raw stderr Before theorising, get the bytes. Run the binary in the same environment with stderr attached, or configure the supervisor to capture it. The stack trace is the whole diagnosis: it names the package and the function inside it whose initialization panicked, and the frames below it show what that code was doing -- reading the environment, parsing a template, dialling something. ### Step two: recognise the family of causes Almost every startup panic in initialization is one of these: - **Ambient state read at import time.** A package-level variable or init() reads an environment variable and panics when it is empty. This works on every developer laptop and fails in the one deployment where the variable was renamed. - **Flags read before they are parsed.** `flag.Parse()` runs inside main(), so a package that reads a flag's value during initialization sees the default, not what the operator passed. Sometimes that default is invalid and the code panics on it. - **I/O at import time.** Opening a file, reading a certificate, dialling a dependency. Any of these can fail for reasons that have nothing to do with your code, at a moment where you have no way to retry, degrade, or report. - **A `Must`-style helper.** `template.Must`, `regexp.MustCompile` and hand-rolled equivalents are designed to panic, and in a package-level variable they panic at import time. ### Step three: reproduce it small Build or test the smallest package that transitively imports the suspect one. A test binary runs the same initialization sequence as the real binary for the packages it links, so `go test` on that package will reproduce the panic without deploying anything. The mirror-image symptom is worth knowing because it points at the same root cause from the other side: **a test that fails only when its package is tested on its own**. Under a full run, some other package in the same binary had already performed the initialization -- populating a registry, setting a package-level variable -- and the test under investigation was quietly depending on it. Tested in isolation, that package is not linked in, the initialization never happens, and the test fails. When you are triaging a CI build that is red only for one package, this is the first hypothesis to check. ### Why main() cannot rescue you A natural instinct is "wrap it in recover". It does not work from main(): `recover` only has an effect inside a function deferred by the panicking goroutine, and while an init() function is panicking, `main.main` is not on the stack -- it has not been entered. The only place a recover can catch it is a deferred function inside that same init() or something it called, which turns a crash into a package that is silently half-initialized. That is usually worse. Equally, an init() function cannot return an error, and it takes no parameters, so it cannot be told what to do differently. Those two facts together are the whole argument. ### The fix, and how to prove it Move the failable work out. Keep only cheap, deterministic, un-failable setup at import time. Everything else becomes an exported constructor: ```go func New(endpoint string) (*Reporter, error) ``` main() calls it after flags are parsed, config is loaded and the logger exists, so the failure is a logged, contextualised error with an exit code you chose -- and a unit test can exercise both the success and the failure path without setting a process-wide environment variable. A useful transitional measure while you migrate: keep the panic but make its message name the package and the missing input precisely, so the raw stderr line is self-explanatory at 3am. ### What a strong answer sounds like Name the ordering fact (initialization precedes main, so the logger does not exist yet), say where the evidence actually is (raw stderr, non-zero exit), name the likely causes, describe reproducing it with a small test binary, explain why recover in main() is not an option, and finish with the structural fix.

  • Why can't main() recover from a panic raised in an imported package's init()?
    Because recover only takes effect in a function deferred by the panicking goroutine while that panic unwinds its stack, and main.main has not been entered yet -- initialization completes first. The only place that can catch it is a deferred function inside that init() itself, which leaves the package half-initialized.
  • A test passes in the full suite but fails when its package is tested alone. How does initialization explain that?
    The test was depending on work another package performed during initialization -- populating a registry, assigning a package-level variable. That package is linked into the full run's binaries but not into this package's isolated test binary, so the setup never happens. The fix is to make the test create what it needs explicitly.
  • Why does a package that reads a command-line flag inside init() always see the default?
    Because `flag.Parse()` is called from main(), which runs after all initialization. During init() the flag variables still hold the defaults registered with them. Read flag values inside main, or after parsing, and pass them down as parameters.
  • What would you change so the next startup failure is diagnosable in one look?
    Move failable setup into constructors that return errors and are called from main after logging and configuration exist, so failures become logged, contextualised errors with a chosen exit code. Where a panic must remain, make its message name the package and the exact missing input.

saying these in an interview costs you the question

  • Suggests wrapping main() in recover to catch an init panic
  • Looks only in the structured logs and concludes there is no output
  • Thinks init() can return an error to signal failure
  • Assumes flags are already parsed during initialization
  • Blames the supervisor without capturing raw stderr
  • Adds a retry loop inside init() instead of moving the work