skip to content

A Go binary runs in CI but exits with no output on the deploy host — how do you prove libc is the cause?

level: seniorimportance: should knowfreq 36%

answer

  1. the process may never have started
  2. ask the file, not the pipeline
  3. the binary remembers how it was built
  4. run the check on the failing host
  5. build settings are recorded inside the binary

basics

~20 s

Interrogate the artefact, not the build script. file and ldd show whether the binary is dynamically linked and against which loader, and go version -m prints the build settings baked into it, including whether cgo was enabled.

solid answer

~40 s

Start from the exact error, because the classic one is misleading: a dynamically linked binary whose loader is absent fails exec with `no such file or directory` even though the file is right there. Then read the artefact. `file ./svc` says `statically linked` or names an interpreter; `ldd ./svc` lists the shared libraries or reports that it is not a dynamic executable; `go version -m ./svc` prints the recorded build settings, so you can see `CGO_ENABLED=1` and the target `GOOS`/`GOARCH` that actually produced this file rather than the ones you believe CI used. If cgo did sneak in, `go list -deps` over the source shows `runtime/cgo` in the graph and tells you which package dragged it there. Only then decide: rebuild without cgo, or make the runtime host carry a compatible C library.

code

text · 12 lines
text
$ ldd ./peer-sync
	linux-vdso.so.1
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
	/lib64/ld-linux-x86-64.so.2

$ go version -m ./peer-sync
./peer-sync: go1.27.0
	path	example.com/peer-sync
	build	-buildmode=exe
	build	CGO_ENABLED=1
	build	GOOS=linux
	build	GOARCH=amd64

go deeper

for a junior

Remember that a Go binary records how it was built, and that ldd and file tell you whether it needs a shared C library. Reach for those before guessing at the source.

for a middle

Explain why no log line appears: a dynamically linked binary is loaded by an interpreter named inside it, and if that is missing, exec fails before any Go code runs.

for a senior

Show a triage order that starts from the exact error message, reads the artefact on the failing host, and only then chooses between rebuilding without cgo and making the host carry a compatible C library.

for a principal

Talk about what you leave behind: a release check that asserts the artefact's recorded build settings against policy, so the next occurrence is a failed build rather than a night-time incident.

## Why this is an artefact question, not a source question The failure everyone reaches for first — "the code must be crashing at startup" — is usually wrong here. If the process never printed a line, and your program logs before it does anything else, the process may never have started. A dynamically linked binary is loaded by an interpreter named inside the file itself; if that interpreter is missing, execution fails **before** any Go code runs, so there is no panic, no stack trace, and no log. That is also why the source tree cannot answer the question. What you need is the provenance of the specific file that is failing, and Go binaries carry that provenance inside them. ## The triage sequence **1. Read the exact error.** `no such file or directory` against a file that exists is the fingerprint of a missing loader. `exec format error` is a wrong architecture. `permission denied` is a mount option or a missing execute bit. These are three different investigations and the message picks one. **2. `file ./svc`.** It reports either `statically linked` or `dynamically linked, interpreter <path>`. If it names an interpreter, check whether that exact path exists on the host — that single comparison often closes the incident. **3. `ldd ./svc`.** For a pure-Go binary it says the file is not a dynamic executable. Otherwise it lists the shared objects, the C library among them, and marks any it cannot find. Run it **on the failing host**, not on your laptop: the whole point is what that host can supply. **4. `go version -m ./svc`.** This is the step people skip and it is the decisive one. Go records build metadata in the binary: the toolchain version, the module path, and the build settings, including `CGO_ENABLED`, `GOOS`, `GOARCH` and any tags. It tells you how the file in front of you was *actually* built, which routinely contradicts the pipeline definition — a cached layer, a locally built binary someone copied in, or a job that ran on a different runner. **5. `go list -deps` over the source.** If cgo was enabled and you want to know *why*, list the dependency graph and look for `runtime/cgo`. On a plain service the usual culprits are the C-backed name and account lookups pulled in through `net` or `os/user` rather than any C code your team wrote. ## Deciding what to do about it There are only two honest resolutions, and they belong to different owners. - **Rebuild without cgo.** The artefact stops depending on any host C library and runs anywhere of the right architecture. The price is behavioural: the pure-Go name and account lookups do not consult the host's name-service switch or its loadable modules, so a lookup that used to work through a directory service or a local name-service module will now fail. That has to be checked against the environment before it ships, ideally on a canary host that actually uses those sources. - **Keep cgo and fix the runtime environment.** The host must carry a compatible C library, and the *build* environment must not be newer than the oldest runtime you support, because symbol versioning is backward compatible only. This makes the runtime environment part of your release contract, which is fine as long as somebody has agreed to own it. ## The trap in "just link it statically" A tempting middle path is to keep the C-backed lookups but link the C library statically. With glibc this does not deliver what people expect: its name-service switch loads modules as shared objects at run time, so a statically linked binary still needs matching shared libraries present in order to perform those lookups, and the linker usually warns about exactly that. You end up with the portability problem you were trying to remove, now harder to see. ## What to leave behind afterwards A good outcome from this incident is not just a fixed binary. It is a check that fails the build when the artefact's recorded settings do not match the policy — `go version -m` output is machine-readable enough to assert on — plus a note in the runbook that `no such file or directory` on an existing binary means a missing loader. Both of those turn a 3am puzzle into a two-minute confirmation the next time.

  • ldd reports that the binary is not a dynamic executable. Where do you look next?
    Somewhere else entirely — a static binary has no run-time C library dependency, so the startup failure is not a linking problem. Check the architecture with `file`, the execute bit and mount options, and then whether the process starts and exits on its own: a missing config file, an unreadable path or an immediate `os.Exit` all look like "no output" from outside.
  • go version -m says CGO_ENABLED=1 but nobody on the team wrote any C. How do you find what pulled it in?
    Run `go list -deps` on the command's package and look for `runtime/cgo` in the graph. On a typical service it arrives through the C-backed hostname or account lookups reachable from `net` and `os/user`, not from a package your team owns. That also tells you which behaviour you are about to change if you rebuild without it.
  • The rebuilt cgo-free binary starts fine, but peer hostname lookups now fail. What changed?
    The lookup implementation. Go's own resolver reads the resolver configuration and hosts file and queries DNS directly; it does not consult the host's name-service switch or its loadable modules. Names that only those modules answer are gone. Confirm on a host that uses them, and either ship the mapping into the runtime filesystem or query the real source directly.
  • Why is running ldd on your laptop a weak check?
    Because the question is what the *failing* host can provide. Your machine almost certainly has the C library the binary was built against, so the output looks healthy and proves nothing. Run it where the exec fails, or compare the interpreter path from `file` against that host's filesystem.

saying these in an interview costs you the question

  • Assumes a crash and hunts for a panic that was never printed
  • Trusts the pipeline definition over the binary's recorded settings
  • Runs ldd on the build machine and calls it clean
  • Suggests statically linking glibc as an equivalent fix
  • Rebuilds without cgo without checking what name lookups break