A Go binary built on a glibc host dies on a musl host with "no such file or directory" — why?
answer
- the message is about a different file
- an ELF binary names a helper first
- Go only links dynamically when cgo was involved
- run file on the shipped artifact
- the loader path differs between C libraries
basics
~20 sThe binary was linked with cgo enabled, so it is dynamically linked and names a glibc dynamic loader such as /lib64/ld-linux-x86-64.so.2. That path does not exist on a musl host, so exec fails and reports the missing interpreter.
solid answer
~50 sThe message is about a file the binary *names*, not the binary itself. A cgo-enabled build links externally and produces an ELF executable with an interpreter entry — the glibc dynamic loader, usually `/lib64/ld-linux-x86-64.so.2`. On a musl-based host that path does not exist, so `execve` returns ENOENT and the shell prints "no such file or directory" about a file that is plainly there. Confirm it in seconds with `file ./app`, which prints the interpreter path, or `ldd ./app`, which lists the shared libraries. The fix is at build time: rebuild with `CGO_ENABLED=0` so the Go linker links internally and emits a static executable with no interpreter. If cgo is genuinely required, the alternative is to build against the same C library the target host uses so the loader and libraries match. Copying loaders onto the target is not a fix.
code
text · 10 lines$ ./app
./app: no such file or directory
$ file ./app
./app: ELF 64-bit LSB executable, x86-64, ... dynamically linked,
interpreter /lib64/ld-linux-x86-64.so.2, ...
$ CGO_ENABLED=0 go build -o app .
$ file ./app
./app: ELF 64-bit LSB executable, x86-64, ... statically linked, ...go deeper
Remember that this error can be about a file the binary points at rather than the binary itself, and that running file on the artifact is the first thing to try.
Explain the mechanism: cgo forces external linking, the resulting ELF names a dynamic loader, and a host with a different C library has no such path. Know that CGO_ENABLED=0 removes the interpreter entirely.
Diagnose from the shipped artifact rather than a fresh local build, identify which dependency pulled cgo in, and weigh rebuilding without cgo against matching the build environment to the target. Say what you verify before the rebuild ships.
Turn the incident into a rule: an artifact check in the release pipeline, a stated default, and a documented exception path. The interesting question is who pays for the exception when one team's dependency needs cgo.
## Read the error precisely "No such file or directory" from an exec of a file you can `ls` is one of the most confusing messages on Linux, and it has one common cause: the kernel could not find the *ELF interpreter* the binary asks for. A dynamically linked ELF executable carries an `INTERP` entry naming the dynamic loader that must run first to map its shared libraries. If that path is absent, `execve` fails with `ENOENT`, and the shell has no way to tell you which of the two files was missing. On 64-bit x86 Linux, a glibc-linked binary names `/lib64/ld-linux-x86-64.so.2`. A musl-based system has no such file — its loader lives elsewhere and has a different name. So a binary built on a glibc builder simply cannot start there. ## Why a Go binary has an interpreter at all Go is famous for static binaries, which is why this surprises people. The rule is: - **cgo disabled** — the Go linker links internally. The output is a statically linked executable, no `INTERP`, no shared libraries. - **cgo enabled** — the Go linker uses *external* linking, handing objects to the system C toolchain. The output is dynamically linked against the builder's C library. So the question is never "is Go static?" but "did anything in this build use cgo?" A single dependency that binds a C library is enough. So is building on a machine where cgo defaults on and some package quietly enables it. ## The diagnosis, in order 1. `file ./app` on the artifact. Either `statically linked`, or `dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2`. 2. `ldd ./app`. Either it reports the file is not a dynamic executable, or it lists the shared libraries — which is your list of things the target host must provide, at compatible versions. 3. If the answer is "dynamic", the question moves to *why*: which package pulled cgo in. Building with `CGO_ENABLED=0` and reading the compile error tells you immediately, because the package that needs C will fail to build. Do this on the **artifact you shipped**, not on a fresh local build. The whole class of bug is that the build environment and the run environment differ, so evidence from the build environment is exactly the evidence that will mislead you. ## The fixes, best first **Rebuild with `CGO_ENABLED=0`.** One static file, no loader, no libc coupling. This is the answer for the large majority of services, and it is why teams that ship a binary into an image containing nothing else make it the default in CI rather than a thing developers remember. **If cgo is genuinely required**, make the build environment match the run environment: build with the same C library the target uses, so the interpreter path and the library versions line up. That couples your builder to your target, which is a real cost — it is why the first option is preferred. **A third, narrower option** is to keep cgo but link the C library statically, so no loader is needed. That works, but with glibc it comes with its own trap around functions that load modules at run time, so it is not the casual fix it looks like. ## What is not a fix - Copying the build host's loader or libraries onto the target. You now maintain a private, undocumented runtime and will discover the version skew later, under load, rather than at startup. - Stripping the binary. Stripping removes symbols; it does not remove the interpreter entry or the library dependencies. - Rebuilding on a different kernel. Linux's system-call interface is stable across versions; the kernel is not what is missing. ## Why it only shows up in production The failure is environment-shaped, so it hides beautifully. Developers run the binary on the machine that built it. Tests often run in the builder's own environment. The first host with a different C library is the deployment target, and the first person to see the error is on call, staring at a file that exists and a message saying it does not. Making the artifact check — `file` on the built binary, fail the pipeline if it is dynamic — part of the release is what turns this from an incident into a red build. ## The sentence to leave the interviewer with The binary is not missing; its dynamic loader is. Go only produces a dynamically linked binary when cgo was in the build, so the question is which dependency pulled C in, and the default fix is `CGO_ENABLED=0`.
- How would you find which dependency pulled cgo into the build?Build the same package with `CGO_ENABLED=0` and read the failure: the package that needs C is the one that stops compiling, because its `import "C"` files are excluded by build constraints. If it builds cleanly, cgo was enabled but nothing required it, and you can simply set the variable in CI.
- The service now starts on the target host, but one internal hostname stops resolving after the rebuild. What happened?Turning cgo off also switched the `net` package to its pure-Go resolver, which understands the ordinary configuration sources but cannot load the C library's pluggable name-service modules. A name that only exists behind one of those is now invisible. That is a behaviour change to verify before release, not a side effect to discover on call.
- Would a pipeline check have caught this, and what would it assert?Yes. Run `file` (or `ldd`) on the built artifact and fail the release if it reports a dynamic executable, unless a recorded exception applies. Asserting that `CGO_ENABLED=0` was exported is weaker, because a new dependency can force external linking without anyone editing the build script.
The binary is a letter addressed to a helper who lives at a fixed street address. On the new host nobody lives there, so the delivery bounces — and the bounce notice names your letter, not the empty address.
saying these in an interview costs you the question
- Concludes the binary was not copied or lacks execute permission
- Says Go binaries are always static, so this cannot happen
- Proposes copying the builder's libc onto the target host
- Blames a kernel version mismatch
- Thinks stripping the binary removes the library dependency