A debugger attached to a container hits the wrong lines or no breakpoints at all - what causes that?

### A worked case A team ships a Rust document-indexing service. The Dockerfile compiles it in a builder stage under `/usr/src/app`, and the final stage copies just the binary onto a slim base. Their CI runner keeps a 92 GB build-cache directory so the builder stage is fast. An engineer starts the debug image, attaches, sets a breakpoint in `src/index/merge.rs`, and the debugger reports the breakpoint as unverified. Nothing stops. The attach itself worked - the debugger is connected and can list threads. That combination - a healthy connection but no breakpoint binding - is almost always about the *artefact*, not the transport, and it splits into three causes. ### Cause 1: the running code is not the code you are looking at Breakpoints are set by file and line. If the compiled unit in the image came from a different revision of that file, every line number after the first edit is off by the number of lines you added or removed, so breakpoints either fail to resolve or resolve to the wrong statement - you stop, and the highlighted line is nonsense. How the drift happens in container workflows specifically: - The image was built minutes ago but from a tree that has since changed, and the process is still running the old container. - A cached builder stage was reused because the layer's inputs looked unchanged - a `COPY` whose context excluded the changed file, for example, or a mount-based cache holding a stale compiled object. - A bind mount shadows part of the source at run time, so the debugger sees your files but the process runs code compiled from the image's copy. Confirm before theorising: check the container's image ID against the image you just built, and check the build actually recompiled rather than replaying cache. ### Cause 2: compile-time paths do not exist on your machine Debug information for compiled languages records the path each source file had *at compile time*. Compiled inside a builder stage, that is a container path such as `/usr/src/app/src/index/merge.rs`. Your checkout is `/home/dev/indexer/src/index/merge.rs`. The debugger has a file open and a compiled unit loaded and no way to know they are the same file, so the breakpoint never binds. The fix is a mapping, and each debugger spells it differently: delve takes `substitute-path` rules (from the container path prefix to the local one), debugpy takes `pathMappings` entries of `localRoot`/`remoteRoot`, and IDEs expose an equivalent remote-source mapping in the run configuration. Some toolchains can also rewrite the recorded prefix at compile time so the paths come out relative or already match; that is the more durable fix when your build owns the compiler flags, because it makes every debug session work without per-developer configuration. A useful property of the JVM here: class files record the source *file name*, not an absolute path, and the debugger resolves sources by fully-qualified class name against a source path you configure. So JVM debugging rarely hits this exact mismatch - which is why an engineer whose experience is all JVM is often surprised by it the first time they debug a Go or Rust container. ### Cause 3: the final stage has no debug information Multi-stage builds exist to leave things behind, and debug symbols are exactly the kind of thing they leave. A release binary built with optimisation and stripped of symbols carries no line table at all; the debugger can attach and show you machine state, but there is nothing to map to a source line. Optimisation alone is enough to cause weirdness even with symbols present - inlined functions produce breakpoints that never hit, and variables read as optimised out. The practical arrangement is a debug variant of the image: a build target that compiles without stripping (and usually with optimisation reduced), keeps the source tree in the image at the same path the debug info records, and includes whatever debug server the language needs. It is a bigger image and it is not what you ship - that is the point. ### The order to check 1. Does the container's image ID match the image you just built? If not, stop - it is staleness. 2. Did the build actually recompile, or replay a cached stage? 3. Does the final stage contain debug information at all? 4. What path prefix does the debug info record, and what mapping have you configured? Working in that order takes minutes. Starting at step 4 - which is where people start, because path mapping is the interesting answer - burns an afternoon on a configuration that was correct all along while the container quietly runs last week's binary.