A staged build's final image was given a package manager and a compiler because the service failed to start — what went wrong?

## Read the symptom precisely An artifact that runs in the toolchain stage and dies immediately in the final stage almost never has a logic problem. It has a **resolution** problem: at start-up it looks for something by name, and the final base does not have it. The usual candidates are: - **A dynamically linked system library.** The artifact was built against libraries present in the toolchain base and records the names it needs; the final base carries a different set, or none. - **An interpreter or managed runtime** that must be present because the artifact is not a self-contained executable. - **A certificate store.** The workload starts, then fails the first outbound connection whose server certificate it must verify, because there is no trusted-root file to read. - **Timezone or locale data**, which turns into wrong timestamps or an outright failure in date handling. - **An architecture mismatch**, when the toolchain stage and the final base were not built for the same processor architecture and the artifact cannot be executed at all. All of these are the same underlying fact from the other leaf of this subject: *nothing is inherited between stages*. The artifact crossed; its surroundings did not. ## Why the applied fix is worse than the bug Adding a package manager and a compiler to the final stage is a fix in the sense that the service starts. What it costs: | Property | After the staged build | After the toolchain is reinstalled | |---|---|---| | Image size | tens of megabytes | back to hundreds, and growing | | Tools inside the boundary | none worth stealing | compiler, package manager, shell | | Components in the inventory | the run-time set | every build package, re-declared | | Where the fix is visible | nowhere — the build still passes | nowhere — the build still passes | That last row is the real problem. This regression produces no failure, no warning and no red pipeline. It is discovered months later when someone asks why an image that was 50 MB is now 700 MB, or when an incident review notes that the workload that was compromised had a compiler in it. Anything that installs at **start-up** rather than at build time is worse again: it makes every start depend on a package source being reachable and on whatever version it serves that day. ## What to do instead 1. **Find out what is actually missing**, in the toolchain stage, while the tools to answer that still exist. Resolve the artifact's dynamic dependency list there and write it out as a build output; compare it against what the final base contains. Do this before choosing a remedy — the remedies are very different for one missing library and for a missing interpreter. 2. **Make the artifact self-contained** if the toolchain supports it. A statically linked executable resolves nothing at start-up, which makes the final stage's contents almost irrelevant and is the cleanest outcome for a batch job like this one. 3. **Copy the specific files across.** If a handful of libraries are genuinely needed, copy those files from the toolchain stage alongside the artifact. Copy the whole dependency set, not a single file — a library that itself depends on others fails in exactly the same way one step later. 4. **Copy the run-time data too** — the trusted-root certificate file, the timezone database — or pick a base that ships them. 5. **Choose a different final base**, deliberately. If the workload genuinely needs a substantial library set, a base that carries it is a legitimate answer; that decision has its own trade-offs about size and patch cadence and belongs to the base-choice discussion, not to an emergency edit of the final stage. Option 1 followed by option 2 or 3 keeps the property the staged build was built for. Installing a toolchain keeps nothing. ## Guarding against the regression - **Measure the shipped size** as part of the build and treat a jump as a defect to explain, since nothing else will report it. - **Treat a package manager in a final stage as a review finding**, not a style preference — it is the marker that this regression happened. - **Never install anything at start-up.** A workload that reaches out for a dependency when it starts has made a package source part of its availability story and its supply chain. Diagnosing the failure inside the running container, when the image has no shell to open, is its own subject with its own techniques; the point here is that the *cause* is almost always the stage boundary doing exactly what it was designed to do, and the *cure* must not be to dismantle it.