A service on a near-empty base fails certificate verification on every outbound call — why?

## What verification actually needs When a process opens an outbound connection over an encrypted transport, the server presents a **certificate chain**: its own certificate, plus the intermediates that link it back towards a **root certificate**. The client's job is to decide whether that chain terminates in something it already trusts. It does that by comparing against a **trust store** — a set of root certificates present on the local filesystem, in a file or directory the client library knows to read. Two consequences follow, and both are the whole of this failure: - **Trust is local to the image.** Nothing arrives over the wire that establishes trust; the wire is what is being doubted. If the roots are not in the image, there is no trust to be had. - **The store is data, not code.** It is a file the base image happened to ship. Strip the base and it goes with everything else. ## Why the full distribution hid this from you On a full distribution base, the trust store is a maintained package. Someone else curates which roots belong in it, withdraws the ones that should no longer be trusted, adds new ones, and ships the updates through the same stream as every other package. Your process reads it, and no one on the team has ever had to know it exists. A minimal base may or may not carry one — some variants deliberately include a bundle because so many workloads need it, and many do not. An empty base carries nothing by construction. The failure therefore appears at exactly the moment a team congratulates itself on shrinking the image. ## The symptom, and how it misleads | What you observe | What it points at | |---|---| | Every outbound endpoint fails identically | No trust store at all in the image | | Only one internal endpoint fails | A private issuer missing from an otherwise present store | | One public endpoint fails, others fine | That server's chain or expiry, not your image | | Worked on the build host, fails in the image | The build host is a full userland and supplied the store | The first and last rows together are the signature. The most common wrong turn is to treat a total failure as a server-side problem, because the error text usually names the remote host — and the remote host is the only variable a developer can see. ## Supplying the store yourself 1. **Put a maintained bundle of root certificates into the image**, at the path the client library reads. The path is not universal; different libraries look in different places, and some accept an environment value pointing at a file. 2. **Add your internal issuer separately** if you call internally issued endpoints. That is an *addition* to the public roots, not a replacement for them — replacing them breaks every public call. 3. **Refresh it on every rebuild.** A trust store ages: roots are withdrawn and added. A bundle copied in once and never touched again will eventually trust something it should not, or fail to trust something it should. 4. **Verify in the pipeline.** Start the final image and make one real outbound call before publishing. This failure is cheap to catch and expensive to discover on several hundred hosts. 5. **Never disable verification to make the error go away.** That converts a missing-file problem into an unauthenticated connection, permanently, in an image that will be copied. ## What else in that family disappears The trust store is one member of a class: **data files the userland supplied that the process reads at run time.** The same strip usually takes: - **Timezone and locale data**, so converting an instant into a named zone either falls back to a single zone or fails outright. - **Name-resolution configuration**, which some processes read from a file rather than asking the platform. - **User and group files**, so a numeric identity has no name behind it. - **Message catalogues and character-set tables**, which surface as mangled output rather than an error. Each one behaves the same way: absent, silent at build time, and loud the first time the code path runs. That is why the honest place to catch all of them is a start-and-exercise step against the final image, rather than a review of what was copied in. ## The reasoning to show in an interview Name the direction: the client checks a presented chain against roots it holds locally, so the missing piece is inside the image, not on the network and not on the server. Then distinguish the two shapes — an empty store fails everything, a missing private issuer fails one thing — and say how you would tell them apart from the failure pattern alone. That reasoning is portable across every platform, because none of it depends on which one is running the container.