skip to content

The host trusts the TLS-intercepting proxy's CA and `docker pull` works, so why does the JVM inside the container still fail?

level: seniorimportance: nice to knowfreq 24%

answer

  1. Count the trust stores on the path
  2. The daemon's trust never enters the image
  3. The OS bundle and the JVM keep separate lists
  4. update-ca-certificates plus keytool, at build time

basics

~20 s

Trust is per trust store, and the container has its own. The host bundle dockerd verified against is not inside the image, and a JVM reads its own cacerts keystore rather than the operating system's, so the CA must be added in both places in the image.

solid answer

~50 s

An intercepting proxy terminates TLS and re-issues every certificate from a corporate CA, so every client on the path must trust that CA separately. Installing it on the host fixed **dockerd**, which is why `docker pull` succeeds - but that bundle lives on the host filesystem and is never copied into an image. Inside the container there are two more stores: the image's own OS bundle, used by `curl`, `apt` and anything built on OpenSSL, and the JDK's `cacerts` keystore, which a JVM reads instead of `/etc/ssl/certs`. The tell is `javax.net.ssl.SSLHandshakeException: PKIX path building failed ... unable to find valid certification path to requested target`. The durable fix is to bake the CA into the base image - copy the certificate in, run `update-ca-certificates`, and import it into `cacerts` with `keytool` - rather than disabling verification or mounting host paths.

code

dockerfile · 8 lines
dockerfile
FROM eclipse-temurin:21-jre
COPY corp-root-ca.crt /usr/local/share/ca-certificates/corp-root-ca.crt
RUN update-ca-certificates && \
    keytool -importcert -noprompt -trustcacerts \
      -alias corp-root -file /usr/local/share/ca-certificates/corp-root-ca.crt \
      -cacerts -storepass changeit
COPY doc-indexer.jar /app/doc-indexer.jar
ENTRYPOINT ["java", "-jar", "/app/doc-indexer.jar"]

go deeper

for a junior

Take away one rule: a container does not inherit the host's trusted certificates. If an application inside a container has to trust a certificate authority, that authority has to be present inside the image.

for a middle

Be able to name the stores and their owners - the host bundle for the daemon, the image's OS bundle for OpenSSL-based tools, the JDK cacerts keystore for the JVM - and show the build-time commands that populate the last two.

for a senior

Show the diagnosis path: recognise PKIX path building failed as a trust failure rather than a network one, prove interception by checking the issuer of the served certificate, and explain why a working docker pull is no evidence about the application.

for a principal

Own where the certificate lives across the estate. One shared base image carrying the CA beats forty Dockerfiles, and the rotation plan - overlapping validity, rebuild fan-out, what breaks in air-gapped or off-network builds - is the part worth designing before the certificate expires.

## What interception does to the handshake A TLS-intercepting egress proxy does not forward your handshake. It terminates the connection itself, opens its own connection to the real destination, and presents *you* with a certificate it generated on the spot, signed by a corporate certificate authority. Every client behind that proxy therefore sees a chain rooted in an authority that no public trust store contains, and each client will reject it until that authority is added to whatever list of trusted signers that particular client consults. The word "trusted" has no global meaning on a machine. It is always relative to a store, and a container image drags at least one extra store into the picture. ## The stores on the path, and who reads which 1. **The host's system bundle**, plus `/etc/docker/certs.d/<host>/` for registry-specific CAs. This is what `dockerd` uses. Fixing it makes `docker pull` and `docker push` work, and that is the whole extent of what it fixes. 2. **The image's own OS bundle** - `/etc/ssl/certs/ca-certificates.crt` on Debian-derived images, the equivalent under `/etc/pki` on RHEL-derived ones. This is what `curl`, `wget`, `apt`, and anything linked against OpenSSL inside the container consults. It comes from the base image and knows nothing about the host it happens to be running on. 3. **The JDK's `cacerts` keystore**, shipped inside the JDK or JRE in the image. A JVM does not read `/etc/ssl/certs` at all; it reads this keystore, or whatever `-Djavax.net.ssl.trustStore` points at. So a Scala service on a JDK base image can sit in a container whose OS bundle already trusts the corporate CA and still refuse every outbound HTTPS call. Other runtimes have their own conventions - Python's `certifi`, Node's bundled roots and `NODE_EXTRA_CA_CERTS` - so "the container trusts it" is never a single fact. ## The symptom, precisely A Scala document-indexing job, running with a 128 MiB memory limit and pulling documents from an internal HTTPS endpoint, fails on its first request with: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target That is not a timeout, a DNS failure or an out-of-memory kill. "Path building failed" says the JVM received a chain and could not connect it to anything in its keystore. Confirm the interception by looking at who signed what the endpoint serves from inside the affected network: openssl s_client -connect docs.internal.example.com:443 </dev/null 2>/dev/null | openssl x509 -noout -issuer If the issuer is the corporate CA rather than a public one, you are behind the interceptor and the missing trust is your problem. ## Fixing it in the image The durable repair is at build time, in a base image the whole organisation shares, so that no application team ever has to think about it: FROM eclipse-temurin:21-jre COPY corp-root-ca.crt /usr/local/share/ca-certificates/corp-root-ca.crt RUN update-ca-certificates && keytool -importcert -noprompt -trustcacerts -alias corp-root -file /usr/local/share/ca-certificates/corp-root-ca.crt -cacerts -storepass changeit `update-ca-certificates` rebuilds the OS bundle for the OpenSSL-based tools; `keytool ... -cacerts` writes into the JDK's own keystore for the JVM. On Alpine-based images you need `apk add --no-cache ca-certificates` first. If you cannot rebuild the image, you can mount a prepared keystore and point the JVM at it with `-Djavax.net.ssl.trustStore`, but that couples the container to files on a particular host and quietly breaks the moment it is scheduled somewhere else. Note that build steps hit exactly the same wall. Proxy build arguments only tell a client *where* to connect; if the proxy is also intercepting, the very first `RUN` that fetches a dependency fails on certificate verification unless the CA is installed before it - which is another argument for putting the certificate in the base image rather than in each application's Dockerfile. ## What not to do Disabling verification in the application - a trust manager that accepts everything, `curl -k`, `-Dcom.sun.net.ssl.checkRevocation=false` cargo-culted from a forum - makes the symptom vanish and removes the only protection the client had against being fooled by something that is *not* the corporate proxy. It also hides genuine certificate expiry until it becomes an incident somewhere else. Equally, resist per-developer keystores checked into a repository. The certificate has a lifetime, and a CA rotation with the trust scattered across forty Dockerfiles is a very different exercise from a rotation with it in one base image. ## The reasoning to show in an interview The valuable answer is not the `keytool` line, it is the model: name the trust stores on the path, say which process reads each one, and observe that fixing a store only fixes the clients that read it. That model also explains the neighbouring puzzles - why the image works on a laptop off the corporate network, why the same container fails only in one data centre, and why `docker pull` succeeding tells you nothing at all about the application.

  • The same image works fine on a developer laptop that is off the corporate network. What does that tell you?
    That the image is not broken - the certificate it is offered differs by network. Off the corporate path the endpoint serves its real, publicly-rooted certificate, which the image already trusts; on the corporate path an interceptor re-issues it from a CA the image has never seen. Comparing the issuer of the served certificate from both networks confirms it in one command.
  • Would mounting the host's /etc/ssl/certs into the container fix the JVM?
    No. It would fix curl, apt and other OpenSSL-based clients, because those read the OS bundle, but a JVM reads its own `cacerts` keystore and ignores that directory entirely. You would still need an updated keystore or `-Djavax.net.ssl.trustStore`. It also ties the container to one host's filesystem layout, which is why baking the CA into the image is the better answer.
  • The build itself fails with certificate errors on a RUN step even though the proxy build args are set. Why?
    Proxy variables only tell a client where to send the connection; they say nothing about trusting what comes back. If the proxy is intercepting, the first RUN that fetches over HTTPS fails verification until the CA is installed inside that build step. Install the certificate before any network-using RUN, or start from a base image that already carries it.

The host, the image and the JVM each keep their own address book of signers they will believe. Writing a name into one book does not add it to the others.

saying these in an interview costs you the question

  • Disables certificate verification in the application
  • Assumes the host trust store is visible inside containers
  • Believes a JVM reads /etc/ssl/certs
  • Puts the CA in /etc/docker/certs.d and expects the app to see it
  • Reads PKIX path building failed as a network timeout
  • Ships per-developer keystores instead of a shared base image

context