An nginx server loads its certificate with `ssl_certificate /etc/letsencrypt/live/example.com/cert.pem;`. Browsers show the site as secure, but curl and a Java client fail certificate verification. What is wrong, and how do you fix it?
answer
- nginx sends the file, nothing more
- cert.pem versus fullchain.pem
- browsers fetch it, curl does not
- leaf first, then the intermediate
- count certificates with s_client -showcerts
basics
~20 scert.pem holds only the leaf certificate, so nginx sends no intermediate. Browsers hide the gap by fetching or reusing a cached intermediate; strict clients cannot. Point ssl_certificate at fullchain.pem, which is the leaf followed by the intermediate.
solid answer
~40 s`cert.pem` is the leaf certificate on its own. nginx sends exactly the certificates in the file named by `ssl_certificate` and builds nothing — so the handshake presents a leaf whose issuer the client has never seen. Browsers usually still succeed, because they cache intermediates seen on other sites or fetch the missing one through the certificate's AIA extension. curl/OpenSSL, Java and Go do not chase AIA, so they fail with an unknown-issuer error. The fix is to point `ssl_certificate` at `fullchain.pem` — leaf first, then intermediate — keep `ssl_certificate_key` on `privkey.pem`, run `nginx -t` and reload. Confirm with `openssl s_client -connect example.com:443 -servername example.com -showcerts`: you should see two certificates and a clean verify return code, not "unable to verify the first certificate".
code
bash · 2 linesopenssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null 2>/dev/null | grep -c 'BEGIN CERTIFICATE'
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | grep 'Verify return code'go deeper
Know which certbot file each directive wants: ssl_certificate takes fullchain.pem and ssl_certificate_key takes privkey.pem. Say plainly that nginx serves the file's contents and nothing else.
Explain that the server must present leaf plus intermediates because clients only trust roots, and describe why browsers mask the gap through AIA fetching and cached intermediates while curl and Java do not.
Show the diagnosis: openssl s_client -showcerts -servername, count certificates, read the verify return code, and reason about which client populations break first. Chain problems surface as partner integration failures, not browser complaints.
Own the prevention story: certificate issuance and deployment automated so the chain file is never hand-assembled, plus external monitoring that validates the served chain from a clean trust store rather than from a developer's browser.
## What nginx actually sends During a TLS handshake nginx transmits, in order, the certificates it read from the file named by `ssl_certificate`. It does not consult the system trust store, does not look up the issuer, and does not construct a path for you. Whatever chain the client sees is exactly what you concatenated into that one file. This is the single most important fact behind the whole class of "works in my browser" TLS bugs. Certbot writes several PEM files into `/etc/letsencrypt/live/<domain>/`: - `privkey.pem` — the private key; this is what `ssl_certificate_key` wants. - `cert.pem` — the leaf (server) certificate **only**. - `chain.pem` — the intermediate certificate(s) only. - `fullchain.pem` — `cert.pem` followed by `chain.pem`. `fullchain.pem` is the file `ssl_certificate` wants. Choosing `cert.pem` is an easy mistake because it is the file whose name sounds like "the certificate". ```nginx server { listen 443 ssl; server_name example.com; ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; } ``` ## Why browsers forgive it and tooling does not A client validating a certificate needs an unbroken path from the leaf up to a certificate it already trusts. Public CAs sign leaves with an intermediate, not with the root, so the intermediate has to come from somewhere. When the server omits it, clients differ sharply: - Major browsers may fetch the missing issuer over HTTP using the URL in the leaf's Authority Information Access extension, and they commonly have the intermediate cached from another site they visited. So the page loads. - curl (OpenSSL), Java, Go and most language HTTP clients do no AIA fetching. They fail immediately: OpenSSL reports `unable to get local issuer certificate`, Java reports `PKIX path building failed: unable to find valid certification path to requested target`. That asymmetry is why the bug reaches production: a human tests in a browser and sees a padlock, while a freshly built container, a mobile SDK or a partner's server-to-server integration fails. It is also why the failure often appears "suddenly" — the day a new client, a new base image or a new CA intermediate enters the picture. ## Order matters, and the root does not belong there The file must start with the leaf, then each issuer moving up the chain. If the first certificate in the file is not the one matching the private key, nginx refuses to start with a key-values-mismatch error from `SSL_CTX_use_PrivateKey_file()`. Do **not** append the root CA: every client that would accept the chain already has the root in its own trust store, so sending it only adds bytes to every handshake and is ignored. The root has one legitimate home in nginx config — `ssl_trusted_certificate`, used for verifying OCSP staples and client certificates, not for the served chain. ## Diagnosing it in thirty seconds ```bash openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null \ | grep -c 'BEGIN CERTIFICATE' ``` One certificate for a publicly issued site means the chain is incomplete. Also read the `Verify return code:` line — `21 (unable to verify the first certificate)` is the signature of this bug, whereas `0 (ok)` means the chain validated against your local trust store. `-servername` matters: without it, OpenSSL sends no SNI and a multi-site nginx will answer from its default server, so you may be inspecting the wrong certificate entirely. ## Neighbouring traps in the same directive - **Dual certificates.** nginx accepts multiple `ssl_certificate`/`ssl_certificate_key` pairs in one server, typically an ECDSA and an RSA chain; OpenSSL picks per client capability. Each pair needs its own complete chain. - **Stapling verification.** With `ssl_stapling_verify on;`, `ssl_trusted_certificate` must contain the issuer chain **and** the root, which is a different file from the one you serve. - **Reload required.** Fixing the path changes nothing until nginx reloads and re-reads the file.
- Why does the same site validate in Chrome on your laptop but fail inside a freshly built container?The laptop browser has the intermediate cached from other sites and will fetch it via AIA if not; the container has an empty intermediate cache and its client (curl, Java, Go) does no AIA fetching. Nothing about the server differs — only what the client can supply for itself.
- Should the root CA certificate also go into fullchain.pem?No. Any client that would accept the chain already trusts the root from its own store, so sending it wastes handshake bytes on every connection and is ignored during path building. The root belongs in `ssl_trusted_certificate` when you need it for OCSP stapling verification, not in the served chain.
- How do you serve both an ECDSA and an RSA certificate from one nginx server block?Declare two `ssl_certificate` directives with their matching `ssl_certificate_key` directives. OpenSSL selects per handshake based on the client's advertised signature algorithms and cipher suites, so modern clients get ECDSA while legacy ones still negotiate RSA. Each certificate needs its own complete chain in its own file.
Handing over an ID card issued by a branch office nobody has heard of. A well-connected checker phones around and confirms the branch is legitimate; a checker following the rules on paper just says no.
saying these in an interview costs you the question
- nginx builds the chain automatically from the system trust store
- An incomplete chain is only a browser warning, never an outage
- Put the root CA first in the file
- cert.pem is enough because the CA is publicly known
- Installing the intermediate in the server's OS trust store fixes it