After a base-image change, a Go log shipper's tls.Dial fails with 'x509: certificate signed by unknown authority'. What is the real fix, and how do you keep the skip flag out of the repo?

## Read the error before changing anything Two handshake failures dominate, they look alike on a dashboard, and they have opposite fixes. - `x509: certificate signed by unknown authority` — the client could not build a chain from the presented certificate to anything in its trust set. The issuer is the problem. - `x509: certificate is valid for collector.internal, not 10.0.4.17` — the chain verified fine, but the name you demanded is not in the certificate. The expected name is the problem. A log shipper that worked yesterday and fails today with the first message, right after a base-image change, is almost never a certificate problem at all. It is a client that lost its roots. ## Why an image change breaks trust With `tls.Config.RootCAs` left nil, Go verifies against the platform trust store. On Unix-like systems `crypto/x509` reads the CA bundle from the conventional filesystem locations, and honours the `SSL_CERT_FILE` and `SSL_CERT_DIR` environment variables as overrides. A `scratch` or otherwise minimal image contains none of those files, so the effective root set is empty and *every* outbound handshake fails with unknown authority — a fact that surprises people who assume a statically linked Go binary carries its own root list. It does not. The second common cause is that the collector's certificate is now issued by an internal CA. Public roots cannot chain to it by construction, so the image needs the company root regardless of how complete its public bundle is. ## The fixes, narrowest first 1. **Set `RootCAs` explicitly** to a pool holding the company root, loaded from a path the platform mounts. This is the narrowest option: it names exactly what this client trusts, it is visible in code review, and it is testable. For a shipper that only ever dials one collector, refusing every public CA is a feature. 2. **Install the CA bundle in the image** (add the certificates package to the base, or copy the bundle in during the build). Right when the binary talks to many destinations and should follow the host's trust policy. 3. **Point `SSL_CERT_FILE` at a mounted bundle.** Useful when you cannot rebuild, but it is process-wide, Unix-only, and silently does nothing if the path is wrong — a poor default, an acceptable emergency lever. All three keep verification on. The one-line alternative does not, and it is the one that gets copy-pasted: a debugging workaround committed at 3am appears in the next service that dials the same endpoint, then the one after that, because it visibly "worked" the last time. ## Making the workaround stick out Two mechanisms, and you want both. **A check that fails the build.** `crypto/tls`'s skip flag is a plain struct field, so a small analyzer over the AST — the kind you write with `golang.org/x/tools/go/analysis` — can flag any composite literal that sets it true, with an allow-list for specific test files. A CI grep is a cruder version of the same guard and still worth having on day one. The value is that the change stops being invisible: a reviewer sees a build failure and an explicit exception rather than one extra field in a struct literal. **A negative test.** This is the part people miss. Tests normally assert that things work, and a client with verification disabled passes every such test — that is exactly why the workaround survives review. So write the test that only passes while verification is on: dial with a deliberately wrong `ServerName` and require the handshake to fail, ideally asserting the failure really is a hostname mismatch rather than a connection refused. If someone later sets the skip flag, that dial now succeeds and the test turns red, which is the behaviour you want from a guard. A companion test that dials with an empty `RootCAs` pool and requires an unknown-authority failure covers the trust-root half in the same way. ## Handling the 3am version When the pager has gone off and the deploy is broken, the pressure is to take the one-line fix. The honest answer is that the correct fixes are also small: mounting a bundle and restarting, or shipping a config change that points `RootCAs` at a file, are the same order of effort as flipping a boolean. If you truly must ship the flag to stop the bleeding, treat it like any other emergency measure: scope it to the single call site, open the ticket before you merge, and put the negative test in the same change so the follow-up is red until it is done. The failure mode this leaf exists to prevent is not one deliberate exception — it is an undated workaround that quietly becomes the house style.