Rotating Keys Without Downtime

## What the race actually is Go defines a data race precisely: two goroutines access the same memory location, at least one access is a write, and there is no happens-before relationship ordering them. Nothing in that definition mentions the size of the value. A `*tls.Certificate` is one machine word, and on common hardware the store instruction is indivisible — but indivisibility of a store is not ordering. The compiler may hoist the read out of a loop, keep it in a register across the reload, or reorder it against other memory operations, and the hardware may make the write visible to other cores later than you expect. A racy program has undefined behaviour in Go; it is not "probably fine". The TLS certificate hot-swap hits this squarely because of how the callback is invoked. Each incoming connection is handled on its own goroutine, and each of those goroutines calls `GetCertificate` during its handshake. So a busy server has many concurrent readers by construction. Add one reload goroutine — woken by a timer, a signal, or a watcher — and the racy shape appears the moment somebody writes the obvious code. ```go var cert *tls.Certificate func reload() error { c, err := tls.LoadX509KeyPair("cert.pem", "key.pem") if err != nil { return err } cert = &c // unsynchronised write return nil } ``` ## What the race detector does and does not tell you Building or testing with `-race` instruments memory accesses and maintains happens-before information at runtime. When it sees the reload's write and a handshake goroutine's read with no edge between them, it prints both stacks and the goroutine that created each. That report is authoritative: a reported race is a real one. The converse is not true. The detector only sees accesses that actually execute in that run. A test that reloads once before the server starts serving proves nothing, because the two accesses never overlap. This is why the useful artefact for rotation code is a test that *renews while connections are being served*: start a server, drive handshakes from several goroutines in a loop, and run the reload repeatedly in parallel with them, all under `-race`. That test is what makes a hot-swap defensible in review. ## The fix: publish, do not mutate `atomic.Pointer[T]` gives a typed atomic pointer cell with `Load`, `Store`, `Swap` and `CompareAndSwap`. Reads and writes through it are synchronised, so the race disappears and readers stay lock-free. ```go type reloader struct { cert atomic.Pointer[tls.Certificate] } func (r *reloader) reload(certFile, keyFile string) error { c, err := tls.LoadX509KeyPair(certFile, keyFile) if err != nil { return err // keep serving the previously published pair } r.cert.Store(&c) return nil } func (r *reloader) GetCertificate(*tls.ClientHelloInfo) (*tls.Certificate, error) { return r.cert.Load(), nil } ``` The discipline that matters as much as the atomic itself is **immutable publication**. Once `Store` has made a `tls.Certificate` visible, treat it as frozen. Do not append to its `Certificate` chain, do not set its `OCSPStaple`, do not swap its `PrivateKey` — a handshake goroutine may be holding that exact pointer and reading those fields right now, and mutating them is a fresh race the atomic does not cover. Build the complete replacement value first, then publish it in one `Store`. Old certificates are collected normally once the last handshake that grabbed one has finished with it. Note also that the failed-reload branch returns without storing. That is deliberate: a truncated file or a key that does not match its certificate should leave the server running on material that still works, not publish a broken pair. ## Alternatives and when they are right A `sync.RWMutex` around a plain field is equally correct: `RLock` in the callback, `Lock` in the reload. It costs more than an atomic load on a read-mostly path and it is easy to widen the critical section accidentally, but if you need to publish *several* related fields together — say a certificate and a matching root pool that must change as a unit — a mutex is the honest tool, because one atomic pointer protects one word, not an invariant across several. The other way to keep the atomic in that case is to put both fields in a struct and publish a pointer to the whole struct. The older `atomic.Value` does the same job without generics, but it panics if you store values of inconsistent concrete types and it cannot hold a nil interface, so the typed `atomic.Pointer[T]` is the cleaner choice where it is available. ## The read the callback returns One subtlety worth stating: the callback returns whatever pointer `Load` produced, and that pointer stays valid for the whole handshake even if a reload publishes a different one microseconds later. Each handshake sees a consistent snapshot; there is no torn state and no need to hold anything for the duration of the connection.