Your ACME automation renews several names in parallel and intermittently fails with badNonce — what is happening and what should the client do?

The nonce is the part of ACME that works perfectly in a test run of one certificate and starts failing the week the estate reaches fifty. Understanding it is understanding why. ## What the nonce is for A signed ACME request is not confidential. Anyone who can see it has a complete, validly signed object; if the server accepted it twice, a captured request could be replayed — a revocation, an order, a challenge-ready POST. So each request's protected header carries a `nonce` that the server issued and will accept **once**. Together with the `url` field, this pins a signature to one request, at one resource, at one time. Nonces arrive two ways: - from the `newNonce` resource, which a client can query directly; and - in the `Replay-Nonce` response header that the server attaches to responses, so a client can chain one request to the next without an extra round trip. ## Why parallelism breaks it A nonce is consumed by the request that uses it, not by the worker that took it. Renewing many names at once means several signed requests in flight, and the moment two of them are signed with the same value, one is rejected. Symptoms line up with this exactly: - the failure is **intermittent** and scales with concurrency; - it never reproduces in a single-threaded rerun; - the rejected request is well formed, so nothing else in the log looks wrong; - retrying "the whole renewal" usually succeeds, which hides the cause. A stale nonce fails the same way: a value harvested, then held while a long DNS propagation wait ran, may no longer be accepted. ## The contract on the error On a `badNonce` error the server **must** include a fresh `Replay-Nonce` header that it will accept on a retry, and the client **should** retry the request using it. Those are two different strengths of requirement and it is worth keeping them apart: the server is obliged to make the retry possible; the client is advised to take it. The error document is a problem object with the type `urn:ietf:params:acme:error:badNonce`. The retry is of the **same** request, re-signed. The payload does not change — only the nonce in the protected header, and therefore the signature. ## What a client should do 1. **Keep a nonce pool** and add the `Replay-Nonce` of every response to it, including error responses. Fetch from `newNonce` only when the pool is empty. 2. **Take a nonce per request, never per batch**, and never reuse one after a response has come back. 3. **Retry on badNonce with the nonce from that error response**, bounded — three to five attempts, not an unbounded loop, so a server that rejects every nonce surfaces as a failure instead of a spin. 4. **Do not alert on a badNonce retry.** It is expected traffic. Alert on the renewal not completing, which is the outcome that matters. 5. **Consider serialising per account** if the client cannot manage a pool safely; the throughput cost is trivial next to a renewal that silently stops. ## The operational shape | Symptom | Likely cause | Action | |---|---|---| | badNonce under concurrency only | one nonce used by two in-flight requests | pool nonces, one per request | | badNonce after a long wait step | the held value went stale | take the nonce immediately before signing | | badNonce on every attempt | clock or state problem at the server side | stop after the bounded retries and report | For a chain of pop-up sites renewing dozens of names on one nightly job, the dangerous version of this bug is not the noisy one. It is the job that treats a badNonce as a hard failure for that name, logs it, and moves on — so a handful of names quietly miss their renewal window every night while the job's exit status stays green.