Replacing a Credential

## The swap is three stages, and the order is fixed A credential used in front of a broker cluster is **accepted** by the nodes and **presented** by the clients. Those are two populations that change at different speeds, and keeping traffic flowing means never asking them to change at the same instant. The order is: 1. **Widen the trust set on every node** so that both the outgoing and the incoming credential authenticate. Nothing presents the new one yet — the cluster is merely prepared to accept it. 2. **Move the clients onto the new credential**, at whatever pace the fleet deploys. Throughout this stage some clients present the old credential and some the new one, and both succeed. 3. **Narrow the trust set** so only the new credential authenticates. This is the step that can break something, and it runs last, on evidence rather than on a date. Reverse any pair and you get the outage the exercise exists to avoid. Narrow first and every holder is refused at once. Move clients first and they present something no node will accept. ## Why 'every node' is the part that bites A client is normally handed an address for the cluster rather than for one member, and it reconnects to whichever member answers — after its own restart, after a blip, after a member is patched. The trust set is therefore not one thing you change; it is the same change applied to every member, and until the last member has it, the new credential works or fails depending on where a connection happens to land. The symptom is the awkward one. Not a clean break that points straight at the change, but a small share of connection failures that moves around as clients reconnect, and that reads like a network problem to everyone looking at it. ## Two rollouts, two verifications The node half and the client half fail in different places, so they are tracked and proved separately: | Stage | Where it lands | What proves it done | What its failure looks like | |---|---|---|---| | Widening the trust set | every node | both credentials authenticate against each member | intermittent refusals that follow reconnections | | Moving the clients | every client process | each principal seen authenticating with the new credential | one service left behind, quiet until it reconnects | | Narrowing the trust set | every node | the old credential is refused everywhere | the outage you were avoiding, arriving later | Tracking them as one change is how the second half gets assumed rather than checked. A pipeline reporting a successful deploy says the new credential is in place in that service; it does not say the service has opened a connection with it. ## The node-to-node hop is a third rollout Where members authenticate to one another, each node is both a presenter and an acceptor on the node-to-node hop, and the same three stages apply to it. The failure is different in kind: a mistake there does not refuse a client, it stops members talking, which halts replication and can stop the cluster accepting writes at all. Sequence and verify that hop on its own rather than folding it into the client rollout because the material looks the same. ## Three preconditions that vary, and must be settled first - **Whether a node picks up new trust material without a restart.** Some platforms re-read it periodically or on an explicit reload; others read it once at start. If yours is the second kind, the node half becomes a touch-every-member operation with all the care that implies, and that changes the plan rather than the order. - **Whether the replacement yields the same principal.** If the identity the broker derives from the new credential is unchanged, nothing downstream has to move. If it differs, that principal's grants must already be in place before any client switches — what those grants should say is a separate subject, but their existence is a precondition of this swap. - **Whether the two credentials can be told apart afterwards.** Stage three is decided on evidence, and that evidence is far weaker if the cluster's records cannot say which of the two a client presented. ## Why this is sharper on a broker than in front of a request-per-connection service Two properties do it. First, the acceptor is a cluster rather than a process, so the change is many applications of the same edit and is routinely left partly done. Second, clients hold long connections and are not asked to prove themselves again, so the client half completes invisibly and late. Widen everywhere, move, then narrow is the order that absorbs both.