An operator replaced a payments service's database password in one step and requests began failing — what went wrong?
answer
- no single instant
- three copies, one truth
- the accepting side decides
- both values valid at once
- withdraw after, not during
basics
~20 sA one-step swap has no instant at which every holder switches. The database stopped accepting the old password while running processes still held it, so their next connection failed. Safe replacement needs both values accepted at once.
solid answer
~40 sAt any moment a credential exists in three places: the value the store serves, the copy each consumer is holding in memory, and the value the database will accept. Changing the database in one step moves the third without moving the second, so every process that read the password at start-up keeps presenting a value that is now rejected. The fix is an overlap window: have the database accept both the old and the new password for that account, publish the new value, let each consumer re-read and re-open its connections, and only then withdraw the old one. Note what this requires — the capability lives in the *accepting* system, not in the store. A store that holds two versions changes nothing if the database accepts only one.
code
pseudocode · 13 lines# one-step swap - the failure
setPassword(account = "payments_app", value = newValue) # old value refused from here
writeToStore(name = "payments/db-password", value = newValue)
# every process still holding oldValue fails on its next connection
# the same change with an overlap window
step 1: addSecondPassword(account = "payments_app", value = newValue) # both accepted
step 2: writeToStore(name = "payments/db-password", value = newValue)
step 3: for each consumer holding oldValue:
consumer.rereadValue()
consumer.reopenConnections()
step 4: wait until acceptingSide.noConnectionAuthenticatedWith(oldValue)
step 5: removePassword(account = "payments_app", value = oldValue) # withdrawalgo deeper
Remember that a service reads its password once, usually at start-up, and holds a copy. Changing it elsewhere does not change what that running process is presenting.
Explain the three populations — stored value, held copies, accepted value — and why only a period in which both values are accepted lets them converge without a gap.
Show the ordered change: add at the accepting side, publish, move consumers, confirm, withdraw. Name what fails in each direction if two steps are swapped, and what your back-out is.
The interesting judgment is which systems in the estate can offer an overlap at all, and what you require of a new downstream so that its credential is replaceable without a change window.
## Three copies, not one The word "the password" hides the problem. A credential in use exists in at least three places at once: - **The value the store serves** — what a reader gets if it asks right now. - **The value each consumer is holding** — a copy taken when that process read it, usually at start-up, and kept in memory for as long as the process runs. - **The value the accepting system will accept** — here, what the database checks a connection against. A rotation is the work of getting all three onto a new value. The one-step swap assumes that changing the third drags the second along, and nothing in the system makes that true. Nobody pushes a new value into a running process; the process holds what it read. ## Why the single step fails The operator changed the database side and, in the same breath, made every existing holder wrong. There is no instant at which the fleet switches together, so the change creates a gap in which the only value the consumers have is the one value the database now refuses. The failure looks like an authentication error on the next connection each process opens — which is why it often appears minutes later, in bursts, as pools recycle connections rather than all at once. Two details make it worse than it first looks. First, the population of holders is larger than the running services: a nightly job, a reporting task, a maintenance script and a standby instance are all holders, and some of them will not fail until the next time they run. Second, a fleet restart does not rescue you, because restarts take time and every process that has not yet restarted is failing during it. ## What an overlap window is An overlap window is a period in which the accepting system accepts **both** the outgoing and the incoming value for the same identity. During it, a holder of either value works, so consumers can move across one at a time, in any order, at whatever pace their restart or re-read cadence allows. The window ends with a separate, deliberate act: **withdrawal**, which is the change at the accepting side that makes the old value stop working. This is the distinction most candidates blur. Putting a new value in place is **rotation**. Making the old value stop working is **withdrawal**. A rotation that never withdrew the old value has left it live; a withdrawal with no replacement in the consumers' hands is the outage above. | | One-step swap | Overlap window | |---|---|---| | A process that read at start-up | fails at its next connection | keeps working until it re-reads | | A job that runs tomorrow | fails on its first run | works, then picks up the new value | | Back-out if something is wrong | re-apply the old value under the outage | point consumers back; both still accepted | | When the old value stops working | immediately, by accident | when you withdraw it, on purpose | ## The order that works 1. Add the new value at the accepting system so that **both** are accepted for that identity. 2. Publish the new value to the store, so a reader now gets it. 3. Move consumers across: each re-reads and re-opens its connections, by restart, by a helper process beside the workload where one exists, or by whatever mechanism that consumer already has. 4. Confirm the old value is no longer being used. 5. Withdraw the old value at the accepting system. Steps 1 and 5 are the ones people forget, and they are the only two that touch the thing doing the checking. ## What the window does not do An overlap window removes the instant in which no value works. It does not move consumers for you: a process that never re-reads still holds the old value at step 5, and withdrawal will break it exactly as the one-step swap would have. It also does not make the rotation finished. Until withdrawal, the old value is still a working credential in the hands of whoever has a copy — which is the entire reason the rotation was scheduled. One more limit worth stating plainly: the ability to accept two values at once is a property of the accepting system. Some accept a second credential on the same identity; some require you to create a second identity and move callers to it; some accept exactly one and give you no overlap at all. Designs genuinely differ, and the rotation plan has to be written against the one in front of you.
- What must the accepting system support for an overlap window to exist at all?Two credentials accepted for one identity at the same time, or a second identity you can move callers to. Where it offers neither, there is no overlap available and the rotation needs a different shape — a planned gap, or an intermediary that holds the credential for everyone.
- The database now accepts both passwords and the new value is in the store. What still has to happen?Each holder must re-read the value and re-open its connections; nothing pushes it. A long-running process that read once at start-up keeps the old copy until it restarts or is made to re-read, and it is that population — not the store — that decides when the old value can be withdrawn.
- Why does restarting the whole fleet at once not remove the need for an overlap?Restarts are not instantaneous, and during the roll every process that has not yet come back is still presenting the old value against a database that no longer accepts it. You have converted a permanent failure into a shorter one, not into a safe change.
saying these in an interview costs you the question
- Assumes the store pushes the new value into running processes.
- Calls the rotation finished once the new value is in the store.
- Claims a simultaneous fleet restart removes the need for an overlap.
- Assumes every accepting system can hold two credentials for one identity.
- Treats writing the new value and withdrawing the old one as one action.