You wrote a replacement value for a leaked fleet credential an hour ago, yet the old value still authenticates — why?
answer
- three systems, three states
- the store is not the verifier
- holders keep what they last read
- delete removes your copy only
- withdrawal acts on the accepting system
basics
~20 sWriting a new value into the store is only half a replacement. The system that honours the credential was never told to stop accepting the old value, and every holder still presents it. Withdrawal is a separate action against a different system.
solid answer
~40 sA credential lives in two places at once: where it is kept and served, and where it is honoured. Writing a replacement changes only the first. The accepting system still honours the old value because nobody asked it to stop, and whoever took the value is presenting it directly there — they never read your store. The holders are a third state again: they keep whatever they last read until they restart or refresh. So an emergency replacement is three actions, not one — put a new value in place, move the holders across, and withdraw the old value at the system that accepts it — and only the third one ends the leak. Rotating adds a working value; revoking makes the old one stop working. Say which you mean every time.
go deeper
Recall that a credential lives in two places at once: where it is kept and where it is honoured. Changing what is kept does not change what is honoured.
Explain the three actions — write a replacement, move the holders, withdraw the old value — and say which system each one touches and what it deliberately leaves untouched.
Show how you would catch this in the room: a stolen credential produces successful use, so failure alerting stays quiet, and the dashboard shows the value you wrote rather than the values still being honoured.
The real trade is how much of this you refuse to leave to a runbook. Bounded-lifetime credentials turn withdrawal into an acceleration rather than the only remedy, and that is a design decision taken long before the incident.
## What the write actually changed An emergency replacement is three separate actions against three separate systems, and writing a new value is only the first of them. - The **store** — wherever the value is kept and served from — now serves a different string. - The **holders** — the processes, devices and jobs that use the credential — still hold whatever they read last. Most will keep holding it until they restart, until a cached copy ages out, or until something makes them fetch again. - The **accepting system** — the service that decides whether a presented credential is good — has been told nothing at all. It keeps honouring the old value because nobody asked it to stop. That split is the whole answer to why the old value still authenticates an hour later. Whoever took the value is not reading your store; they present the value straight to the system that accepts it. Nothing you did is visible to them. ## Rotation adds, withdrawal removes | action | acts on | effect on the old value | effect on holders | |---|---|---|---| | write a replacement | the store | none | none until they re-read | | move holders across | each holder | none | they now present the new value | | withdraw the old value | the accepting system | refused from that moment | any holder still on it breaks | Read the middle column. Only the last row touches the leaked value. "We rotated it" describes the first row; "we revoked it" describes the third; an incident that did only the first left the exposure running while everyone believed it was closed. The two actions are usually done in the same hour and are still separate, and the order between them is a deliberate choice with a real cost either way. ## Deleting is not withdrawing The common instinct is to delete the compromised value from the store so it is gone. It is not gone. You removed the one copy you controlled and left every copy you did not: the copy in each holder's memory, the copy in a rendered configuration file on disk, the copy in a shell history, the copy in whoever's hands. The accepting system never consulted your store, so its behaviour is unchanged. Deletion also costs you two things while buying nothing: - you lose the ability to compare a value someone presents against the one you had, which is how you tell "the leaked value" from "a holder that simply never updated"; - you lose the record of what that value was and which identities read it, if deleting prunes the history with it. ## The one exception, and why it is not a get-out If the credential was generated with a bounded lifetime, it stops working at its expiry whether or not anyone acts. That is genuinely different from a static value typed in once and shared: the withdrawal accelerates an ending that was already scheduled. It is the strongest argument for short-lived credentials, and it is not permission to wait. Under a live exposure you still withdraw, unless the remaining lifetime is demonstrably shorter than the withdrawal would take — and that is a number you should be able to state, not assume. ## Why this goes unnoticed for hours This failure is quiet by construction: - a stolen credential **succeeds**, so alerting built on failed authentication stays silent; - the dashboard shows the store's current value, not the set of values the accepting system honours; - holders that moved across are healthy, and holders that did not are also healthy, so nothing breaks to draw attention; - the only signal is a successful use from an identity, at an hour, or at a volume nobody recognises. ## What finished actually looks like 1. The replacement exists and is being served. 2. Each holder is confirmed to be presenting it — confirmed, not assumed. 3. The old value is withdrawn at every system that honours it. 4. An attempt to authenticate with the old value is refused, and the refusal is recorded with a time. 5. Records show no successful use of the old value after that time. Steps three to five are the half that gets skipped at two in the morning, and step two is what decides whether step three causes an outage — which is why the order between them is the real decision under pressure, not an implementation detail.
- Does destroying the old value in the store count as withdrawal?No. Destroying it removes the copy you controlled and leaves every copy already delivered, and the system that honours the credential never consulted your store in the first place. Withdrawal is an action against that system. Destruction also costs you the ability to recognise the old value later and, if it prunes history, the record of who read it.
- If the credential had been generated with a short lifetime instead of typed in once, what changes?The old value stops working at its expiry without anyone acting, so withdrawal becomes an acceleration rather than the only remedy. That is the argument for short-lived credentials. It does not mean waiting is acceptable: you withdraw anyway unless the remaining lifetime is shorter than the withdrawal would take, and you should know that number rather than guess it.
Cutting a new key and handing copies round is not the same as changing the lock. Until the lock itself is changed, every old key still opens the door — including the one that was copied.
saying these in an interview costs you the question
- Says writing a new value into the store revokes the old one
- Believes deleting the leaked value from the store stops it working
- Assumes every holder picks up a new value the instant it is written
- Uses rotation and revocation as if they were one action
- Calls the incident closed because the store now shows one value