The secret store is rebuilt from last night's backup — what worked before the loss that the restored store does not bring back?
answer
- the store moved back, the estate did not
- anything after the restore point is gone
- issued accounts outlive the record of them
- orphans nobody will rotate or withdraw
- the read trail has a hole for the window
basics
~20 sA restore rewinds the store and nothing else. Values, grants and registrations written after the restore point are gone, as is the read trail for that window — and credentials the store issued into other systems stay live there, unrecorded.
solid answer
~40 sA restore returns the store to a moment in time while every system that consumed it stayed where it was. So anything written after that moment is absent: new values and new versions, new names, grants and caller registrations added since, and the record of who read what in the window between the backup and the loss. The sharpest consequence is the credentials the store generated into other systems after the restore point — those accounts are still live at the far end, and the restored store has no record to renew, rotate or withdraw them by. A restore is therefore the start of a reconciliation, not the end of an incident, and re-establishing that callers are who they claim is its own piece of work.
go deeper
Recall one fact and it will carry you: restoring a secret store puts it back to the moment the backup was taken, so anything written after that moment is simply not there.
Explain the gap concretely: values and versions written since, names created since, grants and registrations added since, and the record of reads in that window are all absent from the restored store.
Demonstrate the reconciliation. Name the orphaned downstream account as the dangerous case, say how you would enumerate it from the far end, and explain why the failures arrive staggered.
Own what must survive the store to make this survivable: the access trail and the issuance record shipped somewhere else, and a stated recovery posture that budgets time for reconciliation, not just for the restore.
## A restore rewinds one system, not the estate Recovering a secret store from an archive puts that store back into the state it held at the **restore point** — the moment the backup was taken. Nothing else moved. The services that read from it, the systems it created accounts in, the people who changed grants and the jobs that ran overnight are all still at the present moment. Every problem that follows a restore comes from that gap, and the interview answer is an inventory of what lives inside it. ## What does come back - Every value exactly as it stood at the restore point, including the versions of it that existed then. - The rules and registrations that existed at the restore point. - The store's own structure: the names it held and the shape of the tree they sat in. That is a real and useful state. It is also a *past* state, and the store gives no signal that it is one — it serves the value it holds with complete confidence. ## What does not come back - **Values written after the restore point**, and any versions created after it. A value changed twice since the backup comes back as it was before either change. - **Names created after the restore point.** A caller that reads one of them now gets a not-found, which reads as a missing configuration rather than as a restore artefact. - **Grants and caller registrations added since.** A service onboarded this morning is unknown to the restored store, and re-establishing that trust is separate work in its own right. - **Anything the store issued to callers since** — the bounded handles it handed out are not in the archive, so callers holding them are no longer recognised. - **The access trail for the window.** Who read which value between the backup and the loss is answerable only from whatever sink received those records before the loss; if the trail lived only inside the store, that question now has no answer. - **The store's knowledge of credentials it generated elsewhere.** This is the one that bites. ## The orphans, in both directions Where a store mints credentials into other systems on request — a downstream account created for a consumer, with an expiry the store is supposed to manage — a restore separates the two halves of that arrangement. Designs differ in how much a store records about what it issued, but none of them can recover a record that was written after the restore point. | What happened after the restore point | Where it is now | Who is left holding it | |---|---|---| | The store created a downstream account | Live in the downstream system | Nobody: the restored store has no record to renew, rotate or withdraw it by | | A value was rotated | The new value is live downstream | The store serves the old one, confidently | | A consumer was registered with the store | Nowhere | The consumer, which can no longer authenticate | | Reads happened | Only in an external sink, if one exists | Whoever has to answer an exposure question later | An orphaned downstream account is the worst of these because nothing alerts on it. It is a working credential, at full privilege, with no expiry anyone is managing and no owner who knows it exists. ## The reconciliation nobody scheduled A restore should be followed immediately by work that only the restore makes necessary: 1. Enumerate, from the downstream systems' own account listings and creation times, every account created after the restore point, and either adopt it into the store or remove it. 2. Probe the credentials the restored store now serves against the systems that accept them, and rotate forward wherever the probe fails. 3. Re-register the callers onboarded since the restore point, and re-apply the grants changed since. 4. Record explicitly that the access trail has a hole, and state its boundaries, so a later question about that window is not answered with false confidence. ## Why the breakage arrives staggered Consumers do not all notice at once. A process that already holds a value keeps working with it until it restarts or its copy ages out; a process that restarts an hour later reads the restored, older value and fails then. So the symptoms spread over hours and arrive detached from the restore that caused them — which is exactly why the reconciliation is done deliberately at the time rather than left for the alerts to find. ## The framing to say out loud "Restored" is a statement about the store's own data at one moment. It is not a statement about the estate. The useful question after any restore is not "is the store back?" but "what did the rest of the estate do between the restore point and now, and which half of each of those changes is now missing?"
- How do you find the downstream accounts the store created after the restore point?Not from the restored store — its record of them was rewound with everything else. You enumerate them where they landed: the downstream systems' own account lists and creation timestamps, plus any issuance records shipped to a sink outside the store. Anything created after the restore point is then adopted into the store or removed.
- What does the hole in the access trail actually cost you?For the window between the restore point and the loss you cannot say who read which value. If a question about exposure arises for that window later, the store cannot answer it, and the only answer comes from a sink that received those records before the loss. That is the argument for shipping the trail off the store.
- Why does breakage after a restore appear over hours rather than at once?Because consumers read at different times. A process still holding the newer value keeps working until it restarts; one that restarts after the restore reads the older value and fails then. The failures therefore spread out and arrive looking unrelated to the recovery.
Putting back last week's switchboard restores every line that existed then, but the lines someone wired since are still connected at the far end — live, and ringing an operator whose board has no record of them.
saying these in an interview costs you the question
- Says a restore returns the system to a working state because the data is back
- Assumes credentials minted after the restore point vanish along with the record of them
- Treats missing read records for the gap as a cosmetic problem
- Believes consumers will re-fetch after the restore and recover on their own
- Plans no reconciliation between the restored store and the systems it issued into