An on-call engineer reads a credential at 2am by authenticating as the payments service itself — what does that make impossible?
answer
- the check passed; the meaning did not
- one caller name, two kinds of caller
- never captured, so never recoverable
- reach of the service, not the person
- the missing human path is the defect
basics
~20 sTelling a person from a process, ever again. The store verified a valid workload identity and recorded the service as the caller, so no later question about whether a human read that value — and which human — has an answer anywhere.
solid answer
~40 sThe check succeeded, which is what makes this hard to see: the store was presented with the payments service's identity, verified it, released the value, and recorded the service as the caller. From that moment the read is indistinguishable from the service's own routine reads at restart. Anyone asking later whether a person touched that credential, who it was, or whether this was an incident or normal operation, has nothing to work from. The engineer also inherits everything the workload may reach, not just the one value they needed, and inherits it under a name that is not theirs. The shape that works is the opposite: the person authenticates as themselves, and their own identity is granted the read.
code
json · 7 lines{
"caller": "payments-service",
"callerKind": "workload",
"credentialRead": "payments-database-owner",
"readAt": "02:14",
"humanBehindIt": null
}go deeper
Recall that a store records the identity that was presented, not the person holding it. If a human uses a service's identity, the record says the service.
Explain why the information is unrecoverable rather than merely absent: it was never part of the call. Name what else the engineer could reach while holding the workload's identity.
Separate the steps out loud — authentication succeeded, attribution was destroyed — and identify the real finding as the missing human path to that value, not the engineer's decision at 2am.
Decide what on-call access to production credentials looks like across the estate, fast enough that nobody borrows an identity under pressure, and say what you accept in exchange for that speed.
## What the store recorded Nothing failed. A valid workload identity was presented, checked and accepted, and the store wrote down what it saw: the payments service read the payments database credential at 02:14. That record is accurate. It is also, from now on, wrong about the thing anybody will want to know. The critical point is that the missing information was never available to be captured. The store had no way to know a human was driving, because at the moment of the call there was nothing that distinguished this from the service's own unattended read. No amount of retention, detail or tooling recovers it afterwards. ## Three questions that now have no answer 1. **Did a person read this value at all?** The service reads it at every restart. A human read inside that stream is invisible. 2. **Which person?** If four engineers can authenticate as that service, the answer is the same for all four. 3. **Was this the incident, or normal operation?** The timestamp says 2am, which is when unattended work happens too. And a fourth that is really about scope: **what else did they reach?** The workload's identity is not narrowed to the one credential the engineer needed. Whatever the service may read, the engineer could read, for as long as they held that path. ## Why the borrowed identity is the wrong instrument The workload's identity is built for the opposite set of constraints from a person's. It exists to work at any hour with nobody present, to keep working across restarts indefinitely, and to cover everything the service needs during its whole life. Every one of those properties is a liability when a person is behind it. | | The engineer's own identity | The payments service's identity, borrowed | |---|---|---| | Name on the record | the individual | the service | | Reach | what that person was granted | everything the workload may read | | Interactive proof | possible at the moment of use | none, by design | | End of access | with the person, by an existing routine | with the service, which is to say not at all | | Distinguishable from routine work | yes | no | ## The honest part of the answer Borrowing happens for a reason, and the reason is usually good: at 2am the engineer needed the value, and no path existed for them to get it as themselves. Saying "they should not have done that" is a weak answer. The strong answer names the missing thing — a human path to that value — and says that its absence is the defect the incident exposed. The same applies to the reverse arrangement, which shows up in the same estates: a workload running as somebody's personal identity. Both are the same mistake, which is one identity serving two caller classes with incompatible requirements. ## What to do instead - **Let the person authenticate as themselves** and grant that identity the read. The record then names them by construction, not by convention. - **Grant the human path the value, not the workload's whole reach.** The engineer needed one credential; the service's identity carries everything. - **Expect the human path to differ from the service's.** It can be interactive, it can be short, and it can end — three things the unattended path cannot afford. - **If a person must ever act as a workload,** treat that as an exception with its own record naming the person, not as the normal route. Whether such an exception needs approval in front of it, and how the read is later reconstructed, are separate subjects with their own answers. - **Do not solve it by copying the value somewhere the engineer can read it.** That creates a second copy that is not authenticated at all, and moves the problem rather than fixing it. ## What a weak answer looks like The common weak answers all misidentify the failed step. "The store should have detected a human" — it cannot, and nothing in the presented proof carries that information. "The workload identity is less secure" — it may well be verified more strongly and more consistently than the person's. "Time-box the borrowed identity" — a time limit bounds how long the reach lasts, which is worth something, but it puts no name in the record, and the record is what the question is about. The answer an interviewer is listening for separates the two things cleanly: **authentication succeeded, attribution was destroyed**, and attribution was destroyed at the moment of the call, not later.
- Could the store have detected that a human was driving the call?Not from what it was given. It received the workload's proof, which is exactly what the service presents at every restart, and verified it. Timing or volume signals might later look unusual, but they are inference about a pattern, not identification of a caller. The information identifying a person was never part of the call, so nothing downstream reconstructs it.
- Does time-boxing the borrowed identity fix the problem?It fixes half of one part. A limit bounds how long the engineer holds the workload's full reach, which is genuinely worth having. It puts no name in the record, so every question about who read the value remains unanswerable. Treat it as damage limitation on an exception, never as the design.
- The estate has no human path to that credential at 2am. What is the actual finding?That the human path is missing, and the borrowing is the symptom. Write the finding against the gap, not against the engineer: on-call needs a way to reach specific production credentials as themselves, fast enough to be the obvious choice under pressure. If it is not fast, people will route around it again on the next incident.
saying these in an interview costs you the question
- Says the store should have detected the human behind the call.
- Claims the record can be re-attributed to a person afterwards.
- Treats a time limit on the borrowed identity as the fix.
- Assumes the engineer only reached the one value they needed.
- Blames the engineer instead of the missing human path.
- Thinks copying the value somewhere readable is a safer workaround.