A six-hour nightly export read its credential once at start-up and failed at hour five — why did the process never notice?

## What the process is actually holding A credential is two things kept in two different places. One is the **value** — the bytes the job copied into a variable when it started. The other is a **record at the issuer**, which says who holds it and until when. Only the second one has a clock. The copy in the process is inert. It does not count down, it does not get recalled, and nothing in the runtime distinguishes it from a hostname or a batch size. That is the whole of the answer to *why did it not notice*: **there is no channel from an issuer to a holder.** A store hands a value out and then has no session with the process that took it — no callback, no socket to push a warning down, often not even a record of which host ended up with it. A holder learns its credential stopped being valid the way anyone does: by presenting it and being refused. ## Where the expiry is actually enforced | Party | What it knows | What it does when the window closes | |---|---|---| | The process holding the value | the bytes it copied at start-up | nothing — it has no reason to ask again | | The issuer that minted it | who was issued what, and until when | stops vouching for it; what happens to anything already open depends on the design | | The system that accepts it | whether the credential presented right now is valid | refuses the next request that presents it | Notice which row produces the error text. The export's failure message comes from the **partner's object store** — the acceptor — not from the credential's issuer and not from any library inside the job. It is usually a flat authentication refusal with no mention of time, which is exactly why the first instinct is to open a ticket with the partner. ## Why the failure lands at hour five The job did not survive five hours by luck. It survived because, for those five hours, every request either fell inside the validity window or travelled over a session that had been authenticated inside it. - Work done before the window closed was simply valid. - Work done after it closed still succeeded on any session authenticated earlier that the acceptor does not re-check. - The refusal arrives at the **first request after the window closes that actually re-presents the credential** — a new connection, a re-authentication, or a per-request proof the acceptor validates every time. So the hour is not a property of the credential. A consumer that proves itself on every request fails within seconds of the window closing. A consumer that proves itself once and then streams fails at whatever unrelated moment forces it to prove itself again — a connection the pool replaced, a socket that reset, a second phase of the job that opens a new destination. Same expiry, wildly different visible hour. ## The margin nobody measured Put numbers on it. The export resolves its credential at 22:00 and is handed a window that closes at 03:00 — five hours. The run needs six. The failure is already scheduled the moment the job starts: at 03:00, or at the first reconnect after it. It ran clean for months because the margin was positive and quietly shrinking. The data grew, the run got slower, the scheduled start drifted later, or the window it is issued got shorter. None of those is a bug on its own; together they crossed a line nobody was watching, because **nothing in the system compares the two numbers**. ## Why restarting appears to fix it Re-running the export re-resolves the credential, gets a fresh window, and the second attempt fits inside it. This is the most expensive kind of green: the design is untouched, five hours of work were paid for twice, and tomorrow night the same arithmetic applies with slightly less margin. A restart that succeeds is **confirmation of the diagnosis**, not the remedy — it is the cleanest evidence you will get that the failure was time, not the partner. ## What the consumer owes instead 1. **Record the two times** at resolve time — when the value was resolved and when it stops being valid. The times, as ordinary telemetry; never the value. 2. **Re-resolve at a boundary it chooses**, before the window closes, rather than at the moment it is refused. 3. **Re-establish the sessions** that were authenticated with the previous value, because a fresh value in a variable changes nothing on a connection that is already open. 4. **Keep the unit of work small and restartable**, so that step 2 costs a chunk rather than a night. The first of those is the one teams skip, and it is the one that turns this from an incident into a graph.