The platform's own identity service is down, so credential refresh fails — why do your services keep serving at first, and what ends that?

## The dependency nobody draws Every workload on a large cloud platform proves what it is to that platform before it can do anything useful — read from an object store, write to a managed database, publish a message. On a modern estate it does that with a **short-lived credential**: an artifact carrying an issue time and an expiry, obtained from the provider's own identity service and refreshed on a schedule long before it lapses. The refresh is invisible. It has no dashboard, no owner on the team, and it usually lives inside a platform client library that application code never calls directly. That makes the platform's identity service a **shared dependency**: one service, operated by the provider, that every tenant's every workload touches continuously. When it fails it does not fail for you — it fails for everyone at once. That is the defining property of this failure class and it is why the reflexes learned on ordinary outages do not apply. There is nowhere to shift traffic that is not also calling the same service, and the usual mitigation of replacing unhealthy processes makes things strictly worse. ## Why the first minutes look normal A short-lived credential is a **bearer artifact**. On most platforms the service that accepts it validates it locally — signature and expiry — rather than asking the issuer whether it is still good. Platforms differ in how much of that validation is local and how much is live, but the practical shape is the same everywhere: the refresh path breaks before the call path does. So in the opening minutes: - a process holding an unexpired credential keeps calling platform services and keeps succeeding; - a process whose scheduled refresh fails writes an error and carries on with what it already has; - a process that is starting up, or that has never held a credential, fails at once — it has nothing to fall back on; - your customers see nothing, and your error dashboards are flat. ## The cliff What follows is not degradation. A credential is fully valid right up to its expiry and worth nothing one instant later, so the failure for any given process is a step from roughly zero errors to total rejection. | Moment | What the workload sees | What customers see | |---|---|---| | Identity service fails | Refresh calls error; the held credential is still valid | Nothing | | Between failure and expiry | Repeated refresh failures in logs; a process that restarts cannot come back | Nothing, unless a process restarts | | At expiry | Every platform call rejected as unauthenticated | Total failure for that process | | After the service returns | Refresh succeeds, possibly only after a retry storm subsides | Recovery, sometimes slower than the outage itself | The last row is the one teams are least ready for. When the identity service comes back, every workload that has been retrying is waiting on it simultaneously, and an unbounded retry loop turns a recovering service into an overloaded one. ## Why a whole fleet goes together The cliff is per-process, but the processes rarely fall independently. Three things line them up: 1. **Shared deployment time.** A fleet rolled out in one window fetched its first credential in one window, so every subsequent refresh is on the same clock. 2. **A single lifetime policy.** The credential lifetime is usually one value applied across the estate, so every process has the same amount of grace. 3. **No jitter.** Refresh schedules derived deterministically from the issue time reproduce the alignment on every cycle. The result is correlated failure inside your tenant stacked on top of correlated failure across tenants — a narrow window in which almost everything expires. ## What you can actually do - **Refresh ahead, with jitter.** Refresh at a fraction of the lifetime rather than near its end, and add a random offset so the fleet's expiries spread out instead of stacking. - **Treat a failed refresh as a countdown, not an error.** The useful alarm is *time until the oldest held credential expires*, which tells you how long you have; a refresh-error rate tells you only that something started. - **Retry with backoff and jitter.** This protects the identity service on the way back up, which is when it is most fragile. - **Know which product paths need a platform call at all.** Paths served entirely from memory or from an already-open connection survive the cliff; naming them in advance is what makes a deliberate degraded mode possible instead of an improvised one. - **Do not conclude that long-lived static keys are the fix.** They genuinely would have survived this outage. They buy that by staying valuable to an attacker forever, which is a far worse trade than a rare correlated failure of the provider's identity service.