Who holds the renewal clock for a leased credential, and how early should it fire relative to the expiry?

A lease is a promise with a deadline, and someone has to act before that deadline. "The renewal clock" is that piece of the contract: who owns it, when it fires, and what it does when the store says no. ## Exactly one owner per lease The renewal call can sit in two plausible places, and the failure mode is having both: - **In the consumer itself.** The process that holds the credential also extends it. Simple, no extra moving part, and the process knows whether it still needs the credential at all. The cost is that every consumer, in every codebase, carries timer and retry logic, and each gets it slightly wrong. - **In a companion process beside the workload.** One implementation of the clock serves many consumers, which is why the pattern exists. The cost is a second thing that can die quietly: if the companion stops, the consumer keeps working perfectly right up to the expiry it never heard about. What must not happen is both, or neither. Both means two renewal streams against one lease — twice the load on the store, and no meaningful answer to "has anything renewed this lease recently?", which is the signal that tells you a holder has gone. Neither happens more often than anyone admits: a credential fetched once at start-up by a helper that was never asked to keep it alive. ## How early the clock fires Renewal at the moment of expiry is not renewal; it is a coin toss. Fire at a fraction of the window so that what remains is a budget: - **Around two-thirds through the window** is a common shape: it leaves roughly a third of the lifetime for failures, and it keeps the call rate modest. - The remaining time must cover several attempts with backoff **plus** a plausible short outage of the store, not one attempt. - Compute the schedule from the window the store actually granted, not from what was requested — a store may grant less than you asked for, and a clock built on the request will fire late. - Re-derive the next fire time from the **new** expiry after each success, rather than adding a fixed interval to the old one, so drift does not accumulate. ## Spread the fleet Consumers that start together renew together. A fleet redeployed in one rollout has every holder's clock aligned to the same second, and that alignment persists for as long as the processes live, because every renewal returns the same window length. Add a small random offset to the **first** renewal — a few percent of the window is enough — so the fleet's renewals arrive spread out. This is worth doing even when the store would cope with the burst, because a synchronised fleet also means every consumer fails at once during a short store outage, turning a blip into a correlated event. ## What the clock does when the answer is no Two refusals look alike at a glance and must be handled differently: 1. **A transient failure** — the store is briefly unreachable or busy. Back off and try again, still inside the budget the clock reserved. 2. **A refusal to extend any further** — the credential has reached the ceiling on its total life. No retry will ever succeed, and the correct response is to obtain a new credential and adopt it. Routing the second case into the first is the classic way a well-instrumented consumer still runs out of validity: it retries a permanent answer until there is no time left. ## What to record The clock is cheap to observe if you decide what it should say: - Outcome of the last renewal per lease, and when it happened. - Time remaining on the current window at the moment of each attempt — a number that should be roughly constant and is a good alarm when it starts shrinking. - Leases with no renewal in longer than one window, which are the ones whose holder has probably gone. ## Where designs differ Some stores tell a holder exactly how much life a renewal granted and how much of the ceiling remains, which makes the clock trivially correct. Others return only a new deadline, and the holder has to track the total age itself. Some cap the length any single renewal may add, so the clock has to fire more often than the original window would suggest. Build the clock against the numbers the store returns, and treat everything else as an assumption to verify.