Where does your service provider's assertion replay cache live when several replicas sit behind a load balancer, and how long do entries last?

## Why this is infrastructure rather than a field A service provider is required to reject a message it has already consumed. The only way to know it has already consumed one is to have written that fact down somewhere. That makes the replay cache a piece of shared, availability-critical infrastructure with a capacity, a latency budget and a failure mode — the same class of thing as a session store — and not, as it first appears, a flag on a row. In the permit system, every contractor firm's sign-in passes through it, several times a minute at shift change, across however many replicas are running. ## A per-process cache is a hole, not an optimisation The tempting implementation is an in-memory map with an expiry, one per process. It is fast, it needs no dependency, and it passes every test you are likely to write, because a test runs one process. Behind a load balancer it does not enforce single use across the fleet. A replayed message is routed by the balancer, not by whoever holds the cache entry, so a replay lands on whichever replica has not seen that id and is accepted there. With a handful of replicas, an attacker who can post the same body repeatedly gets accepted with high probability, and the defect is invisible in logs unless you correlate accepted message ids across instances. It also fails on deploy: a rolling restart empties every map at once. The correct shape: - a **shared key-value store** the whole fleet reads and writes; - entries created by a single **atomic set-if-absent** operation; - a per-entry expiry the store enforces for you; - a key of **issuer plus message id**, because two counterparties may legitimately mint the same id string. ## The claim must be atomic A read-then-write implementation has a lost-update race. Two posts of the same message arriving milliseconds apart at two replicas both read *not present*, both proceed, and both issue a session. This is not theoretical under a deliberate replay, because a replay is exactly a burst of identical requests. The operation you want is the store's native *create only if absent* with a time-to-live set in the same call, returning whether you won. The claim also has to happen **before** the session is issued. Claiming afterwards means the window between validation and issuance is a window in which a concurrent copy also succeeds. ## Sizing the time-to-live from the right clock An entry only needs to exist for as long as the message it names could still pass your other checks. That is the end of the message's own validity window, extended by the clock skew you tolerate. Concretely: ``` ttl = (window_end - now) + skew_allowance ``` Three consequences worth being able to state: 1. **Widening skew widens retention.** Tolerating more clock difference keeps every entry alive longer and enlarges the cache proportionally. It is not a free knob. 2. **The local session lifetime is irrelevant.** A session may last eight hours; the message that started it stopped being acceptable minutes after it was issued. Pinning the entry to the session lifetime wastes memory by orders of magnitude. 3. **A fixed hour is a guess.** Derive it, do not choose it. Capacity follows directly: entries in flight are roughly sign-ins per second multiplied by the window in seconds. That number is small, which is the good news — this store is cheap if it is sized from the window and expensive if it is sized from a habit. ## Eviction, and what it actually costs A store configured to evict under memory pressure rather than strictly by expiry will, when it fills, quietly drop entries that are still inside their window. Every dropped entry is a message that can now be replayed. This is the failure that does not page anyone: sign-ins keep working, throughput is fine, and the property you thought you had is gone. So: configure expiry-driven eviction, size for the real peak rather than the average, and alarm on evictions and on the store's memory headroom. Treat an eviction event as a security event with a blast radius you can name — every message whose window had not closed when it was dropped. ## Failing closed If the shared store is unreachable, you have a choice, and it should be made in advance rather than by a timeout default. Accepting messages you cannot claim means abandoning single use for the duration of the outage. Rejecting them means federated sign-in is down for every counterparty while the store is down. For a system that authorises work on live infrastructure, failing closed is the defensible answer, and it makes the replay cache's availability a first-class dependency of sign-in — which is exactly what it is. Say so in the design, and put it on the same monitoring tier as the session store.