A service reads a derived copy from a shared tier that stops answering; what decides whether its requests degrade or fail?

## Start with what the tier holds The scenario says the entries are **derived copies**: faster copies of data a **system of record** still holds authoritatively. That one fact is what makes degradation possible at all. Had the entries been the **sole home for ephemeral state** - facts with no other holder - the same outage would not be a latency event but a loss of facts, and no read path could recover them. Every answer about a tier that has stopped answering begins by saying which of the two it is, because the correct answers are opposites. ## What the caller experiences, in order 1. A request that would have consulted the tier does not get an answer promptly. An unreachable component typically does not refuse politely; it leaves the caller waiting, holding a thread and a connection, until the caller stops waiting. How that waiting is bounded is a separate subject, but the waiting itself sits on the hot path of every request that consults the tier. 2. If the code has a route to the system of record from that point, the read is served - slower, but correct. The user may see a slower page rather than an error. 3. That route now carries **all** the reads the tier had been absorbing, arriving together with no ramp. This is the **rebuild path**: the load the system of record takes while the tier is absent or refilling. 4. If the source cannot take that rate, the failure moves rather than disappearing. The tier's outage becomes the source's outage, and requests that never touched the tier start failing too. ## The facts that decide degrade or fail - **The role.** Derived copy: degradation is possible. Sole home: the fact is gone, and there is nothing to degrade to. - **The route.** Is the read to the system of record actually present on that code path, or does the code assume an answer comes back? A design with no route is not degrading; it is failing more slowly. - **Headroom at the source.** The unfiltered rate has to be survivable, or degradation is just a different failure. A source sized in the presence of the tier has never served this rate. - **How the tier stopped answering.** The four modes below produce four different caller experiences. - **How many services share the tier.** A shared tier's bad minute is shared: every service on it degrades in the same minute, and each one's system of record takes its own unfiltered rate simultaneously. ## Stopping answering is not one behaviour | How it stops | What the caller sees | | --- | --- | | Unreachable | Nothing returns; the request waits, then fails, on every call that consults the tier | | Answering slowly | Every consulting request pays the delay and holds its resources while it does | | Emptied by a restart | Every read is a miss; answers stay correct while the full read load lands on the source | | At its memory ceiling | Depends on the store: some reclaim entries to accept writes, others refuse writes and keep serving reads | That last row is where confident answers most often go wrong. There is no single ceiling behaviour in this class of store, so a design that survives only one of them is half written. The same caution applies to restart: what a store keeps across one varies, and the caller cannot assume a populated tier afterwards. ## Why slow can be worse than absent A component that refuses immediately at least releases the caller. A component that answers eventually keeps a thread and a connection occupied on every request while it does, so the service's own capacity drains from the inside even though nothing has reported an outage. This is why "is it up?" is the wrong monitoring question for a tier on the hot path, and why an answer that treats the outage as binary misses the mode that actually takes services down. ## What a strong answer states explicitly Name the role, name the route, name who takes the load, and say what the user sees at the end: a slower page, a page missing a section, or an error. "It just reads the system of record instead" is only true if that branch exists in the code and the source can take the traffic; otherwise it is an assumption doing the work of a design. The candidate who has operated one of these volunteers the rebuild-path load without being asked, because that is the part that turns a tolerable outage into an incident.