Overload Protection & Load Shedding

## The selection problem, not the volume problem When a service is over capacity, the amount it must reject is fixed by arithmetic — offered load minus capacity. The only free variable is *which* requests get rejected, and that variable is worth a great deal. A random 10% shed is a 10% failure rate for payments, a 10% failure rate for login, and a 10% failure rate for a background image prefetch. Nobody would choose that distribution if asked. Priority-based load shedding is simply the mechanism that lets the system make the choice a human would make, at machine speed, without anyone being paged first. ## What criticality is, and where it comes from Criticality is a property of the *request*, assigned where its business meaning is still visible. That is almost never inside the service being protected — a storage node cannot tell from an RPC whether the row it is fetching backs a checkout page or a nightly report. So the label is stamped at the origin: the API gateway or front-end handler that knows this is an interactive checkout; the scheduler that knows this is a batch backfill; the cache-warming job that knows its work is speculative. Google's SRE book (2016) documents four values used in its internal RPC system as an example: `CRITICAL_PLUS` for the most important user-facing traffic where failure is directly visible, `CRITICAL` as the default for user-facing requests, `SHEDDABLE_PLUS` for work with partial-unavailability tolerance such as batch jobs that can be retried later, and `SHEDDABLE` for work that is expected to fail routinely. The specific names are not an industry standard; what matters in an interview is the shape — a small number of ordered tiers, few enough that people can reason about them, and coarse enough that they map onto real product behaviour. ## Propagation is the part people forget A criticality label that stops at the edge protects only the edge. A single user-facing request typically fans out to a dozen internal services, and the ones nearest to a shared resource — the database, the cache tier, an authentication service — are the ones most likely to be the actual bottleneck. If the label does not travel in RPC metadata alongside the trace context and the deadline, those services shed uniformly and the shared bottleneck keeps spending its scarce capacity on sheddable work. Propagation also means the label must be *inherited*: any call a service makes on behalf of a request carries that request's criticality, and it may lower it but should never raise it. ## Retries and criticality A retry is the single most dangerous thing to mis-tier. If retried requests keep or gain priority, then a service under stress fills up with retries that displace fresh first attempts, and the failure sustains itself. The rule is that a retry is never more important than the original — many designs deliberately mark retried requests one tier lower, so that under pressure the system serves new work and sheds the amplification. ## Shedding against a local signal Each server decides independently, using its own saturation measure — queue wait, in-flight count, CPU pressure — and progressively raises the tier below which it refuses. At mild pressure it drops `SHEDDABLE`; as pressure worsens it drops `SHEDDABLE_PLUS`, and only under extreme stress does it touch user-facing traffic. This has to be against a measured local signal rather than a static request-per-second cap, because the cost of a request varies by an order of magnitude across endpoints and changes with every deploy. ```python # shed the numerically lowest-importance tier first as pressure rises if pressure >= 0.95: keep_tier = 0 # only the most critical elif pressure >= 0.85: keep_tier = 1 elif pressure >= 0.70: keep_tier = 2 else: keep_tier = 3 ``` ## The failure mode: tier inflation The predictable organisational failure is that every team labels its own traffic with the highest tier, because from inside any one team the traffic genuinely is important. Once that happens the tiers rank nothing and shedding is random again with extra machinery. The defences are enforced quotas — a service is allowed only so much of its traffic at each tier, verified from telemetry rather than from intent — plus a periodic audit of tier distribution, and a default that is *not* the top tier. Treat the criticality distribution as a metric you graph, not as a configuration you set once. ## The interview point The decision with a cost is what you are willing to sacrifice first, and having decided it *before* the incident. The strong answer names the label, insists it is stamped at the origin and propagated end to end, handles retries explicitly, and then admits the governance problem — because the mechanism is easy and keeping the tiers honest is the part that actually fails.