Load doubles and the personalised scorer cannot serve every keystroke inside the strip's deadline — which requests do you shed to the frequency list, and why those?
answer
- shed early, not at the deadline
- concurrency equals rate times service time
- shed where the rungs differ least
- long prefixes need little personalisation
- stable per session, not per keystroke
basics
~20 sShed the keystrokes where the rungs differ least — long, nearly unambiguous prefixes — and shed before the work queues rather than after the deadline passes. Pick the shed set deterministically per session so the strip does not flip rungs mid-sentence.
solid answer
~50 sThis is deliberate degradation, not failure-driven fallback, so the decision belongs at admission. The scorer's sustainable rate is fixed by its concurrency and its service time: with 64 concurrent slots and 8 ms per keystroke it clears about 8,000 keystrokes per second, so at 12,000 arriving, roughly 4,000 per second must go to the cheap rung or the queue grows and every request pays the wait before missing its deadline anyway. Which 4,000 is a quality question, not a random one: send the keystrokes where the personalised scorer adds least, which are the long prefixes that leave few plausible words — the frequency list nearly matches it there. Keep the ambiguous short prefixes on the scorer. Decide the shed set from a stable hash of the session so one user stays on one rung for a sentence, and tag the reason as shed rather than failed.
code
pseudocode · 13 linescapacity = 64 // concurrent scorer slots
serviceTime = 0.008 // seconds to score one keystroke
sustainableRate = capacity / serviceTime // 8000 per second
on keystroke(session, prefix):
overloaded = (inFlight >= capacity) or (arrivalRate > sustainableRate)
if overloaded:
cheapIsClose = length(prefix) >= 4
inShedSlice = stableHash(session) mod 100 < shedPercent
if cheapIsClose or inShedSlice:
return frequencyList(prefix) tagged rung = "frequency_list",
reason = "shed_for_load"
return scorer(session, prefix) tagged rung = "personalised"go deeper
Know that under overload a system can choose to answer some requests from a cheaper source rather than making everyone wait for the expensive one.
Compute how much has to be shed from concurrency and service time, and explain why queueing the excess only converts it into late failures.
Choose the shed set by the quality gap between rungs, keep it stable per session, and tag shed responses with a reason distinct from failures.
Decide how much degradation is a legitimate substitute for capacity, and what the product accepts in exchange for not sizing the scorer for peak.
## Two kinds of degradation on the same ladder The rungs are the same but the trigger is not, and conflating them makes both harder to operate: - **Failure-driven fallback** fires after a request has already waited: the scorer was unreachable or did not answer in time. The user pays the wait and then gets the cheap answer. - **Deliberate shedding** fires at admission: the system knows it cannot serve everyone well, so it decides up front who gets the cheap answer, and they get it immediately. They need different reason tags, because afterwards the only way to distinguish a capacity problem from a dependency problem is the tag. ## How much has to go The scorer's sustainable arrival rate follows from concurrency and service time. **Little's Law** states that concurrency equals arrival rate multiplied by service time, so the rate a fixed pool can sustain is its concurrency divided by the per-request service time: - 64 concurrent slots, 8 ms per scored keystroke: 64 / 0.008 s = **8,000 keystrokes per second** - arriving: **12,000 per second** - therefore **4,000 per second** must be served from a cheaper rung Admitting all 12,000 does not make the scorer faster. Work piles up, queueing delay grows without bound while the overload lasts, and the extra requests miss the deadline *after* consuming the wait. Adding queue depth in front of the scorer does not buy time; it buys latency. Shedding converts a late, empty strip into an immediate, less personal one. ## Choosing which keystrokes, by the gap between rungs The interesting decision is not how many but which. Shed where the two rungs differ least, so the traffic that loses personalisation is the traffic that needed it least: - A **long prefix** — the user has typed most of the word — leaves few plausible completions, and a global frequency table keyed on that prefix is close to what the personalised scorer would have said. Cheap to shed. - A **short prefix or a word boundary** is where intent is widest and personal history matters most. This is exactly the traffic to protect. - **Repeat prefixes within a session** may already have a cached prediction, which is a better shed target than the frequency list because it keeps personalisation. | shed candidate | quality gap to the cheap rung | verdict | |---|---|---| | long, nearly complete prefix | small — few words remain plausible | shed first | | prefix already seen this session | small if a cached entry exists | shed to the cache, not the list | | first keystroke of a word | large — intent is widest here | protect longest | ## Keep the choice stable Shedding at random on each keystroke spreads the degradation thinly over everyone, which sounds fair and behaves badly. The strip alternates between a personalised suggestion and a generic one inside a single word, which reads as flicker, and every user's experience becomes a blend, so per-rung acceptance is measured on a mixture rather than a population. A stable hash of the session, compared against the shed percentage, keeps a user on one rung for the sentence and keeps the measurement interpretable. ## Why the cheap rung can absorb it The frequency-list rung is a lookup against a static table: its cost per keystroke barely depends on how much traffic it takes, which is what makes it a valid shed target for an arbitrary fraction of the load. A rung that is itself a service with its own scoring cost is a much weaker target, because the fraction you divert is exactly the fraction it was not sized for. ## Coming back Shedding should be released on the same signal that raised it — measured arrival rate and in-flight work against the sustainable rate — and released gradually. Dropping the shed percentage to zero the moment in-flight work dips re-admits the full arrival rate into a pool that has not drained, and the system oscillates between shedding and queueing instead of settling.
- Why decide the shed set from a stable hash of the session rather than at random per keystroke?Random per-keystroke shedding makes one user's strip alternate between a personalised and a generic suggestion inside a single word, which reads as flicker. It also mixes both rungs into every user's experience, so per-rung acceptance measures a blend rather than a population. A stable hash keeps each session on one rung and keeps the measurement interpretable.
- How is shedding different from the fallback that fires when the scorer times out?Same ladder, different trigger and different timing. Failure-driven fallback happens after the request has already spent its wait, so the cheap answer arrives late. Shedding decides before the work is admitted, so it arrives immediately. They also need separate reason tags, or afterwards you cannot tell a capacity problem from a dependency problem.
saying these in an interview costs you the question
- Lets requests queue and relies on the deadline to shed them
- Sheds uniformly at random on every keystroke
- Assumes shedding is only for failures, not for load
- Sheds the ambiguous short prefixes the scorer helps most
- Thinks adding queue depth buys time instead of latency