A dispatch request scores 40 nearby drivers on 30 online features each, issuing one read per driver per feature — why does its p99 collapse?

seniorshould knowfreq 51%

answer

count the reads per request
the request waits for the slowest
1,200 draws from one tail
one row per entity, not per value
41 keys in one multi-get

basics

~20 s

That shape issues 1,200 reads per request, and the request cannot finish until the slowest one returns. With a 1% per-read tail, almost every request contains at least one slow read, so request latency tracks the read tier's extreme tail rather than its median.

solid answer

~50 s

Forty candidates times thirty features is 1,200 online reads for one dispatch decision. Run them sequentially at a millisecond each and the fetch alone is 1.2 seconds. Run them in parallel and the request still waits for the maximum of 1,200 draws: if each read independently has a 1% chance of exceeding its p99, the chance that all 1,200 come back fast is 0.99^1200, about six in a million. In other words the request's typical latency sits near the read tier's p99.99, not its p50. The fix is to change the read count, not the reads: co-locate all of a driver's features in one row keyed by driver, read the shared zone-level row once, and issue the 41 resulting keys as a single multi-get. One round trip, and roughly two requests in three now touch no slow key at all.

Forty candidates times thirty features is 1,200 online reads for one dispatch decision. Run them sequentially at a millisecond each and the fetch alone is 1.2 seconds. Run them in parallel and the request still waits for the **maximum** of 1,200 draws: if each read independently has a 1% chance of exceeding its p99, the chance that all 1,200 come back fast is `0.99^1200`, about six in a million. In other words the request's typical latency sits near the read tier's p99.99, not its p50. The fix is to change the read count, not the reads: co-locate all of a driver's features in one row keyed by driver, read the shared zone-level row once, and issue the 41 resulting keys as a single multi-get. One round trip, and roughly two requests in three now touch no slow key at all.

code

pseudocode · 11 lines

pseudocode

keys = []
for each driver in candidates:
    keys.append("driver:" + driver.id)
keys.append("zone:" + request.zone_id)

rows = online_store.multi_get(keys)

zone_row = rows["zone:" + request.zone_id]
features = []
for each driver in candidates:
    features.append(merge(rows["driver:" + driver.id], zone_row))

go deeper

for a junior

Recall that a request cannot score until its last feature read returns, so the number of reads it issues matters more than how fast a single read looks on a dashboard.

for a middle

Explain the mechanics of fan-out: many independent draws from the same latency distribution, and the request pays the maximum. Then describe the fix — one row per entity, shared keys read once, one batched round trip.

for a senior

Do the arithmetic out loud and connect it to capacity: keys per request, the probability that none is slow, and what a larger shortlist costs the feature tier. Diagnose by counting keys before touching timeouts or replicas.

for a principal

The judgment call is where the shortlist size is owned. Ranking wants more candidates, the feature tier pays for them linearly, and somebody has to hold the budget that decides which side gives.

Count the reads before you tune them

The arithmetic is the answer. A dispatch request shortlists 40 candidate drivers and the ranking model consumes 30 features per candidate. One read per driver-feature pair is 1,200 reads for one dispatch decision, at a rate of thousands of decisions per second.

Sequentially, at 1 ms per read, the feature fetch alone takes 1.2 s. The request budget is gone before the model runs.
In parallel, the wall-clock cost is not the average read but the slowest read, because the model cannot score until its last input arrives.

Tail amplification, worked

Treat the reads as independent, each with a 1% chance of landing at or above its own p99.

The probability that all 1,200 reads come back under the p99 is 0.99^1200, roughly 6 in a million.
So essentially every request contains at least one read in the tail, and the request's latency distribution tracks the maximum of 1,200 draws — in practice the read tier's p99.9 to p99.99 region, not its median.
This is why a tier that measures beautifully in isolation produces an ugly request p99: the service-level number you actually experience is the per-read tail raised to the fan-out.

Raising per-read timeouts does not help; it widens the tail you are drawing from. Adding replicas does not help either, unless the slowness was queueing from saturation: it lowers the tail a little, but the exponent is still 1,200.

Change the exponent

The lever is the number of keys a request touches.

Co-locate by entity. Store all of a driver's features as fields of one row keyed by driver identity, so thirty values arrive in one lookup. The key count falls from 1,200 to 40.
Deduplicate shared keys. The zone-level features — open requests in this zone, current surge state — are the same for all forty candidates. Read that row once, not forty times, which adds one key rather than forty.
Batch into one round trip. Issue the 41 keys as a single multi-get rather than 41 calls. That removes 40 round trips' worth of connection and serialization overhead on top of the tail benefit.

After the change: 0.99^41 is about 0.66, so roughly a third of requests still touch one slow key — a real number to state, and far from the six-in-a-million before.

	per-value reads	per-entity rows
keys per request	1,200	41
round trips	up to 1,200	1
P(no key in the tail)	~0.000006	~0.66
what a bigger shortlist costs	30 keys per driver	1 key per driver

The capacity number this hands you

Key count scales with the shortlist, so the shortlist is a feature-fetch capacity decision, not only a ranking one. Growing the candidate set from 40 to 200 takes the request from 41 keys to 201, and multiplies the tier's key throughput by roughly five at the same request rate. In a design round, say that out loud: the ranking team's proposal to consider more candidates lands on the feature tier's bill and on the request's tail before it lands on ranking quality.

What to check in a real system

Measure keys per request, not just read latency; the first number explains the second.
Watch the multi-get's own tail as the key count grows — a batch is one round trip but still touches many partitions.
Confirm the shared zone row is genuinely read once; a loop that fetches it per candidate is the easiest forty-fold waste to leave in place.

## Count the reads before you tune them The arithmetic is the answer. A dispatch request shortlists **40** candidate drivers and the ranking model consumes **30** features per candidate. One read per driver-feature pair is **1,200 reads for one dispatch decision**, at a rate of thousands of decisions per second. - **Sequentially**, at 1 ms per read, the feature fetch alone takes 1.2 s. The request budget is gone before the model runs. - **In parallel**, the wall-clock cost is not the average read but the **slowest** read, because the model cannot score until its last input arrives. ## Tail amplification, worked Treat the reads as independent, each with a 1% chance of landing at or above its own p99. - The probability that **all** 1,200 reads come back under the p99 is `0.99^1200`, roughly **6 in a million**. - So essentially **every** request contains at least one read in the tail, and the request's latency distribution tracks the maximum of 1,200 draws — in practice the read tier's p99.9 to p99.99 region, not its median. - This is why a tier that measures beautifully in isolation produces an ugly request p99: the service-level number you actually experience is the per-read tail raised to the fan-out. Raising per-read timeouts does not help; it widens the tail you are drawing from. Adding replicas does not help either, unless the slowness was queueing from saturation: it lowers the tail a little, but the exponent is still 1,200. ## Change the exponent The lever is the number of keys a request touches. 1. **Co-locate by entity.** Store all of a driver's features as fields of one row keyed by driver identity, so thirty values arrive in one lookup. The key count falls from 1,200 to 40. 2. **Deduplicate shared keys.** The zone-level features — open requests in this zone, current surge state — are the same for all forty candidates. Read that row **once**, not forty times, which adds one key rather than forty. 3. **Batch into one round trip.** Issue the 41 keys as a single multi-get rather than 41 calls. That removes 40 round trips' worth of connection and serialization overhead on top of the tail benefit. After the change: `0.99^41` is about 0.66, so roughly a third of requests still touch one slow key — a real number to state, and far from the six-in-a-million before. | | per-value reads | per-entity rows | |---|---|---| | keys per request | 1,200 | 41 | | round trips | up to 1,200 | 1 | | P(no key in the tail) | ~0.000006 | ~0.66 | | what a bigger shortlist costs | 30 keys per driver | 1 key per driver | ## The capacity number this hands you Key count scales with the shortlist, so the shortlist is a **feature-fetch capacity decision**, not only a ranking one. Growing the candidate set from 40 to 200 takes the request from 41 keys to 201, and multiplies the tier's key throughput by roughly five at the same request rate. In a design round, say that out loud: the ranking team's proposal to consider more candidates lands on the feature tier's bill and on the request's tail before it lands on ranking quality. ## What to check in a real system - Measure **keys per request**, not just read latency; the first number explains the second. - Watch the **multi-get's own tail** as the key count grows — a batch is one round trip but still touches many partitions. - Confirm the shared zone row is genuinely read once; a loop that fetches it per candidate is the easiest forty-fold waste to leave in place.

What happens to the key count when the candidate shortlist grows from 40 drivers to 200?
It grows linearly: 201 keys per request instead of 41, and the tier's key throughput rises about fivefold at the same request rate. The batch's own tail widens too, since more keys means more partitions touched. Shortlist size is therefore a feature-fetch capacity decision as much as a ranking one.
Why does raising the per-read timeout not rescue the original 1,200-read shape?
The request waits for the slowest read regardless of where the timeout sits. A higher timeout only lets that slowest draw run longer; a lower one converts it into a missing feature. Neither changes the number of draws, which is what put the request in the extreme tail.

saying these in an interview costs you the question

Parallel reads make the request as fast as the median read.
More replicas on the key-value tier fix a fan-out tail.
Raise the per-read timeout until the slow reads fit.
One read per feature per candidate is fine at this scale.
The forward pass, not the feature fetch, owns this budget.

A dispatch request scores 40 nearby drivers on 30 online features each, issuing one read per driver per feature — why does its p99 collapse?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

Count the reads before you tune them

Tail amplification, worked

Change the exponent

The capacity number this hands you

What to check in a real system

▾common follow-ups2 questions

▾red flags5 flags

Count the reads before you tune them

Tail amplification, worked

Change the exponent

The capacity number this hands you

What to check in a real system

answer

go deeper