In a jobs-for-you rail over four million postings, why retrieve a few hundred candidates before scoring?

## The arithmetic that forces two stages A "jobs for you" rail has a page budget measured in tens of milliseconds and a catalogue of roughly four million open postings. A scoring function worth having — one that fetches per-seeker and per-posting features and runs a model over the pair — costs on the order of ten microseconds per posting. Four million postings times ten microseconds is **forty seconds of CPU for one request**. Spread across forty cores that is still a second of wall clock, and the rail serves thousands of requests a second. No fleet size fixes this, because the cost is incurred per request, not once per day. The funnel splits the work where the two cost curves cross: - **Retrieval cost is roughly independent of catalogue size.** An approximate nearest-neighbour index over precomputed posting vectors inspects a small fraction of the corpus per query, so doubling the catalogue costs memory and very little latency. - **Scoring cost is linear in candidates.** Each candidate means a feature fetch and a forward pass, so the shortlist size *is* the serving bill. - The architecture therefore treats the shortlist size as a **fixed budget** — a few hundred — and makes retrieval's job to fill that budget with the most plausible postings it can find. ## What the retrieval stage is allowed to know Retrieval has to rank candidates it has not enumerated, which constrains what it can compute: - Its scoring function must be **precomputable and searchable**: a seeker vector produced on the request, posting vectors produced in batch, one similarity metric the index can search over. A signal the index cannot search over cannot influence retrieval. - Beside the learned source sit **cheap lookup sources** that need no model at all: postings created in the last few hours inside the seeker's commute radius, postings matching a saved alert. They are ordinary queries against ordinary stores. - Hard constraints that would waste the budget — an expired posting, one the seeker already applied to — are applied as filters around the search rather than as a learned signal. ## The two stages side by side | | retrieval stage | scoring stage | |---|---|---| | input size | the whole catalogue, millions | the shortlist, hundreds | | cost per candidate | a fraction of a microsecond, amortised over the index | tens of microseconds plus a feature fetch | | what it may use | precomputed vectors and cheap lookups | features fetched per candidate | | output | a set of eligible candidates | an ordered list | | its failure mode | a good posting never enters the set | a good posting is placed too low | ## Recall is capped at stage one The funnel is strictly narrowing, and that is the consequence candidates most often miss: 1. Retrieval returns a shortlist. 2. The scorer permutes that shortlist; it is a ranking function over a given set, not a search over the catalogue. 3. Whatever edits the final slate operates inside the scored list. Nothing in that chain introduces a posting the first stage did not supply. Two practical consequences follow. First, **the ordering produced by retrieval barely matters** — membership is everything, because the scorer will re-order anyway. Second, **the retrieval stage's own quality metric is recall of the shortlist**, not any notion of precision: it is measured by how many of the postings that deserved a slot were present, not by how tidy its own ranking was. ## What an interviewer is listening for - The **cost asymmetry stated with a number**, not as "it would be slow". - That retrieval emits a **set under a size budget**, and that the budget is chosen deliberately. - That a miss at stage one is **unrecoverable within that request**, which is why retrieval is measured on recall. - That several cheap sources can feed the same shortlist; the learned index is not the only way in. The answer that fails is "score everything, just add machines": it misreads a per-request cost as a capacity problem, and it scales with catalogue size times traffic, which is exactly the product the funnel exists to avoid.