A feed ranker trains on the clips its own funnel showed - what does that training set never contain, and what does the gap cost?

The second-stage scorer of a short-video feed is trained on logs from the feed it ranks. That sounds obviously right and quietly biases everything, which is why interviewers reach for it. ## Three populations, one of them labelled For a single request the catalogue splits into three groups: 1. **Never retrieved.** No stage saw the clip. No score, no row, no outcome. 2. **Retrieved and scored, never shown.** The clip was in the shortlist, the heavy scorer produced a number, and the slate stopped above it. The viewer made no decision about it. 3. **Shown.** The viewer had a chance to finish, like or share, so an outcome exists. Only group three carries a label. Group two is the one that causes trouble, because a row for it exists somewhere in the system and it is tempting to file it as a negative. ## Unobserved is not negative A negative example asserts a fact: the viewer had the chance and declined. A clip that was never rendered supports no such assertion. Writing it down as a negative injects a claim the data does not contain, and the claim has a direction - it says that whatever the current ranker placed low is bad. Train on that and the model is rewarded for agreeing with its predecessor, which is the one thing that produces no improvement at all. The cost shows up as a mismatch between what the model saw and what it is asked to do: - At training time it sees mostly clips the funnel already liked, so its evidence is dense in the region it already handles well. - At serving time it must score every clip on the shortlist, including many whose kind it has almost never observed with a real outcome. - Confidence is therefore highest exactly where it matters least, and the scores that decide marginal clips rest on the least evidence. There is a second distortion inside group three. An outcome on the top slot is partly an outcome about the top slot: the same clip is finished at very different rates at the top of a feed and ten positions down. Unless position is recorded and handled, the model learns that whatever was shown first is good. ## The logging that makes the gap fixable Every remedy here is an architecture decision made before any model is trained, and it is what the interviewer is checking for. | record this | why it matters | |---|---| | the whole shortlist, not just impressions | separates never-retrieved from scored-but-unshown | | each clip's stage scores | shows what the funnel believed about the clips it dropped | | the slot the clip would have occupied | lets the model separate the clip from its position | | feature values as read at scoring time | the row matches what the scorer saw, not a later recomputation | | the exposure probability of the slate | supports weighting rows by how likely they were to be seen | The fourth row deserves emphasis. Recomputing a feature from the offline store after the fact produces a value the scorer never saw - the counters have moved on - so the training row describes a decision nobody made. Logging the value at the moment of scoring is the only way the row is true. ## The one deliberate cost The structural remedy is to spend a little traffic on honesty: a small randomised fraction of requests whose slate is not strictly score-ordered produces outcomes for clips the ranker would have buried. Those rows are the only unbiased evidence the funnel generates about its own margins, and the cost is a slightly worse slate for that fraction of traffic. How much to spend is a business decision; that some is spent is what makes the next model better than the current one rather than a copy of it. ## The compact answer Say it as a chain: the log is the funnel's output, not a sample; unshown is unobserved, not negative; the model therefore knows most about what the funnel already agreed on and least about what it must decide; and the fixes are logging decisions - full shortlist, stage scores, slot, serving-time feature values - plus a small deliberate randomisation. A candidate who describes the model instead of the logging has answered a different question.