Retraining Loops & Feedback

## The two roles A query autocomplete suggester turns a half-typed prefix into a short list of completions. It is refit on a cadence — weekly, say — because the query log it learns from moves: new product names, new spellings, a news event that invents a phrase overnight. Each run of the retraining pipeline produces an **artifact**: a trained model plus the data snapshot, feature definitions and configuration it was built from. That artifact is the **challenger**. The model answering live prefix requests right now is the **champion**. Champion-challenger promotion is the rule that decides, with no human in the room, whether this week's challenger takes the champion's place. The load-bearing word is *replaces*. There is exactly one live suggester behind the search box, and a model registry records which version that is. Promotion is not "ship the new model" — it is "move the registry's live pointer, and keep the ability to move it back". ## Why newer data does not mean a better model Retraining is an industrial process whose output nobody reads. Plenty of ordinary things make a refresh worse than the model it would replace, and none of them fail the pipeline: - **A skewed input week** — an outage, a bot wave or a campaign distorts the query distribution the candidate learned from. - **A silent upstream break** — a logging change stops populating a field, and the training job succeeds on a hole. - **A shorter memory** — a refresh weighted to recent traffic forgets rare but real prefixes the champion still handles. - **A non-quality regression** — the candidate is larger and slower, so suggestions land after the user has finished typing. No error is raised; the box is just worse. - **Plain run-to-run variance** — two refits of the same pipeline are not identical, and one of them is a little worse. Each of these produces a green pipeline run and a bad artifact. That is precisely the case a promotion gate exists to catch. ## The champion is the baseline, not an absolute bar It is tempting to gate on a fixed number: promote anything scoring above some threshold. That bar is hard to justify in the abstract and it drifts out of date, because the only change promotion makes in the world is *this model instead of that one*. Comparing candidate against champion on identical data also cancels much of what the two have in common — the same prefixes, the same seasonality, the same catalogue — so what remains is closer to the difference the swap would actually cause. The cost of a relative rule is worth stating plainly: it never asserts that the champion is good. If the world moved and the live suggester is now poor, a gate demanding "better than the champion" will happily promote a slightly-less-poor challenger and say nothing about the underlying decay. The quality of the live model is a monitoring question; promotion is a comparison question. ## What the standing rule has to state | element | what it fixes | for a suggester | |---|---|---| | the comparison set | both models scored on identical sessions | a frozen window of logged sessions, held out of training | | the promotion metric | what "better" counts as | share of replayed sessions whose submitted query appeared in the suggestions | | the margin | how far ahead is far enough | a stated improvement over the champion on that same window | | the veto conditions | what disqualifies regardless of the metric | suggest latency, empty-suggestion rate, error rate on live prefixes | | the pointer | what promotion actually changes | the model registry's live version for this model | | the reversal | what happens when it was wrong | an automatic model-version rollback | ## The loop, end to end 1. The scheduled refresh runs and registers its artifact as a challenger; the registry's live pointer still names the champion. 2. The gate replays the frozen window for both models and compares the two scores against the stated margin. 3. A candidate that clears it is exercised against live prefix traffic with its output discarded, to show it runs inside its latency and error budget. 4. If nothing vetoes it, the live pointer moves to the challenger, which becomes the champion. 5. Live guardrails keep watching; a regression moves the pointer back automatically and quarantines that version. ## Where a standing gate differs from a one-off launch A new suggester architecture, hand-built once a year, gets a human: someone designs the comparison, watches the traffic and decides. The standing gate is the opposite case — it reaches the same kind of decision every week with nobody watching, which is why every part of it is configuration rather than judgment. The price of that automation is that anything the human would have noticed has to be written down in advance, and anything not written down is not checked. ## What an interviewer is listening for The weak answer treats retraining as deployment: "the pipeline runs weekly and pushes the new model". The answer that lands names three moving parts — a candidate that is not live, a comparison against the model that is, and a pointer that can move both ways — and then admits the limit: the gate cannot tell you the champion is good, only whether the challenger is better.