When a new photo-tagging scorer ships, what does a precomputed tag store owe that a live scoring path does not?
answer
- a stored row is a past decision
- deploy changes the scorer, not the rows
- corpus size divided by backfill rate
- eager finishes, lazy never does
- rollback needs a second pass
basics
~20 sIt owes a rescore of every stored row. A precomputed prediction is a copy of a decision an older scorer made, so a deployment does not change it; a live path produces the new output on the very next read.
solid answer
~40 sDeploying a scorer changes what the scorer will say, not what was already written. A live path therefore costs nothing on a model change — the next read uses the new version. A precomputed store carries 400 million rows produced by the old one, and they stay wrong-by-version until something rewrites them. At 5,000 photos per second of spare batch throughput that backfill runs about 22 hours, during which a single album can legitimately show a mix of old and new tags. Two strategies exist: an eager full pass, which ends the mixed window quickly but pays for rows nobody reads; or lazy rescore-on-read, where a row stamped with an old version is recomputed when it is next requested, which is free for the unread 80% but leaves search answering from old-version tags indefinitely.
go deeper
Know that a precomputed prediction was made by the model that was live at the time, so shipping a new model does not change what is already stored.
Divide corpus size by backfill throughput and state the window in hours. Explain why a live path owes nothing on a deploy and why the stored version has to be on the row.
Choose between an eager pass and lazy rescore-on-read from the read shapes, plan the mixed-version window, and design the pass to be restartable and rate-limited against the serving fleet.
Price the placement over the model's whole life, not per request: at a weekly retrain a corpus pass is a permanent background job, and that recurring bill may be the strongest argument against precomputing at all.
## A stored prediction is a copy of a decision, not a live view This is the asymmetry the question is about. A request-time scorer holds no state: change the deployed version and every subsequent read gets the new behaviour automatically. A precomputed store holds the *output* of a scorer that ran at some point in the past. Deploying a new one changes nothing that is already written. Four hundred million rows keep the old model's opinion about what is in each photo until a job rewrites them, and no amount of care in the deployment changes that. The practical consequence is that choosing precompute is choosing to make every future model change a corpus-sized job. That cost is invisible in a per-request comparison and is often the largest line in the placement's real bill. ## What each placement owes on a model change | | precomputed tag store | live scoring path | |---|---|---| | work on deploy | rescore every stored row | none | | when the new version is visible | when the backfill reaches the row | on the next read | | mixed-version window | exists, as long as the backfill runs | none | | rollback to the old version | needs the previous rows, or another pass | redeploy the old scorer | | cost of the change | proportional to the corpus | zero extra, the reads already pay | ## Eager backfill: do the arithmetic before you promise a date An eager backfill is a full pass over the corpus at whatever throughput the batch capacity allows: ``` 400,000,000 rows / 5,000 rows per second = 80,000 seconds ~= 22 hours ``` Twenty-two hours is a real operational window, not a deployment step, and it has to be planned as one: 1. **Throughput is what you bought, not what you hoped.** If the spare capacity is 2,000 per second, the same corpus takes about 55 hours. 2. **The pass competes with the serving fleet** if they share capacity, so the backfill rate and the read path's latency are the same budget viewed twice. 3. **It must be restartable.** A pass that has to begin again from zero after a failure at hour 20 turns a day into three. ## Lazy backfill: rescore what somebody actually asks for The alternative is to stamp every row with the scorer version that produced it and rescore on read: when a request finds a row whose version is not current, it scores the photo live, writes the fresh row and returns it. Its properties are the mirror image of eager: - **It costs nothing for the 80% of rows nobody reads**, which is the bulk of the corpus. - **It puts the rescore inside a user-facing read**, so the live tier absorbs a burst of work in the hours after a deployment. - **It never finishes.** Rows nobody opens keep the old version indefinitely, which is fine for album views and not fine for a tag search that ranges over the whole library — search would answer half from each version forever. That last point is usually what decides it: if a corpus-wide read exists, some eager pass has to complete. ## The mixed-version window is a real product state While any backfill runs, two versions of the tags coexist in the store, and a single album can show some photos tagged by the old scorer and some by the new. This is normal and usually acceptable — the tags are per-photo and nobody compares them across an album. It stops being acceptable when the output is *relative*: a ranked list, a threshold applied consistently across a set, or anything a user would read as a comparison between photos. In those cases the pass has to write into a separate namespace and the read path has to switch over atomically once it is complete, at the cost of holding both copies. ## What this means for the placement decision - **Stamp the version on the row.** Without it you cannot tell a current row from a superseded one, cannot drive a lazy rescore and cannot measure how far a backfill has got. - **Count the backfill in the placement's cost**, once per model version, not once ever. - **Model cadence and placement interact.** A model retrained weekly makes a 22-hour corpus pass a permanent background job; a model that changes twice a year makes it an event. - **A rollback is a second backfill.** Reverting the scorer does not revert the rows, so plan the reverse direction before shipping the forward one. - **A hybrid is usually the answer:** an eager pass for the keys a corpus-wide read touches, lazy rescore-on-read for the long tail nobody opens.
- How do you roll back to the previous scorer version once the backfill is half done?Only by rewriting the rows again, unless you kept the previous ones. Redeploying the old scorer fixes nothing already written, so either the backfill writes into a separate namespace the read path can switch away from, or the rollback is a second pass of the same length as the first.
- The scorer is retrained weekly. What does that do to the placement?It makes a corpus-sized rescore a standing background job rather than an occasional event, and if a full pass takes 22 hours the store is never entirely one version. At that cadence, either the backfill is limited to the keys a corpus-wide read touches, or the placement itself deserves re-examination.
- Is a mixed-version tag store ever unacceptable?When the output is read as a comparison between photos rather than a fact about one — a ranked set, or a threshold meant to apply uniformly. Per-photo tags survive a mixed window fine; anything relative needs a separate namespace and an atomic switch once the pass completes.
A printed catalogue has to be reprinted when a price changes; a clerk quoting the price on the phone is correct the moment the price is updated. The catalogue is faster to read and more expensive to be right.
saying these in an interview costs you the question
- Assumes a new scorer version reaches already-stored rows automatically
- Says a live scoring path also needs a backfill after a model change
- Believes the backfill can run at serving throughput at no cost
- Ignores that both versions coexist in the store during the pass
- Treats stamping the scorer version on a row as optional detail
- Plans the forward backfill without planning the reverse one