A policy classifier costs 600 accelerator-hours per quarterly retrain and scores 1.9 billion clips at 40 ms each — which half dominates?

## Put both halves in one unit The halves are quoted in incompatible units — a training run in hours, an inference in milliseconds — and that mismatch is exactly why people guess the answer wrong. The fix is mechanical: express both as **accelerator-hours over the same interval**, which here is one retrain interval, a quarter. 1. **Serving.** Clips in the interval times per-clip accelerator time. About 1.94 billion clips (250 per second averaged over a 90-day quarter) at `0.040 s` each is `77,760,000 s`, or **21,600 accelerator-hours**. 2. **Training.** One run, given as **600 accelerator-hours**. 3. **Compare.** `21,600 / 600 = 36`. The serving half is thirty-six times the training half. ## The crossover, not the ratio The ratio 36 is specific to this quarter's traffic. The reusable number is the **crossover clip count** — the predictions per retrain interval at which the halves are equal: `crossover = training_hours x 3600 / service_seconds_per_clip = 600 x 3600 / 0.040 = 54,000,000 clips` That single figure answers a family of questions without redoing the arithmetic. Fifty-four million clips per quarter is the break-even; anything above it is serving-dominated, and the further above, the more decisively. | Clips per retrain interval | Serving hours | Dominant half | |---|---|---| | 5 million | 56 | training, heavily | | 54 million | 600 | equal — the crossover | | 540 million | 6,000 | serving, 10x | | 1.94 billion | 21,600 | serving, 36x | ## What actually moves the crossover Only two inputs appear in the formula, so only two things move it: - **The retrain's accelerator-hours** move it *proportionally*. A run twice as long pushes the crossover to 108 million clips — the fixed half now needs twice the traffic before it is outweighed. - **Per-clip accelerator time** moves it *inversely*. Cutting 40 ms to 20 ms doubles the crossover to 108 million; raising it to 80 ms halves the crossover to 27 million. And two things that feel relevant but are not: - **Upload volume** does not move the crossover. It moves *where you sit* relative to it, which is a different statement. - **Retraining cadence** does not move it either. Halving the interval halves both the clips in that interval and the number of intervals, so the break-even *per interval* is unchanged; what changes is that you cross it later in each cycle. ## Reading the result honestly Three refinements stop the number from being quoted too confidently: - **Failed runs and sweeps count.** The 600 hours is the winning run. If a release typically burns three abandoned runs and a hyperparameter sweep, the honest fixed half might be two or three times the headline figure — which moves the ratio from 36x to roughly 12x, and still does not change the verdict. - **Accelerator time is not machine time.** A clip that occupies 40 ms of accelerator work may occupy far more wall-clock time on a machine that is also decoding video, fetching features and waiting on the network. Compare like with like, or state which one you are counting. - **You pay for provisioned hours, not consumed ones.** A fleet held at peak size bills through the trough, so the *paid* serving hours exceed the 21,600 consumed. That correction pushes the ratio further in serving's favour, never back toward training. ## Why an interviewer asks this The question is not arithmetic for its own sake. It checks three things at once: - that you **convert to a common unit** before comparing, rather than arguing from the feel of the numbers; - that you know **which lever belongs to which half**, so you do not propose a change to the fixed half as a fix for a variable-half problem; - that you can state the **ceiling on a saving** before doing the work. With training at 600 of roughly 22,200 accelerator-hours, even deleting the training run entirely saves under 3%. That is the whole argument for pointing the next optimisation at the per-clip path instead. The answer an interviewer wants is therefore two sentences and a number: serving dominates by about 36x, the crossover is 54 million clips per retrain interval, and here is the division that produced both.