How do you measure a model's on-device latency so another team can trust the number?

## A latency number is a claim about conditions "The model runs in 12 ms" is not a fact until you say under what. A trustworthy report names, at minimum: - **Device and state** — which hardware, which compute unit, plugged in or on battery, what else was running, whether performance governors were pinned. - **Input shape and precision** — resolution or sequence length, and what numeric format weights and activations run in. - **Batch size** — and therefore whether the number is a per-request latency or a per-item throughput figure. - **Scope** — model compute only, or the end-to-end path including decoding the input, resizing, normalization and post-processing. Products pay for the end-to-end path; benchmarks often quote only the middle. - **Cold or warm** — first request after load, or steady state. Omitting these is how two teams end up with numbers that differ by 3x and both are right. ## Warm-up The first iterations are systematically unrepresentative: one-time initialization, memory allocated on first touch, caches cold, and processors sitting at low clocks until demand ramps them up. Standard practice is to run a set of warm-up iterations and discard them, then time the rest. If you also care about the **cold-start** case — an app that loads a model and immediately runs one inference — measure that separately and report it as its own number. It is a real user experience, but it is a different number, and averaging it into steady-state timings corrupts both. ## Sustained load and thermal throttling On phones and small edge devices, a benchmark that runs for ten seconds measures the device at its best and tells you nothing about the product. Under continued load the device heats, clocks are reduced to stay inside a thermal envelope, and latency degrades — often materially, after a minute or two of sustained work. If your feature runs continuously (a camera pipeline, a live filter), the steady-state throttled number *is* the number, and the peak figure is marketing. Report both, and say how long the device ran before the steady-state figure was taken. Ambient temperature and case material matter enough that the same phone gives different answers on different days. ## Percentiles, not a mean Measure many iterations and report a distribution. The **median** describes the typical request. The **95th percentile** describes what a meaningful fraction of users experience and is what latency budgets are usually written against. A **mean** is dragged around by a few slow samples and describes nobody; a **minimum** or "best of N" describes a condition no user will see. Reporting the number of iterations and the spread lets a reader judge whether your measurement is stable at all. ## Latency and throughput are different deliverables At batch size one, per-layer fixed costs dominate and hardware sits partly idle; at batch 64, the same hardware runs closer to its efficient regime and per-item cost falls. Crucially, the **ranking can flip**: model A can win at batch one and lose at batch 64, because the properties that make a model good at small batch — few operations, shallow graphs — are not the ones that make it good when there is abundant parallel work. So a single number cannot serve both an interactive request path and a bulk offline job. Decide which one the product needs, measure that regime, and label it. If someone quotes items-per-second at large batch to answer a question about how long one user waits, that is an answer to a different question. ## A workable protocol Pin the device state and record it. Load the model, run warm-up iterations, discard them. Run several hundred timed iterations at the target batch size and shape. Continue under load for a few minutes and record the steady-state distribution separately from the early one. Report median and p95 for each phase, plus the conditions above. Re-measure whenever the device, the runtime, the precision or the input shape changes — none of those numbers transfer. ## What a strong answer sounds like Lead with "a latency number needs its conditions", then give the mechanics — warm-up, many iterations, sustained load for throttling, median and p95 — and close by separating single-request latency from throughput. Candidates who have actually shipped to devices reach for throttling and percentiles unprompted; candidates who have not tend to describe timing a single call.