On a phone latency and app-size budget, do you quantize or ship a smaller model?

## Reframe the question "Quantize or shrink?" is a false choice — a smaller architecture can also be quantized, and most shipped mobile models are both. The real question is **which lever to pull first, given a budget and a team**, and that is a sequencing and cost-of-ownership decision rather than a technical one. ## Fix the budget before touching the model Three numbers should exist before any optimisation work starts: 1. **Latency**, expressed as a percentile on the *worst device you support*, not the median device and not the engineer's phone. A p95 on a four-year-old mid-range handset is a different target from a p50 on a flagship. 2. **Size**, as the delta to the installed app, including the runtime and kernel libraries — not just the model file. Store listings and cellular-download thresholds make this a product constraint, not an engineering preference. 3. **Accuracy floor**, expressed on a metric the product actually cares about, with a stated tolerance. Without these, optimisation has no stopping rule and teams grind on the model long after they stopped mattering. ## What each lever actually buys **Int8 quantization** shrinks weights about 4x versus float32 and lets the CPU do integer arithmetic with better memory behaviour. Post-training, it is one pipeline pass: no retraining, no architecture change, reversible if it disappoints. What it does *not* do is reduce the number of operations — a model with too many FLOPs is still too slow, just with cheaper ones — and it does little for peak activation memory in a static graph. **A smaller architecture** — fewer channels, lower input resolution, a mobile-oriented backbone, or distillation into a compact student — reduces FLOPs *and* activation memory. That is the only lever that helps a model which is compute-bound rather than memory-bound. The costs are real: a training cycle, a fresh hyperparameter search, and a genuine accuracy drop that no amount of numerics work recovers. **QAT** is not a size lever at all; it is a way to buy back accuracy you lost to quantization, at the price of a permanent training-pipeline commitment. A useful heuristic: if profiling shows the model is dominated by large weight matrices and memory traffic, quantization is your lever. If it is dominated by convolution FLOPs at high resolution, architecture (or input resolution) is your lever. Profile before choosing. ## Binary size is not just the model A principal-level answer includes the parts juniors forget. The shipped footprint is the model artifact plus the runtime plus every operator kernel linked in. Selective build matters here: the on-device runtime can link only the kernels a given model actually uses rather than a full kernel library, which is often a larger saving than a few megabytes of weights. Multiple delegates, multiple architectures, and float fallback paths all add up. Measure the installed-app delta on a real build, not the file size of the exported artifact. ## Device fleet reality One answer rarely fits the fleet. Low-end devices, which are where latency budgets fail, benefit most from integer arithmetic. High-end devices may have accelerators that prefer float16 or have their own supported numeric formats, so an int8 model tuned for CPU is not automatically the fastest artifact everywhere. That opens a strategy question: ship one artifact and accept it is suboptimal at both ends, or ship per-tier artifacts and pay for the extra build, test and support matrix. The single-artifact answer is usually right until the fleet data says otherwise, because every extra variant multiplies validation cost. ## Cost of ownership is the deciding factor The technical comparison usually ends in a tie; the operational one does not. A PTQ pipeline is a script anyone can rerun after a model update. A QAT pipeline requires training data access, labels, compute and a revalidation pass on every retrain — and if the person who built it leaves, it silently stops being rerun. A new architecture means the research team owns a second model line forever. So the judgment is: buy the cheapest lever that clears the budget, and stop. Reach for the expensive ones only with an explicit accounting of who maintains them and how often. ## Whatever you choose, gate it Optimisation without a gate regresses. The mechanism is a device-tier benchmark and an accuracy check that run in CI on the shipped artifact, with thresholds derived from the three budget numbers. That gate is what makes the decision durable — it turns "we quantized once and it looked fine" into a property the team keeps.