How do you choose LiteRT delegates for a diverse Android device fleet?

## The premise: heterogeneity is the problem, not the model On a server you choose hardware. On a phone fleet the hardware chooses you: hundreds of SoCs, a dozen GPU driver lineages, vendor NNAPI implementations of wildly varying quality, and devices from budget tiers where the "accelerator" is slower than four CPU cores. A delegate decision that is a build-time constant is therefore wrong for a large fraction of your users by construction. The senior-to-principal shift is from "which delegate is fastest?" to "what system decides per device, and how do I change my mind after shipping?" ## Layer one: a baseline that cannot fail XNNPACK with a measured thread count is the floor. It is the default float CPU path, it is present everywhere, and it is a strong competitor — vectorised and multi-threaded. Every acceleration path must be written as an *optimisation over* this baseline, with delegate creation treated as a fallible operation: missing GPU driver, unsupported NNAPI, delegate library absent from a stripped build. Catch the failure, log it with device identifiers, run on CPU. An app that crashes because a delegate would not initialise is a defect on somebody else's silicon. ## Layer two: evidence per device class What you need per tier is a small table: CPU baseline latency, delegate latency, cold-start cost, delegated-node and partition counts, and the accuracy delta. Four facts drive the decision: - **Steady-state latency** — does the delegate win at all against XNNPACK? - **Cold start** — delegate init and kernel compilation are paid on first use. A per-frame camera model amortises them instantly; a model invoked once per session may never recover them. - **Fragmentation** — many partitions mean transfer overhead dominates, and the fix is the model, not the flag. - **Numerics** — GPU backends commonly compute in float16, so validate the shipped metric, not a tensor diff. ## Layer three: a decision you can change remotely The deployment mechanism matters as much as the measurement. A per-SoC allow/deny list delivered by remote config lets you enable a delegate on the chipsets you tested and disable it on one that turns out to produce garbage or hang, without waiting for an app-store release. This is not hypothetical: vendor driver regressions ship in OS updates, and your model is the thing that surfaces them. For the long tail you never tested, a **first-launch acceptance probe** is the pragmatic extension: on first run, execute the model once through the delegate with a fixed input, under a timeout, compare against a cached expected output within tolerance, and persist the verdict. Devices that pass get acceleration; the rest quietly stay on CPU. It costs one inference at install time and converts an unbounded compatibility matrix into a local decision. ## Layer four: costs that are not latency - **Binary size.** Each bundled delegate adds to the APK. On a fleet where install conversion matters, that trade is real, and Play delivery mechanisms that fetch components conditionally are worth considering. - **Contention.** The GPU is also drawing your UI. A model that saturates it may deliver lower inference latency and a visibly worse app. Measure frame rate alongside inference time. - **Power and heat.** Sustained accelerator use drains battery and throttles the device, so a benchmark's steady state must be a *warm* steady state. - **NNAPI's trajectory.** NNAPI was always the most variable option — the same call routes to completely different vendor implementations — and Google has been deprecating it on newer Android in favour of vendor-supplied and Play-delivered acceleration paths. Treat it as legacy support for older devices rather than the strategic direction, and say so if asked. ## Layer five: keep the model delegate-friendly The most durable lever is upstream. A model built from widely supported ops with static shapes delegates cleanly on nearly every backend; one exotic layer in the middle bisects the graph on all of them at once. Constraining the architecture to a supported operator set — and testing partitioning as part of model CI, since converter settings change partitioning between versions — buys more fleet-wide performance than any per-device tuning. ## The telemetry loop Finally, ship instrumentation: inference latency percentiles, delegate-used, delegate-init-failed, and cold-start, all keyed by device model and OS version. That is what tells you a chipset regressed after an OS update, and it turns the allow-list from a static artefact into a maintained one.