How do you choose between ONNX Runtime, Triton, and ExecuTorch for a PyTorch model?

## Frame it as an artifact-and-owner decision The question is not which runtime is fastest. It is which artifact you commit to producing, who maintains the second numerical stack it creates, and what happens to that commitment when the model changes. Answer in that order and the runtime falls out. ## Where the model runs - **On device — phones, embedded, wearables.** ExecuTorch is PyTorch's on-device path: you capture with `torch.export`, lower to a `.pte`, and run it through a small C++ runtime with hardware delegates. Nothing else here targets that environment, so the decision is made for you; the remaining work is operator coverage and binary size. - **A CPU fleet or mixed accelerators, thin surrounding code.** ONNX Runtime is strong here. One artifact runs across CPU, several GPU vendors, and other targets through its execution providers, and the format is stable and inspectable. The price is the export: your model must survive the translation, and you own the parity testing. - **A GPU service with substantial pre- and post-processing.** Triton Inference Server's PyTorch and Python backends let you keep that code where it already works, at the cost of shipping a PyTorch runtime with it. When pre-processing is the complicated part — and in vision and speech pipelines it usually is — this is the pragmatic choice. ## How much of the pipeline is not the model Export paths handle the tensor computation. They handle tokenizers, image decoding, resampling, tracking state, calibration tables and business rules badly or not at all. Teams underestimate this every time. Count what fraction of your latency budget and, more importantly, your *code* lives outside `forward`. If it is large, an export-everything strategy means reimplementing that code in another language and then keeping two implementations agreeing forever. Keeping it in Python behind a PyTorch-native backend is often the lower-total-cost answer even though it looks less clean on a diagram. ## The second numerical stack This is the argument a principal is expected to make and most candidates never do. Every non-PyTorch runtime introduces a second implementation of your model's arithmetic. From then on, each release needs an export step, a parity gate over a fixture matrix, and a tolerance policy — plus someone who can debug the runtime when it and PyTorch disagree at 2 a.m. That is a permanent tax measured in engineer-time. It is worth paying when it buys something concrete: a hardware target you cannot otherwise reach, a language you must integrate with, a serious cost reduction on a large fleet. It is not worth paying to feel more production-ready. ## Export risk should be discovered early Whichever path you pick, the failure mode is always the same shape: an operator that will not export, control flow that must be rewritten, dynamic shapes that specialize. Discover it in week one, not at the end, by exporting a skeleton of the real architecture before committing. If a model family is under active research, its ability to export is a moving target, and betting a serving design on it is how deadlines die. ## The retired option TorchServe was the obvious answer for years and no longer is: last release 0.12.0 on 2024-09-30, no longer maintained. Bringing it up as a live choice tells an interviewer your knowledge is a few years stale. The right sentence is that the PyTorch-native serving story now routes through generic servers with PyTorch backends, ONNX Runtime, or dedicated engines for the model class in question. ## An answer that lands Say what you would do and why it might be wrong. For example: keep the model in a PyTorch backend behind a general-purpose server while the architecture is still moving and the pre-processing is heavy; export to ONNX Runtime once the architecture stabilizes and the fleet is large enough that the second stack pays for itself; go to ExecuTorch only when the target is a device. Add the checkpoints that would make you revisit — a hardware change, an export blocker in a new architecture, a parity failure rate that shows the tolerance budget was wishful. Naming your own reversal conditions is what distinguishes this level.