Your ONNX export of a PyTorch model returns different numbers — how do you debug it?

## First: is it a bug or is it noise? Never compare exported and eager outputs with `==`. ONNX Runtime fuses operators, reassociates arithmetic and may use different kernels, all of which shift the last bits. Set an explicit tolerance — `rtol=1e-4, atol=1e-5` is a reasonable starting point for float32 — and record what tolerance the model actually needs as part of its release criteria. Absolute error at 1e-6 with a max-relative error at 1e-5 is a healthy export. A disagreement in the second significant figure, or a different argmax on a classifier, is a defect. Also measure the *right* thing: for a classifier, check top-1 agreement over a few hundred real inputs, not just tensor closeness on one synthetic batch. A model can be numerically close and still flip labels near a decision boundary — you want to know that before it ships. ## Cause 1: the module was in training mode By far the most common. If the module is not in evaluation mode when you export, dropout is active and batch normalization uses the current batch's statistics rather than its running estimates. Both get captured into the exported graph. The output is not obviously broken — it is stochastic or subtly shifted, which is worse. Put the module in evaluation mode before export and confirm it, especially in a pipeline where a training script hands a model to an export step. ## Cause 2: baked shapes Export is a tracing operation. Every dimension you do not declare dynamic is a constant in the graph. With the legacy exporter you declare them with `dynamic_axes`, naming the input and output axes that may vary; with `torch.onnx.export(..., dynamic_axes=None, dynamo=True)` the modern path runs `torch.export` first and takes `dynamic_shapes`. Skip that step and a batch of 8 through a graph exported at batch 2 will either error on a reshape or, when a reshape uses a literal, produce quietly wrong output. Any reshape or view written with hardcoded numbers is a red flag — derive sizes from the tensor. ## Cause 3: control flow and Python values The legacy exporter runs TorchScript-style tracing, so `if` on a tensor value, `.item()`, `len()`, list indices computed in Python and early returns are resolved once. The exported graph carries only the path the example input took. If the model has real conditionality, the fix is upstream: rewrite it as `torch.where`, or move to the `torch.export`-based path and express the branch as `torch.cond`. ## Cause 4: operator coverage An operator with no ONNX equivalent at the chosen `opset_version` either fails the export loudly or gets decomposed into something with slightly different edge-case behaviour — padding modes, rounding on integer division, `nan` handling in reductions, non-deterministic ops like `topk` on ties. Raising `opset_version` often fixes this because newer opsets carry closer equivalents. Read every export warning; the exporter tells you when it substituted something. ## Bisecting When the cause is not obvious, cut the model in half. Two techniques: - **Export submodules.** Export the backbone alone, then the head, and compare each against its eager counterpart. The first submodule that disagrees owns the bug. - **Expose intermediates.** Temporarily change `forward` to return intermediate activations as extra outputs and export that. Now one ONNX Runtime call gives you every checkpoint in the network, and you can find the first layer where the curves separate. ## Environment causes If eager and ONNX agree on CPU but not on GPU, you are looking at an execution-provider difference: `providers=["CUDAExecutionProvider"]` may pick different kernels or accumulate in different precision than the CPU provider, and any half-precision cast in the pipeline multiplies that. Compare on the CPU provider first to isolate. ONNX Runtime also applies graph optimizations at session creation; lowering the session's optimization level and seeing the difference disappear is strong evidence that a fusion is the culprit, and worth reporting rather than working around silently. ## Make it a gate, not an incident All of this belongs in CI. Export as a build step, run a parity test over a fixture set covering multiple shapes and both classes of input, and fail the build on regression. The reason export bugs reach production is that export is usually a one-off human step performed at the end of a project, by which point nobody remembers what the eager model returned.