Why does an exported ExecuTorch model reject a different input size on device?

## Specialization is the default, not an accident `torch.export.export(model, (torch.randn(1, 3, 224, 224),))` does not merely note that the input is a 4-D float tensor. It records `1, 3, 224, 224` as concrete facts and lets every downstream shape computation constant-fold against them. Reshapes become literal target shapes, broadcasts become fixed, and the graph is valid only for that geometry. The exporter attaches guards expressing those assumptions, and the artifact enforces them. This is deliberate. Static shapes are what allow `to_executorch()` to perform **memory planning** — computing ahead of time exactly how large the intermediate-tensor arena must be and where each temporary lives inside it. That is the reason the device runtime can execute a model with almost no dynamic allocation, which is exactly what you want in an app with a tight memory budget and a hostile OS-level memory killer. ## What the failure looks like On device you get a load-or-execute failure when the input tensor you hand `forward()` does not match the recorded signature. It is not a helpful "expected 224 got 256" from the model author's perspective; it is a contract violation at the runtime boundary. Candidates who have not shipped this often assume the runtime will resize or that the delegate will recompile — nothing recompiles on the phone, because there is no compiler there. ## Declaring a dimension dynamic ``` from torch.export import Dim, export batch = Dim("batch", min=1, max=8) ep = export(model.eval(), (x,), dynamic_shapes={"x": {0: batch}}) ``` The `dynamic_shapes` argument maps each input to the axes that are allowed to vary, and each such axis carries a `Dim` with a name and a range. Naming matters: reusing the same `Dim` object across two inputs asserts that those axes are *equal*, which is how you express "sequence length of the ids matches sequence length of the mask". Bounds matter for a practical reason. The memory plan has to size the arena for the worst case, so an unbounded dimension is either rejected or forces a conservative plan. On a device you usually *want* a tight upper bound — it converts a potential out-of-memory crash in the field into an export-time decision. More recent torch versions also let you write `Dim.AUTO` to have the exporter infer which dimensions must be dynamic rather than asserting them yourself; it is convenient for exploration, but for a shipped artifact an explicit named `Dim` with an explicit range documents the contract. ## Where dynamic shapes get expensive Three things go wrong once a dimension is symbolic: - **Guard failures during export.** A model that internally does `x.view(x.shape[0] * 4, -1)` or indexes by a computed size may need extra constraints, and export will tell you that a guard was added on a dynamic dimension. Each such message is a place the model reasons about size in a way you must either bound or rewrite. - **Reduced delegation.** Backends differ in dynamic-shape support. A partitioner that cannot handle a symbolic dimension in a given operator simply declines that subgraph, which falls back to the runtime's portable kernels. The model still runs; it just runs slower than the static version, and the regression is invisible unless you measure delegated coverage. - **Looser memory planning.** Planning against the maximum means holding the worst-case arena even for small inputs. ## The alternative: normalize on the host For a lot of mobile work, the better answer is not to make the model dynamic at all. Vision pipelines resize and letterbox the camera frame to a fixed network input anyway; doing that in the app with the platform's image APIs keeps the exported graph fully static and fully delegated. Reserve dynamic dimensions for cases where padding genuinely costs you — variable-length sequence models being the obvious one, where padding every request to the maximum length wastes real compute. A middle option is **bucketing**: export a small set of shapes (say 128, 256 and 512 tokens) as separate methods or files and dispatch to the nearest bucket at run time. You keep static-shape performance and bound the padding waste, at the cost of artifact size — which is a size-versus-latency trade you should be explicit about. ## How to answer the diagnosis question "The model works in Python but rejects our camera frames" is nearly always this: exported at the notebook's example resolution, fed the device's native resolution. Confirm by inspecting the exported program's input signature, reproduce in Python by loading the `.pte` through the runtime bindings with the device's actual shape, then choose deliberately — normalize on the host, declare a bounded dynamic dimension, or ship buckets — and re-check how much of the graph still lands on the delegate afterwards.