skip to content

In Transformers.js, why does the same pipeline download a much larger ONNX file after you switch its device from 'wasm' to 'webgpu'?

level: middleimportance: should knowfreq 7%

answer

  1. the device picks more than hardware
  2. unset dtype follows the device
  3. eight-bit on one, full precision elsewhere
  4. suffix in the onnx subfolder
  5. half precision needs an adapter feature

basics

~10 s

With no dtype option (and none in the model's config), Transformers.js picks it from the device: q8 on 'wasm', fp32 on every other device including 'webgpu'. Switching device silently swaps onnx/model_quantized.onnx for onnx/model.onnx.

solid answer

~40 s

Transformers.js resolves `dtype` per model file in a fixed order: the `dtype` option, then `transformers.js_config.dtype` in the model's `config.json` (with a per-device `device_config` override), then the device default, which is `q8` for `'wasm'` and `fp32` for everything else. The dtype becomes a file suffix, so `'wasm'` loads `onnx/model_quantized.onnx` and `'webgpu'` loads `onnx/model.onnx`. `fp16` on browser WebGPU additionally requires the adapter's `shader-f16` feature and otherwise throws `The device (webgpu) does not support fp16.`. `'auto'` and `'gpu'` also default to `fp32`, even when the session ends up on WASM. The fix is to pass `dtype` together with `device`. The `dtype` option itself replaced the v2 `quantized` boolean in Transformers.js v3.

go deeper

for a junior

Recall the two defaults: q8 on 'wasm' and fp32 on every other device, and that dtype is the option that overrides them.

for a middle

Explain the resolution order (option, model config, device default), how a dtype becomes a file suffix in the onnx folder, and the shader-f16 requirement for fp16.

for a senior

Show a per-device dtype policy, why 'auto' downloads fp32 even on WASM, and why silent fallbacks for typos and q4f16 deserve your own checks.

for a principal

Weigh one dtype for every visitor against per-device variants, counting download size, cache footprint and the test matrix each variant adds.

## Two decisions, one file name Every model repository prepared for Transformers.js keeps its ONNX graphs in an `onnx/` subfolder, one file per precision variant. Which file a session downloads is decided by two options together: **`device`** (where it runs) and **`dtype`** (which numeric variant of the weights to load). Leave `dtype` out and the device decides it for you, which is why changing only the device can quietly change the download. ## How the dtype is resolved For each model file, Transformers.js resolves the dtype in this order: 1. **The `dtype` option** you pass to `pipeline()` or `from_pretrained()`, either one string or an object mapping each model file (such as `encoder_model` or `decoder_model_merged`) to its own dtype. 2. **The model's own `config.json`**, under `transformers.js_config.dtype`; a `device_config` entry for the selected device can override it. 3. **The device default**: **`q8` for `'wasm'`**, **`fp32` for every other device** value, including `'webgpu'`, `'cpu'`, `'cuda'`, `'auto'` and `'gpu'`. A string that is not a known dtype name does not raise an error: resolution silently falls back to the device default. A typo such as `'Q8'` therefore loads whatever the device would have chosen anyway. ## From dtype to file The resolved dtype becomes a file-name suffix. For a single-file encoder model whose base name is `model`: | `dtype` | File loaded | Note | |---|---|---| | `fp32` | `onnx/model.onnx` | default on every device except `'wasm'` | | `fp16` | `onnx/model_fp16.onnx` | half precision | | `q8` | `onnx/model_quantized.onnx` | default on `'wasm'` | | `int8` / `uint8` | `onnx/model_int8.onnx` / `onnx/model_uint8.onnx` | 8-bit variants | | `q4` | `onnx/model_q4.onnx` | 4-bit weights | | `q4f16` | `onnx/model_q4f16.onnx` | fp16 model with 4-bit block weights | So a pipeline that pulled `model_quantized.onnx` on `'wasm'` pulls `model.onnx` after the switch to `'webgpu'`, roughly four times the bytes for the weights. What each format costs in accuracy is a quantization question; here the point is only which file runs where. ## fp16 on WebGPU needs shader-f16 Half precision on the GPU depends on the adapter. Before building a browser WebGPU session with `dtype: 'fp16'`, the library requests an adapter and checks `adapter.features.has('shader-f16')`. Without it, loading throws `The device (webgpu) does not support fp16.` before the file is fetched. Three edges of that guard: - **Browser only.** Under Node, the native WebGPU provider is assumed to support fp16 and the check is skipped. - **Exactly `fp16`.** The check does not run for `q4f16`, whose weights are also stored around fp16, so test `shader-f16` yourself before choosing it. - **Exactly `'webgpu'`.** With `device: 'auto'` the guard is skipped, because the resolved device string is `'auto'`. ## Where 'auto' and 'gpu' surprise you The default lookup uses the device *value you passed*, not the provider that eventually runs. `device: 'auto'` in a browser without WebGPU still asks for the `fp32` file, then runs it on WASM: you get the largest download on the slowest provider. The same holds for `'gpu'`. ## A per-device policy - **Pass `dtype` whenever you pass `device`**, so both branches of a WebGPU-or-WASM decision are deliberate. - **Probe what the repository offers**: `ModelRegistry.get_available_dtypes(modelId)` lists the dtypes whose files exist on the Hub, so a policy never asks for a missing variant. - **Keep sensitive modules apart**: the object form of `dtype` lets an encoder stay at `fp16` while a decoder runs `q4`, which the library's guide introduces for quantization-sensitive models such as Whisper and Florence-2. - **Read the model's `transformers.js_config`** before assuming the device default applies; a repository can ship its own default. A minimal pairing, with both checks done in your code before the pipeline is created: ```javascript const adapter = 'gpu' in navigator ? await navigator.gpu.requestAdapter() : null; const device = adapter ? 'webgpu' : 'wasm'; const dtype = device === 'wasm' ? 'q8' : adapter.features.has('shader-f16') ? 'fp16' : 'fp32'; ``` Every branch now names its file, so a review can see what each visitor downloads. ## Where the option came from Before Transformers.js v3 the choice was a boolean `quantized` option: `true` for the 8-bit file, `false` for full precision. v3 replaced it with `dtype` and its wider list (`fp32`, `fp16`, `q8`, `int8`, `uint8`, `q4`, `bnb4`, `q4f16`; 4.3 also lists `q2`, `q2f16`, `q1` and `q1f16`), and the per-device default described here is how 4.3 resolves an unset `dtype`.

  • What happens if you pass dtype: 'Q8' with the wrong case?
    It is not a known dtype name, and Transformers.js does not reject it: resolution falls back to the device default without an error. On `'wasm'` that happens to be `q8`; on `'webgpu'` it is `fp32`. Unknown strings are silently ignored, so validate the value you pass.

  • The default dtype is the same on every device.
  • Switching device never changes which weights file is downloaded.
  • fp16 works on any browser that exposes WebGPU.
  • device 'auto' picks the small q8 file whenever it lands on WASM.
  • A misspelled dtype makes Transformers.js throw an error.

Assumes Transformers.js 4.3; the dtype option replaced the quantized boolean in v3, and the per-device default (q8 on wasm, fp32 elsewhere) is read from 4.3's dtype resolution code.