skip to content

In PyTorch, how does type promotion pick the result dtype of a mixed-dtype op?

level: middleimportance: should knowfreq 50%

answer

  1. category first, then width
  2. bool, integer, float, complex
  3. Python scalars are weak participants
  4. integer division still yields a float
  5. result_type answers it without running

basics

~20 s

PyTorch picks the dtype that can represent both operands: the higher category first (bool below integer below float below complex), then the wider type within that category. Python scalars and 0-dim tensors count only weakly and never widen a dimensioned operand.

solid answer

~40 s

Promotion runs in two steps. First the **category** — bool < integer < floating point < complex — is raised to the higher of the two, which is why dividing two integer tensors yields a floating tensor at `torch.get_default_dtype()` (float32). Then, within a category, the **wider** type wins, so float32 combined with float64 gives float64. The subtlety is participation: dimensioned tensors dominate. A Python scalar or a 0-dim tensor can raise the category but cannot widen the type, so `float32_tensor * torch.tensor(2.0, dtype=torch.float64)` stays float32 while the same op against a `(3,)` float64 tensor gives float64. Defaults matter too: `torch.tensor([1, 2, 3])` is int64, `torch.tensor([1.0])` is float32 (not float64 as in NumPy), and arrays arriving via `torch.from_numpy` are usually float64. `torch.result_type(a, b)` answers the question without running the op.

code

python · 11 lines
python
import torch

print(torch.tensor([1, 2, 3]).dtype)         # torch.int64
print(torch.tensor([1.0, 2.0]).dtype)        # torch.float32
print((torch.tensor([1, 2, 3]) / 2).dtype)   # torch.float32

f32 = torch.ones(3, dtype=torch.float32)
weak = torch.tensor(2.0, dtype=torch.float64)   # 0-dim
strong = torch.ones(3, dtype=torch.float64)     # 1-dim
print((f32 * weak).dtype)                    # torch.float32
print(torch.result_type(f32, strong))        # torch.float64

go deeper

for a junior

Remember the defaults: integer literals give int64, float literals give float32, and dividing two integer tensors gives a float tensor. Know that .float() and .to(dtype) are how you convert.

for a middle

Explain the two-step rule — category first, then width — and the weak participation of Python scalars and 0-dim tensors. Mention torch.result_type and why in-place ops refuse to promote.

for a senior

Talk about the cost side: a float64 that promotes successfully is worse than one that errors, because it silently doubles memory and slows GPU math, and about pinning dtype at ingest with boundary assertions.

for a principal

Set the numeric policy for the codebase — the storage dtype for data, the compute dtype for models, where conversions are allowed to happen — so precision is a deliberate decision rather than an emergent property of whichever library produced the array.

## Why this exists PyTorch did not always promote types; early versions simply refused mixed-dtype arithmetic. Modern PyTorch follows a NumPy-like promotion scheme so that mixing dtypes usually just works, which is convenient and occasionally expensive — a stray float64 can double a tensor's memory and cut throughput, particularly on GPUs where float64 is drastically slower than float32. ## The rules Each dtype has a **category**, ordered bool < integer (including unsigned) < floating point < complex, and a width within the category. Given two operands: 1. Take the higher category of the two. 2. Within that category, take the type wide enough to hold both. 3. Apply participation rules: operands that are Python numbers or 0-dim tensors — internally "wrapped numbers" — can bump the category but do not influence the width. Dimensioned tensors decide the width. So `torch.ones(3, dtype=torch.int64) * 2.5` gives float32: the Python float raises the category to floating point, and since no dimensioned floating operand exists the default dtype supplies the width. But `torch.ones(3, dtype=torch.float32) * torch.tensor(2.0, dtype=torch.float64)` stays float32, because the float64 operand is 0-dim and therefore weak. Two dimensioned floats of different widths, `float32` and `float64`, give float64. Division is special-cased at the category level: `/` on integers is true division, so `torch.tensor([1, 2, 3]) / 2` is float32 rather than integer. Use `torch.div(..., rounding_mode="floor")` or `//` when you want integer division. Comparisons return `torch.bool` regardless of operand types. ## The defaults you have to remember - `torch.tensor([1, 2, 3])` → `torch.int64`. - `torch.tensor([1.0, 2.0])` → `torch.float32`, even though Python floats are 64-bit. This is the single biggest difference from NumPy, whose default is float64. - `torch.get_default_dtype()` returns float32; `torch.set_default_dtype()` changes it globally, which is a blunt instrument best avoided in library code. - `torch.zeros`, `torch.ones`, `torch.rand`, `torch.randn` and friends use the default dtype unless you pass `dtype=`. - `torch.from_numpy` preserves the array's dtype, so float64 arrays produce float64 tensors — the usual way float64 sneaks into a float32 pipeline. ## In-place ops do not promote An underscore op has to write into the destination's existing storage, so it cannot change the destination's dtype. `int_tensor.add_(2.5)` raises rather than promoting to float, with an error about the result type not being castable to the output. This is a useful accidental type check: if in-place arithmetic starts failing, something upstream changed a dtype. The same applies to `out=` arguments, which must already have a castable dtype. ## Where wrong dtypes hurt A float64 tensor entering a float32 model raises a dtype-mismatch RuntimeError at the first matmul — loud, and easy to fix. The quieter damage is when everything promotes successfully: a float64 branch inside a preprocessing pipeline doubles memory and, on GPU, runs an order of magnitude slower than float32 on consumer hardware. In the other direction, integer overflow is silent — int8 and int32 accumulators wrap around without warning, so counting or summing large integer tensors should be done in int64 or a float type. Index tensors are their own trap: indexing and `gather`/`scatter` want int64 (`torch.long`) index tensors, and an int32 index tensor raises. Loss functions such as cross-entropy want int64 class targets, not float. ## Tools for reasoning about it `torch.result_type(a, b)` reports the promoted dtype without executing the op — good for assertions and for writing library code that must allocate an output. `torch.promote_types(dtype1, dtype2)` answers the same question for dtypes alone. `torch.can_cast(from, to)` reports whether a cast is allowed under the promotion rules. To convert, use `Tensor.to(dtype)` (or the shortcuts `.float()`, `.double()`, `.long()`, `.half()`), remembering that `.to()` returns `self` unchanged when the dtype already matches, so it is cheap to call defensively. ## The habit Pin the dtype at ingest — as the data enters the pipeline, not layer by layer — and assert it at component boundaries. Most dtype incidents are a single unconverted NumPy array or an integer label tensor that a helper turned into a float, and both are caught by one assertion at the edge.

  • Why does torch.tensor([1.0]) give float32 when the Python literal is a 64-bit float?
    Because PyTorch's default dtype is float32, as `torch.get_default_dtype()` reports, and factory functions use it rather than mirroring Python's precision. Deep learning runs in float32 or lower, so float64 would be a poor default. This differs from NumPy, which defaults to float64 — the mismatch is why arrays crossing over with `from_numpy` so often arrive as float64 tensors.
  • Why does int_tensor.add_(2.5) raise instead of promoting to float?
    In-place ops write into the destination's existing storage, which has a fixed dtype, so there is nowhere to put a promoted result; PyTorch raises a result-type error rather than silently truncating. The same restriction applies to `out=` arguments. It is a useful signal: an in-place op that suddenly fails usually means an upstream dtype changed.
  • How do you check the promoted dtype of an operation without running it?
    `torch.result_type(a, b)` returns the dtype the operation would produce, accepting tensors or Python numbers; `torch.promote_types(d1, d2)` does the same for dtypes alone, and `torch.can_cast(from, to)` reports whether a cast is permitted. These are useful in library code that needs to preallocate an output buffer, and as assertions at pipeline boundaries.
  • Which dtype do index tensors and classification targets need?
    int64. Indexing, `gather` and `scatter` require int64 index tensors and raise on int32, and cross-entropy-style losses expect int64 class indices rather than floats. Since `torch.tensor([0, 1, 2])` already gives int64 this is usually automatic; it breaks when indices arrive from NumPy as int32 on Windows or get cast to float by a generic helper.

saying these in an interview costs you the question

  • Assumes torch.tensor([1.0]) is float64 like NumPy
  • Thinks integer division returns an integer tensor
  • Believes a 0-dim float64 tensor upcasts a float32 tensor
  • Expects in-place ops to promote the destination dtype
  • Ignores float64 creeping in because the math still runs

context

resolving links…