skip to content

In pandas, why does df.to_numpy() on a feature frame with an Int64 column return an object array, and how do dtype= and na_value= fix it?

level: middleimportance: must knowfreq 50%

answer

  1. one array needs one dtype
  2. nullable plus NumPy columns
  3. pd.NA inside an object array
  4. ask for float64 explicitly
  5. decide what missing becomes

basics

~20 s

DataFrame.to_numpy() builds one array with one common dtype, and a nullable Int64 column mixed with other columns resolves to object, holding pd.NA. Passing dtype='float64' gives a numeric array with NaN; na_value= fills missing values with a number you choose.

solid answer

~40 s

`DataFrame.to_numpy()` returns a single NumPy array, so pandas has to find one dtype for all columns. Plain NumPy columns resolve sensibly (`int64` plus `float64` gives `float64`), but when the common type is a pandas **extension dtype** such as `Int64`, `Float64` or `boolean` (what `convert_dtypes()` produces) mixed across columns, the result is an `object` array holding `pd.NA`. A model-training library then either rejects it or fails converting `pd.NA` to float. The fix is to say what you want: `df.to_numpy(dtype="float64")` returns floats with `NaN` for missing values, and `df.to_numpy(dtype="float32", na_value=0)` also fills them. On a single `Int64` Series the default is already `float64` with `NaN`, but `to_numpy(dtype="int64")` raises `ValueError` while missing values remain, unless you pass `na_value=`.

code

python · 20 lines
python
import numpy as np
import pandas as pd

features = pd.DataFrame(
    {
        "age": pd.array([31, None, 45], dtype="Int64"),
        "spend": [10.5, 3.0, np.nan],
    }
)
print(features.to_numpy().dtype)
print(features.to_numpy(dtype="float64"))
print(features.to_numpy(dtype="float32", na_value=0))

age = features["age"]
print(age.to_numpy())
print(age.to_numpy(dtype="int64", na_value=-1))
try:
    age.to_numpy(dtype="int64")
except ValueError as err:
    print("ValueError:", err)

go deeper

for a junior

Know that to_numpy() returns one NumPy array with one dtype, and that missing values need a decision before a model sees them.

for a middle

Explain why Int64 mixed with other columns comes back as object with pd.NA, and how dtype='float64' and na_value= produce a clean numeric matrix.

for a senior

Make the hand-off explicit in pipelines: select feature columns, convert with a stated dtype and na_value, and assert the dtype, so convert_dtypes() upstream cannot break training.

for a principal

The trade-off is where imputation happens: a filler in to_numpy is invisible downstream, so a lead may require missing-value handling to live in a named, tested step instead.

## One array, one dtype A pandas **DataFrame** stores each column with its own **dtype** (data type). A **NumPy array** has exactly one dtype for all its cells. `DataFrame.to_numpy()` therefore has to pick a single common dtype for every column it packs into the 2-D result. The signature is `to_numpy(dtype=None, copy=False, na_value=<no default>)`, and when `dtype` is not given, pandas works the dtype out from the columns. For plain **NumPy dtypes** the rule is the usual upcast: `int64` with `float64` gives `float64`, `float16` with `float32` gives `float32`. The trouble starts with **extension dtypes**, the pandas-defined dtypes that carry their own missing-value mask: `Int64`, `Float64`, `boolean`, `category`, the `str` dtype and Arrow-backed dtypes. ## When the result is `object` pandas falls back to **`object`**, an array of Python objects with one per cell, whenever no NumPy dtype can hold every column. When several columns share an extension dtype as their common type, pandas does not translate it into a NumPy dtype for the combined array, so the result is `object` there too. In practice this happens: - with an **`Int64`** column next to a **`float64`** column, even when nothing is missing; - after **`convert_dtypes()`**, which turns `int64`/`float64` columns into `Int64`/`Float64`; - with two or more **`Int64`** columns; - whenever a **text** column (`str` or `object` dtype) or a **boolean** column sits among numeric ones, because strings or booleans mixed with numbers only fit in `object`. Missing values inside such an array are **`pd.NA`** objects. Libraries that expect a numeric matrix either reject the array or try to convert it to float and fail, because `float(pd.NA)` raises `TypeError`. | Columns in the frame | `df.to_numpy()` dtype | |---|---| | `int64`, `float64` | `float64` | | `Int64`, `float64` | `object`, missing as `pd.NA` | | `Int64`, `Float64` (after `convert_dtypes()`) | `object` | | `float64`, `str` | `object` | | `Int64` alone | `float64`, missing as `NaN` | ## Why pandas does not pick float for you It can look unhelpful, since `float64` with `NaN` would hold every value in the `Int64` plus `float64` case. What actually happens: - **The common dtype comes first.** pandas computes one common dtype for all the columns, and for `Int64` with `float64` that is the nullable `Float64` extension dtype. The code that packs several columns into one array maps any extension dtype to `object`, rather than to its NumPy counterpart. - **`pd.NA` is not `NaN`.** A nullable column marks missing values with a mask; turning that mask into a float sentinel is a conversion, and the multi-column path does not do it unless `dtype=` asks for it. - **Float is not lossless anyway.** An `Int64` column can hold integers above 2 to the power 53, which `float64` cannot represent exactly, so an explicit `dtype="float64"` is also the point where you accept that rounding. The Series path is more permissive: a single nullable integer column with missing values converts to `float64` with `NaN` by default, documented since pandas 2.2. ## Fixing it with `dtype=` and `na_value=` 1. **`dtype="float64"`** (or `"float32"`) tells pandas the target, so each extension column converts itself and missing values become `NaN`. 2. **`na_value=`** sets what a missing value becomes. `df.to_numpy(dtype="float32", na_value=0)` gives a float32 matrix with zeros where values were missing. Filling with 0 is an imputation decision, so make it deliberately rather than to silence an error. 3. **Select the columns first.** A text column cannot become `float64`; asking for it raises `ValueError` when pandas tries to convert the strings. Keep only the feature columns before converting. 4. **Check `arr.dtype`** before passing the array on; it is cheaper than debugging the library's error. ## Series behave differently A **Series** converts through its own array, and the default for a nullable integer is more helpful: - `Int64` without missing values gives `int64`; - `Int64` with missing values gives `float64` with `NaN`; - `boolean` with missing values stays `object`; - `to_numpy(dtype="int64")` with missing values raises `ValueError: cannot convert to 'int64'-dtype NumPy array with missing values. Specify an appropriate 'na_value' for this dtype.`, so you pass `na_value=-1` or another sentinel. A one-column frame of `Int64` follows the same path as the Series, which is why a quick test on one column can pass while the full feature frame turns into `object`. ## What `copy` means here `copy=False` (the default) does **not** promise a view; it only allows one. An actual dtype conversion, packing several columns into one array, or an `na_value` fill produces a new array anyway. `copy=True` forces a fresh array even when one is not needed. Whether an array that does share memory with the frame may be written to is a Copy-on-Write rule, covered with pandas Copy-on-Write.

  • What does `s.to_numpy(na_value=-1)` return for an Int64 Series with missing values?
    A `float64` array with `-1.0` in the missing slots. Without `dtype=`, a nullable integer with missing values defaults to float64, and `na_value` only chooses the filler. To get an `int64` array you pass both: `s.to_numpy(dtype="int64", na_value=-1)`.
  • Is `df.values` a safer choice than `df.to_numpy()` for a DataFrame?
    No. On a DataFrame, `.values` returns the same common-dtype array as `to_numpy()` with no arguments, so it gives the same `object` result for mixed nullable columns, and it accepts no `dtype=` or `na_value=` to fix it. The pandas docs recommend `to_numpy()` instead.
  • Does `copy=False` guarantee that to_numpy returns a view of the frame's data?
    No. `copy=False` only permits a no-copy result; an actual dtype conversion, packing several columns into one array, or an `na_value` fill builds a new array anyway. `copy=True` is the guarantee in the other direction: you always get fresh memory.

  • to_numpy() always returns float64 for numeric-looking columns
  • na_value= alone turns an Int64 Series into an int64 array
  • Missing values in a nullable column become 0 automatically
  • df.values avoids the object dtype that to_numpy() produces
  • copy=False guarantees the array is a view of the frame