In pandas 3.0, why does 0/0 in a Float64 column now count as missing, when pandas 2.x kept a NaN that isna() ignored?
answer
- two markers in one column
- constructor versus arithmetic
- made consistent in 3.0
- infinity is still a value
- an experimental future option
basics
~20 sIn pandas 3.0 the nullable Float64 and pyarrow float dtypes treat NaN as pd.NA everywhere, so a NaN produced by arithmetic is stored as pd.NA and isna() counts it. pandas 2.x kept such a NaN as an ordinary, non-missing value.
solid answer
~40 s`Float64` keeps values plus a missing-mask, and in pandas 2.x `NaN` meant missing only some of the time: `pd.Series([0, np.nan], dtype="Float64")` stored `<NA>`, but `ser / 0` stored a real `NaN` for 0/0 that `isna()` reported as present, so missing counts were short and skipping reductions did not skip it. pandas 3.0 makes it consistent: `NaN` passed to constructors, `__setitem__` and `in` is treated as `pd.NA`, and arithmetic or NumPy ufuncs that produce `NaN` now produce `pd.NA`. The same holds for pyarrow float dtypes. NumPy `float64` is untouched, since `NaN` is already its marker, and `inf` from 1/0 is still a value. The experimental option `future.distinguish_nan_and_na`, off by default, keeps `NaN` as a distinct float instead.
code
python · 16 linesimport numpy as np
import pandas as pd
booked = pd.Series([4, 0, 0, 5], dtype="Float64")
no_shows = pd.Series([1, 0, 2, np.nan], dtype="Float64") # np.nan stored as <NA>
rate = no_shows / booked
print(rate)
# 0: 0.25
# 1: <NA> 0/0 gives NaN, stored as pd.NA in pandas 3.0
# 2: inf 2/0 is infinity, a value, not missing
# 3: <NA> the missing input propagates
print(rate.isna().sum()) # 2 in pandas 3.0 (1 in pandas 2.x)
plain = no_shows.astype("float64") / booked.astype("float64")
print(plain.isna().sum()) # 2: NumPy float64 always used NaN as its markergo deeper
Recall that Float64 with a capital F is a nullable dtype, and that in pandas 3.0 a NaN produced inside it, such as 0/0, is stored as pd.NA and counted by isna().
Explain the values-plus-mask design that let a Float64 column hold a NaN that was not missing, and what 2.x did differently for constructor input versus arithmetic results.
Anticipate the upgrade effects: missing counts rising, dropna removing more rows, means changing because computed NaN is now skipped, and infinities still passing through unflagged.
Judge whether a pipeline needs to tell undefined results from absent inputs, and whether that justifies relying on an experimental option or a separate validity column.
## Two ways to be "not a number" `NaN` ("not a number") is the floating-point value IEEE 754 defines for undefined results such as 0/0 or the square root of a negative number. `pd.NA` is pandas' marker for "no value recorded". In a NumPy **`float64`** column the two ideas share one representation, because `NaN` is the only marker that column can hold. The nullable **`Float64`** dtype (capital F) is different. It stores a NumPy float array **plus a separate boolean mask**, and the mask alone says which cells are missing. That design can hold a `NaN` in the values array that the mask does not flag, so a `Float64` column can contain a `NaN` that is not missing next to a `pd.NA` that is. pyarrow-backed float dtypes such as `"float64[pyarrow]"` have the same split, because Arrow arrays carry a validity bitmap beside the values. ## What pandas 2.x did pandas 2.x treated `NaN` as interchangeable with `pd.NA` in some places but not in others: - **Input was converted.** `pd.Series([0, np.nan], dtype="Float64")` stored `0.0, <NA>`; a `NaN` you passed in was taken to mean missing. - **Computed NaN was kept.** `ser / 0` produced `NaN` for 0/0 and kept it as a value, with `<NA>` only where the input had been missing. - **Detection disagreed.** `(ser / 0).isna()` returned `False` for that computed `NaN`, so `isna().sum()` under-counted and `dropna()` kept the row. - **Reductions were exposed.** A `NaN` the mask did not flag was not skipped by `skipna`, so it could turn a `sum()` or `mean()` into `NaN`. A clinic computing a no-show rate as `no_shows / booked` hit exactly this: a new patient with 0 bookings and 0 no-shows produced a `NaN` that looked missing when printed but was invisible to every missing-data method. ## What pandas 3.0 does In pandas 3.0 the rule is consistent: by default `NaN` is **equivalent to `pd.NA`** in the nullable float and pyarrow dtypes. 1. `NaN` passed to constructors, to `__setitem__` (assignment) and to `in` checks is treated as `pd.NA`, as before. 2. Arithmetic and NumPy ufuncs that produce `NaN` now store `pd.NA` instead, so 0/0 becomes `<NA>` and `isna()` reports it. 3. Everything downstream follows: `isna().sum()` counts it, and `sum()` and `mean()` skip it by default. | Expression on `Float64` | pandas 2.x | pandas 3.0 | |---|---|---| | `pd.Series([0, np.nan], dtype="Float64")` | `0.0, <NA>` | `0.0, <NA>` | | `ser / 0` where `ser` is `0.0, <NA>` | `NaN, <NA>` | `<NA>, <NA>` | | `(ser / 0).isna()` | `False, True` | `True, True` | | `pd.Series([1.0], dtype="Float64") / 0` | `inf` | `inf` | Two boundaries matter in an interview: - **NumPy `float64` did not change.** `NaN` was always its missing marker, so `isna()` always counted 0/0 there. - **Infinity is not missing.** 1/0 yields `inf` in both releases, and `isna()` returns False for it. The old `mode.use_inf_as_na` option was removed in pandas 3.0; convert infinities to `NaN` explicitly if they should count as gaps. ## The experimental opt-out pandas 3.0 also adds the option `future.distinguish_nan_and_na` (default False, also settable with the environment variable `PANDAS_FUTURE_DISTINGUISH_NAN_AND_NA=1`). It signals the intended long-term direction: `NaN` and `pd.NA` as distinct values. With it enabled: - `NaN` is always kept as a floating-point value, so `pd.Series([1, np.nan], dtype="Float64")` stores a real `NaN`, not `<NA>`; - the same input with `dtype="Int64"` or `"int64[pyarrow]"` raises, because an integer dtype cannot hold a float `NaN`; - `to_numpy()` returns an `object` array when `pd.NA` is present, unless you pass `na_value=np.nan`. The pandas docs mark the option as experimental and subject to change, so production code should rely on the 3.0 default, not on the option. ## Checking a pipeline after the upgrade Because the change only moves cells from "value" to "missing", it rarely raises an error; it shows up as different numbers. After moving a `Float64` pipeline from 2.x to 3.0, look for: - missing-value counts from `isna().sum()` rising where a division or a ufunc can produce `NaN`; - `dropna()` removing rows it used to keep; - `mean()` or `sum()` returning a number where it used to return `NaN`, because the computed gap is now skipped. ## Interview angle This question separates people who have used nullable dtypes from people who have read about them. A strong answer explains the values-plus-mask design that made two markers possible, the 2.x inconsistency between input and arithmetic, the 3.0 rule, and the limits: NumPy floats unchanged and infinity still a value.
- After upgrading to pandas 3.0, a data-quality check reports more missing values in a Float64 column; is that a regression?Usually not. Cells that 2.x stored as a computed `NaN`, such as 0/0, were invisible to `isna()`, and pandas 3.0 now stores them as `pd.NA`, so the count rises to the true number of undefined results. Confirm by recomputing the column under 2.x rules or by checking which expressions divide by zero.
- Why would anyone enable future.distinguish_nan_and_na?To keep "undefined result" (`NaN`) apart from "never recorded" (`pd.NA`), for example to audit failed calculations separately from absent inputs. The cost is that `NaN` stops counting as missing in `Float64`, integer nullable dtypes refuse it, and `to_numpy()` may return `object`. The option is experimental, so the behaviour may still change.
- Float64 and float64 handle NaN identically in every pandas release.
- pandas 3.0 now treats inf as missing in Float64 columns.
- The change in pandas 3.0 also alters NumPy float64 columns.
- Setting mode.use_inf_as_na makes 0/0 count as missing in 3.0.
- future.distinguish_nan_and_na is on by default in pandas 3.0.
Assumes pandas 3.0: pandas 2.x kept a NaN produced by arithmetic in Float64 as a non-missing value; 3.0 stores it as pd.NA by default.