skip to content

In pandas 3.0, which missing marker (NaN, NaT, None or pd.NA) does each dtype hold, and why test for them with isna rather than ==?

level: juniorimportance: must knowfreq 62%

answer

  1. the marker follows the column's dtype
  2. floats and text versus dates
  3. capitalised dtype names carry pd.NA
  4. object keeps whatever you stored
  5. equality never finds a gap

basics

~20 s

NumPy float64 and the pandas 3.0 str dtype hold NaN, datetime64 and timedelta64 hold NaT, object holds what was stored (often None), and Int64, Float64, boolean and string hold pd.NA. pd.isna detects all four; == finds none.

solid answer

~40 s

The marker depends on the dtype. NumPy `float64` uses `NaN`, and so does the `str` dtype that text columns infer in pandas 3.0. `datetime64`, `timedelta64` and period columns use `NaT`. An `object` column keeps whatever you put in, so `None` and `NaN` can sit side by side. The nullable extension dtypes (`Int64`, `Float64`, `boolean`, the opt-in `"string"` dtype and pyarrow-backed dtypes) use the `pd.NA` singleton. You detect any of them with `isna()` / `notna()` or `pd.isna`, never with `==`. For a NumPy-backed column, pandas answers `s == np.nan` or `s == None` with all False (and `!=` with all True) whatever the cells hold, and on a nullable column `s == pd.NA` returns `<NA>` everywhere, so an equality filter finds nothing.

code

python · 16 lines
python
import numpy as np
import pandas as pd

visits = pd.DataFrame({
    "count_float": [3.0, None],                            # float64: NaN
    "count_nullable": pd.array([3, None], dtype="Int64"),  # Int64: <NA>
    "last_visit": pd.to_datetime(["2026-03-02", None]),    # datetime64: NaT
    "clinic": ["north", None],                             # str in 3.0: NaN
    "note": pd.Series(["ok", None], dtype=object),         # object: None kept
})
print(visits.dtypes)
print(visits.isna().sum())                     # 1 in every column

print((visits["count_float"] == np.nan).sum())  # 0: == never finds NaN
print((visits["note"] == None).sum())          # 0: not even None
print(visits["count_nullable"] == pd.NA)       # boolean column of <NA>

go deeper

for a junior

Recall the four markers and their dtypes: NaN for floats and 3.0 text, NaT for dates, None only in object columns, pd.NA in Int64, Float64 and boolean. Always count gaps with isna().sum().

for a middle

Explain why the markers differ: each dtype stores a value in every slot, so the marker is whatever that storage can hold, while nullable dtypes keep a separate mask and report pd.NA.

for a senior

Spot the silent bugs: an == np.nan filter returning zero rows, an is None check broken by the 3.0 str dtype, and if value: raising on pd.NA inside an apply or a loop.

for a principal

Weigh whether a codebase should keep mixed markers or standardise on nullable dtypes, and how a 2.x to 3.0 upgrade changes the marker that text columns hand to downstream code.

## Why pandas has more than one missing marker A **missing marker** (pandas calls them sentinels) is the value pandas stores in a cell to mean "nothing was recorded here". pandas sits on top of NumPy arrays, and a NumPy array of one **dtype** (data type) must store something of that type in every slot. So each dtype got the marker its storage could hold, and the result is four markers rather than one: - **`NaN`** (`np.nan`), the IEEE floating-point "not a number", for NumPy `float64` and `float32` columns; - **`NaT`** (`pd.NaT`), "not a time", for `datetime64`, `timedelta64` and period columns; - **`None`**, Python's own null, which survives only in `object` columns, since those store arbitrary Python objects; - **`pd.NA`**, a singleton added in pandas 1.0 for the **nullable extension dtypes**, which keep the values in one array and a separate boolean mask that marks which slots are missing. ## Which column holds which marker | Column dtype | Missing marker | How a gap gets there | |---|---|---| | `float64` (NumPy) | `NaN` | `None` or `np.nan` in the input | | `str` (default for text in pandas 3.0) | `NaN` | `None` in a list of strings | | `datetime64`, `timedelta64`, period | `NaT` | `None`, `np.nan` or an unparseable date coerced | | `object` | whatever was stored: `None`, `NaN` or `pd.NA` | assigning that value | | `Int64`, `Float64`, `boolean`, `"string"`, pyarrow dtypes | `pd.NA` | any NA-like input is converted | Two rows of that table changed or surprise people: 1. **Text in pandas 3.0.** `pd.Series(["north", None])` infers the new `str` dtype, whose missing value is always `NaN`. In pandas 2.x the same list inferred `object` and kept the `None` as given. Code that checks `x is None` on a text cell therefore stops matching after an upgrade; the lesson on pandas text columns covers the rest of that dtype. 2. **Nullable dtypes convert everything.** `pd.array([1, np.nan, None, pd.NA], dtype="Int64")` stores three `pd.NA` values; the input marker is forgotten. The capital letter matters: `"Int64"` is the pandas nullable dtype, `"int64"` is NumPy's, which cannot hold a gap at all. None of the nullable dtypes is the default. A plain `pd.Series([1, None])` is still NumPy `float64` holding `NaN`; you get `pd.NA` only by asking for a nullable dtype (or by a conversion step, which the lesson on casting covers). ## Detecting a gap: isna, never == `isna()` (alias `isnull()`) and `notna()` recognise every marker in the table, as do the functions `pd.isna` and `pd.notna` on a scalar or an array. Equality does not, for three different reasons: - `np.nan == np.nan` is False by the floating-point standard, and pandas goes further: for a NumPy-backed column, comparing against a missing scalar such as `np.nan`, `None` or `pd.NA` returns all False for `==` and the ordering operators, and all True for `!=`; - that rule applies even to an `object` column holding `None`, although `None == None` is True in plain Python, so `s == None` finds nothing; - `pd.NA` **propagates** through comparisons: `pd.NA == pd.NA` is `pd.NA`, and on a nullable column `s == pd.NA` returns a `boolean` column of `<NA>` values, which selects no rows when used as a mask. The practical rule is short: count with `df.isna().sum()`, test a scalar with `pd.isna(value)`, and never write `== np.nan`, `is np.nan` or `== None` against pandas data. ## Reading the marker back The marker also shapes what you see when you pull a value out. `s.iloc[2]` on an `Int64` column returns `pd.NA` itself (`s.iloc[2] is pd.NA` is True), while on a `float64` column it returns a NumPy float that is `nan`. That difference leaks into plain-Python code: `if value:` raises `TypeError: boolean value of NA is ambiguous` for `pd.NA`, but silently treats `nan` as truthy. The fix is the same in both cases: test with `pd.isna(value)` first. ## Interview angle Interviewers use this question to see whether a candidate knows that pandas missing data is dtype-shaped. A good answer names the four markers, ties each to its dtype, mentions the 3.0 change for text, and closes with the equality trap. A weak answer says "pandas uses NaN" and filters with `== np.nan`, a filter that returns zero rows and no error.

  • Why does s == None return all False even on a pandas object column that holds None?
    For a NumPy-backed column, pandas special-cases a missing scalar on the right of a comparison: `==` and the ordering operators return all False and `!=` returns all True, without looking at the cells. `None` counts as a missing scalar, so the Python rule `None == None` never applies. Use `s.isna()` instead.
  • If none of the nullable dtypes is the default, where does pd.NA come from in a real pipeline?
    Only from a nullable dtype someone chose: an explicit `dtype="Int64"`, `"Float64"`, `"boolean"` or `"string"`, `pd.array` inferring `Int64` from integers with `None`, a pyarrow-backed dtype, or a conversion or reader option that switches to nullable dtypes. A default `pd.Series` or `pd.DataFrame` built from Python lists never contains `pd.NA` unless you put it into an `object` column yourself.

  • NaN is the only missing marker pandas uses.
  • df[df['x'] == np.nan] selects the rows where x is missing.
  • A None in a list of strings stays None in pandas 3.0.
  • A missing date turns a datetime64 column into float64.
  • pd.NA is the default marker for every column in pandas 3.0.

Assumes pandas 3.0: text columns infer the str dtype whose missing value is NaN; pandas 2.x inferred object and kept None as given.