skip to content

In pandas 3.0, what dtype does a new column of text get by default, and what can that column hold?

level: juniorimportance: must knowfreq 60%

answer

  1. no longer the catch-all dtype
  2. a StringDtype variant
  3. missing values print as NaN
  4. storage depends on PyArrow

basics

~10 s

In pandas 3.0 text is inferred as the str dtype, a StringDtype whose missing value is NaN, stored in PyArrow when installed and in Python objects otherwise. It holds only strings and missing values.

solid answer

~40 s

In pandas 3.0, constructors and readers such as `read_csv` infer a column of Python strings as the `str` dtype instead of NumPy `object`. `"str"` is the alias for `pd.StringDtype(na_value=np.nan)`: every missing entry, `None` included, is stored and shown as `NaN`, so missing-value handling matches the other default dtypes. Its storage is `"pyarrow"` when PyArrow is installed and `"python"` (a NumPy object array of Python strings) otherwise, and `ser.dtype.storage` tells you which. Unlike `object`, it accepts only strings or missing values, so assigning a number into it raises `TypeError`. Passing `dtype=object` still gives the old behaviour, and the `.str` methods work as before.

code

python · 11 lines
python
import pandas as pd

skus = pd.Series(["TSH-L-RED", "HOD-XL-BLU", None])

print(skus.dtype)            # str
print(skus.dtype.na_value)   # nan
print(skus.dtype.storage)    # 'pyarrow' if PyArrow is installed, else 'python'
print(skus[2] is None, pd.isna(skus[2]))

legacy = pd.Series(["TSH-L-RED", None], dtype=object)
print(legacy.dtype, legacy[1])

go deeper

for a junior

Recall that pandas 3.0 prints dtype: str for text columns, that missing values show as NaN, and that a number cannot be written into such a column.

for a middle

Explain that str is a StringDtype with na_value NaN, how its storage is picked from PyArrow availability, and how dtype=object still opts a column out.

for a senior

Name what the change breaks in existing code: equality checks against object, tests for None, assignments of non-strings, and how the pandas 2.3 future option exposes them early.

for a principal

Discuss whether PyArrow should be a required dependency so every environment gets the same str storage and performance, rather than silently falling back to python storage.

## What pandas 3.0 infers for text A **dtype** is the type pandas records for a whole column (a `Series`). Before pandas 3.0, a column built from Python strings was stored with NumPy's **`object`** dtype: an array of pointers to arbitrary Python objects, where nothing guaranteed the values were text. In pandas 3.0 a **dedicated string dtype is the default**. Whenever pandas has to infer a dtype and the input is strings (plus missing values), it now picks **`str`**: - in constructors: `pd.Series(["a", "b"])`, `pd.DataFrame({"sku": [...]})`, `pd.Index([...])`; - in readers such as `read_csv` and `read_parquet`, for columns read as text. The printed footer reads `dtype: str` instead of `dtype: object`. ## The str dtype is a StringDtype `str` is not a NumPy dtype. It is a pandas **extension dtype**, an instance of `pd.StringDtype`, configured with `na_value=np.nan`. Two attributes describe any `StringDtype`: | attribute | value for the default `str` dtype | what it controls | |---|---|---| | `na_value` | `nan` | the missing-value marker and its semantics | | `storage` | `"pyarrow"` or `"python"` | the array that holds the characters | You can name it explicitly with the short alias `dtype="str"`, which the documentation recommends over spelling out `pd.StringDtype(na_value=np.nan)`. ## Storage: pyarrow or python The storage is chosen when the dtype is created, from the option `mode.string_storage`, whose default `"auto"` resolves like this: - **PyArrow installed**: `storage="pyarrow"`. The strings live in a PyArrow string array, which brings large memory savings and speed gains for certain operations. - **PyArrow not installed**: `storage="python"`. The strings live in a NumPy object array of Python `str` objects, with roughly the performance of the old `object` dtype. The behaviour you program against (missing values, accepted values, `.str` methods) is essentially the same for both; speed and memory differ, plus edge cases such as invalid Unicode text, which PyArrow storage rejects. ## Missing values are NaN With `object` dtype, `None` and `np.nan` could both sit in a column. With `str`, **any missing input (`None`, `np.nan`, `pd.NA`) is stored as `NaN`**. `isna()`, `fillna()` and `dropna()` work exactly as on other default columns. What changes is code that looked for the exact sentinel: `value is None` is now `False` for a missing SKU, while `pd.isna(value)` stays correct on every dtype. ## Only strings fit The `str` dtype **can hold only strings or missing values**. Writing anything else into an existing column raises an error: ```python ser = pd.Series(["a", "b"]) ser[1] = 2.5 # TypeError: Invalid value '2.5' for dtype 'str' ... ``` Passing non-strings to a constructor *with* `dtype="str"` is different: the values are converted to their string form. Inference without a dtype keeps a column that mixes strings and numbers as `object`, because it is not all text. ## Where object dtype still appears 1. You ask for it: `pd.Series([...], dtype=object)` or `.astype("object")` gives a NumPy object column, which holds any Python object; built with `dtype=object`, it also keeps `None` as `None`. 2. The values are not all strings, for example text mixed with integers. 3. Old code hard-codes `dtype="object"` in constructors; it keeps working but forgoes the new dtype. ## Why pandas changed the default The pandas documentation names these drawbacks of the old `object` default for text: - **Not specific**: an `object` column can mix strings with numbers, lists or `None`, so its dtype says nothing reliable about its content. - **Hard to select**: `select_dtypes` could not pick "just text" without also picking every other object column. - **Slow and large**: an array of Python objects costs a pointer plus a full Python string per value; PyArrow storage packs the characters contiguously. ## Reading the dtype in code - `print(ser.dtype)` prints `str`; the repr is `<StringDtype(na_value=nan)>`, and `storage='python'` appears in it when PyArrow is absent. - `ser.dtype == "str"` is the direct check on pandas 3.0. - `ser.dtype.storage` and `ser.dtype.na_value` expose the two settings. ## What stays the same The `.str` accessor (`lower`, `strip`, `contains`, `extract`, `split` and the rest) behaves as it did on `object` columns, and a missing entry stays missing through these methods. You can try the whole change on pandas 2.3 by setting `pd.options.future.infer_string = True` before running a test suite, which is the documented way to rehearse the upgrade.

  • How do you keep one column as object dtype in pandas 3.0?
    Pass `dtype=object` to the constructor or reader, or call `.astype("object")` on the column. That gives a NumPy object array that accepts any Python object; built that way from the start, it also keeps `None` as `None`, while casting an existing str column leaves its gaps as NaN. Reserve it for columns that genuinely mix types, because you give up the guarantee that every value is text and, with PyArrow storage, the speed and memory gains.
  • Can you rehearse the change before upgrading?
    Yes. pandas 2.3 ships the same dtype behind `pd.options.future.infer_string = True`. Setting it at the start of a test run makes constructors and readers infer `str`, so dtype checks, `None` tests and non-string assignments that would break on 3.0 fail on 2.3 first.
  • Does the str dtype work without PyArrow?
    Yes. With PyArrow missing, `StringDtype` resolves its storage to `"python"`: a NumPy object array of Python strings with NaN for missing values. The semantics are identical; only performance differs, staying close to the old object dtype.

  • Text columns in pandas 3.0 are still NumPy object dtype by default.
  • The default str column stores missing values as pd.NA.
  • A str column accepts any Python object, just like object dtype.
  • The str dtype requires PyArrow and cannot be created without it.
  • None in a str column stays None rather than becoming NaN.

pandas 2.x inferred text as NumPy object; pandas 3.0 infers the str dtype (StringDtype with NaN), which pandas 2.3 previews with pd.options.future.infer_string = True.