skip to content

Text Columns

Since 3.0 text columns infer the str dtype instead of object, with NaN for missing, and the .str accessor runs string methods. Interviewers ask what broke and how extract works.

on this pageshow

questions

7

In pandas 3.0, what dtype does a new column of text get by default, and what can that column hold?

level: juniorimportance: must knowfreq 60%

answer

  1. no longer the catch-all dtype
  2. a StringDtype variant
  3. missing values print as NaN
  4. storage depends on PyArrow

basics

~10 s

In pandas 3.0 text is inferred as the str dtype, a StringDtype whose missing value is NaN, stored in PyArrow when installed and in Python objects otherwise. It holds only strings and missing values.

solid answer

~40 s

In pandas 3.0, constructors and readers such as `read_csv` infer a column of Python strings as the `str` dtype instead of NumPy `object`. `"str"` is the alias for `pd.StringDtype(na_value=np.nan)`: every missing entry, `None` included, is stored and shown as `NaN`, so missing-value handling matches the other default dtypes. Its storage is `"pyarrow"` when PyArrow is installed and `"python"` (a NumPy object array of Python strings) otherwise, and `ser.dtype.storage` tells you which. Unlike `object`, it accepts only strings or missing values, so assigning a number into it raises `TypeError`. Passing `dtype=object` still gives the old behaviour, and the `.str` methods work as before.

code

python · 11 lines
python
import pandas as pd

skus = pd.Series(["TSH-L-RED", "HOD-XL-BLU", None])

print(skus.dtype)            # str
print(skus.dtype.na_value)   # nan
print(skus.dtype.storage)    # 'pyarrow' if PyArrow is installed, else 'python'
print(skus[2] is None, pd.isna(skus[2]))

legacy = pd.Series(["TSH-L-RED", None], dtype=object)
print(legacy.dtype, legacy[1])

go deeper

for a junior

Recall that pandas 3.0 prints dtype: str for text columns, that missing values show as NaN, and that a number cannot be written into such a column.

for a middle

Explain that str is a StringDtype with na_value NaN, how its storage is picked from PyArrow availability, and how dtype=object still opts a column out.

for a senior

Name what the change breaks in existing code: equality checks against object, tests for None, assignments of non-strings, and how the pandas 2.3 future option exposes them early.

for a principal

Discuss whether PyArrow should be a required dependency so every environment gets the same str storage and performance, rather than silently falling back to python storage.

## What pandas 3.0 infers for text A **dtype** is the type pandas records for a whole column (a `Series`). Before pandas 3.0, a column built from Python strings was stored with NumPy's **`object`** dtype: an array of pointers to arbitrary Python objects, where nothing guaranteed the values were text. In pandas 3.0 a **dedicated string dtype is the default**. Whenever pandas has to infer a dtype and the input is strings (plus missing values), it now picks **`str`**: - in constructors: `pd.Series(["a", "b"])`, `pd.DataFrame({"sku": [...]})`, `pd.Index([...])`; - in readers such as `read_csv` and `read_parquet`, for columns read as text. The printed footer reads `dtype: str` instead of `dtype: object`. ## The str dtype is a StringDtype `str` is not a NumPy dtype. It is a pandas **extension dtype**, an instance of `pd.StringDtype`, configured with `na_value=np.nan`. Two attributes describe any `StringDtype`: | attribute | value for the default `str` dtype | what it controls | |---|---|---| | `na_value` | `nan` | the missing-value marker and its semantics | | `storage` | `"pyarrow"` or `"python"` | the array that holds the characters | You can name it explicitly with the short alias `dtype="str"`, which the documentation recommends over spelling out `pd.StringDtype(na_value=np.nan)`. ## Storage: pyarrow or python The storage is chosen when the dtype is created, from the option `mode.string_storage`, whose default `"auto"` resolves like this: - **PyArrow installed**: `storage="pyarrow"`. The strings live in a PyArrow string array, which brings large memory savings and speed gains for certain operations. - **PyArrow not installed**: `storage="python"`. The strings live in a NumPy object array of Python `str` objects, with roughly the performance of the old `object` dtype. The behaviour you program against (missing values, accepted values, `.str` methods) is essentially the same for both; speed and memory differ, plus edge cases such as invalid Unicode text, which PyArrow storage rejects. ## Missing values are NaN With `object` dtype, `None` and `np.nan` could both sit in a column. With `str`, **any missing input (`None`, `np.nan`, `pd.NA`) is stored as `NaN`**. `isna()`, `fillna()` and `dropna()` work exactly as on other default columns. What changes is code that looked for the exact sentinel: `value is None` is now `False` for a missing SKU, while `pd.isna(value)` stays correct on every dtype. ## Only strings fit The `str` dtype **can hold only strings or missing values**. Writing anything else into an existing column raises an error: ```python ser = pd.Series(["a", "b"]) ser[1] = 2.5 # TypeError: Invalid value '2.5' for dtype 'str' ... ``` Passing non-strings to a constructor *with* `dtype="str"` is different: the values are converted to their string form. Inference without a dtype keeps a column that mixes strings and numbers as `object`, because it is not all text. ## Where object dtype still appears 1. You ask for it: `pd.Series([...], dtype=object)` or `.astype("object")` gives a NumPy object column, which holds any Python object; built with `dtype=object`, it also keeps `None` as `None`. 2. The values are not all strings, for example text mixed with integers. 3. Old code hard-codes `dtype="object"` in constructors; it keeps working but forgoes the new dtype. ## Why pandas changed the default The pandas documentation names these drawbacks of the old `object` default for text: - **Not specific**: an `object` column can mix strings with numbers, lists or `None`, so its dtype says nothing reliable about its content. - **Hard to select**: `select_dtypes` could not pick "just text" without also picking every other object column. - **Slow and large**: an array of Python objects costs a pointer plus a full Python string per value; PyArrow storage packs the characters contiguously. ## Reading the dtype in code - `print(ser.dtype)` prints `str`; the repr is `<StringDtype(na_value=nan)>`, and `storage='python'` appears in it when PyArrow is absent. - `ser.dtype == "str"` is the direct check on pandas 3.0. - `ser.dtype.storage` and `ser.dtype.na_value` expose the two settings. ## What stays the same The `.str` accessor (`lower`, `strip`, `contains`, `extract`, `split` and the rest) behaves as it did on `object` columns, and a missing entry stays missing through these methods. You can try the whole change on pandas 2.3 by setting `pd.options.future.infer_string = True` before running a test suite, which is the documented way to rehearse the upgrade.

  • How do you keep one column as object dtype in pandas 3.0?
    Pass `dtype=object` to the constructor or reader, or call `.astype("object")` on the column. That gives a NumPy object array that accepts any Python object; built that way from the start, it also keeps `None` as `None`, while casting an existing str column leaves its gaps as NaN. Reserve it for columns that genuinely mix types, because you give up the guarantee that every value is text and, with PyArrow storage, the speed and memory gains.
  • Can you rehearse the change before upgrading?
    Yes. pandas 2.3 ships the same dtype behind `pd.options.future.infer_string = True`. Setting it at the start of a test run makes constructors and readers infer `str`, so dtype checks, `None` tests and non-string assignments that would break on 3.0 fail on 2.3 first.
  • Does the str dtype work without PyArrow?
    Yes. With PyArrow missing, `StringDtype` resolves its storage to `"python"`: a NumPy object array of Python strings with NaN for missing values. The semantics are identical; only performance differs, staying close to the old object dtype.

  • Text columns in pandas 3.0 are still NumPy object dtype by default.
  • The default str column stores missing values as pd.NA.
  • A str column accepts any Python object, just like object dtype.
  • The str dtype requires PyArrow and cannot be created without it.
  • None in a str column stays None rather than becoming NaN.

pandas 2.x inferred text as NumPy object; pandas 3.0 infers the str dtype (StringDtype with NaN), which pandas 2.3 previews with pd.options.future.infer_string = True.

open as a page

In pandas, how does Series.str.extract turn product SKU codes like 'TSH-L-RED' into separate columns, and what decides the result's shape?

level: middleimportance: must knowfreq 55%

basics

~20 s

Series.str.extract applies a regex to each string and returns one column per capture group, labelled by named groups, from the first match only. Non-matching or missing rows get missing values; expand=False with one group returns a Series.

open as a page

In pandas, why can sku.str.contains('BLK+GRN') miss the value 'CAP-OS-BLK+GRN' while sku.str.replace('+', '/') edits it correctly?

level: middleimportance: should knowfreq 45%

basics

~20 s

Series.str.contains treats its pattern as a regular expression by default (regex=True), so '+' means one or more K. Series.str.replace defaults to regex=False and replaces the literal '+'. Pass regex=False to contains for a literal match.

open as a page

In pandas, what does Series.str.split with expand=True return for SKU codes with different numbers of parts, and how do n and rsplit help?

level: middleimportance: should knowfreq 58%

basics

~20 s

With expand=True, Series.str.split returns a DataFrame with one column per piece, numbered from 0, and pads shorter rows with missing values on the right. n caps the number of splits and rsplit counts them from the end.

open as a page

In pandas 3.0, how do the default 'str' dtype, the opt-in 'string' dtype and object dtype differ for a text column?

level: middleimportance: should knowfreq 42%

basics

~20 s

In pandas 3.0, 'str' is a StringDtype using NaN, 'string' is a StringDtype using pd.NA, and object holds any Python objects. They differ in missing-value semantics and in the dtypes that .str methods and comparisons return.

open as a page

After upgrading a product-catalog pipeline to pandas 3.0, the check df['sku'].dtype == object returns False — what changed, and what else breaks with it?

level: seniorimportance: should knowfreq 48%

basics

~20 s

In pandas 3.0 the sku column is inferred as the str dtype, not object, so the equality is False. The same change turns None into NaN and makes writing a non-string into the column raise TypeError.

open as a page

Your team's shared pandas helper library must run on both pandas 2.x and 3.0 while services upgrade at different times; what rules do you set for code that touches text columns, and how do you test them?

level: principalimportance: should knowfreq 40%

basics

~20 s

Write text code that never depends on which dtype pandas inferred: test dtypes with is_string_dtype, select with ['object', 'string'], check missing values with pd.isna, cast to object before storing non-strings, and run the tests on 2.x, on 2.3 with infer_string enabled and on 3.0.

open as a page