In pandas 3.0, how do the default 'str' dtype, the opt-in 'string' dtype and object dtype differ for a text column?
answer
- three containers for text
- which missing marker each uses
- what .str methods hand back
- comparisons with a missing entry
basics
~20 sIn pandas 3.0, 'str' is a StringDtype using NaN, 'string' is a StringDtype using pd.NA, and object holds any Python objects. They differ in missing-value semantics and in the dtypes that .str methods and comparisons return.
solid answer
~40 sAll three hold text in pandas 3.0. `"str"`, the inferred default, is `StringDtype(na_value=np.nan)`: only strings or NaN, `.str` methods that return booleans give NumPy `bool` with missing entries as `False`, integer results such as `.str.len()` come back `int64`, or `float64` when values are missing, and `==` treats a missing entry as unequal. `"string"` is `StringDtype(na_value=pd.NA)`, opt-in since pandas 1.0: the same methods return nullable `boolean` and `Int64`, and comparisons propagate `<NA>`. Both string dtypes take `storage="pyarrow"` or `"python"`. `object` is a NumPy array of arbitrary Python objects: it accepts mixed values and keeps `None` as `None`, but nothing guarantees the values are text. Use `"str"` by default, `"string"` when the frame already uses `pd.NA` nullable dtypes, and `object` only for genuinely mixed values.
code
python · 12 linesimport pandas as pd
values = ["L", None, "XL"]
default = pd.Series(values) # inferred str
nullable = pd.Series(values, dtype="string") # StringDtype with pd.NA
legacy = pd.Series(values, dtype=object) # NumPy object
for s in (default, nullable, legacy):
print(s.dtype, s.iloc[1])
for s in (default, nullable):
print(s.str.len().dtype, s.str.startswith("X").dtype)go deeper
Recall that pandas 3.0 has two string dtypes, str with NaN and string with pd.NA, and that object can still hold text.
Explain how the missing-value marker changes the dtypes .str methods and comparisons return, and how storage is a separate setting from na_value.
Show you can predict result dtypes in a pipeline mixing these columns, and know that dtype == 'string' matches both StringDtype variants.
Argue when a codebase should standardise on NaN semantics versus pd.NA semantics across all dtypes, since mixing them makes result dtypes harder to predict.
## Three containers for text pandas 3.0 offers three ways to store a column of text. Two are variants of one pandas **extension dtype**, `pd.StringDtype`, which differ only in their **missing-value marker**; the third is NumPy's general-purpose `object` dtype. | | `"str"` | `"string"` | `object` | |---|---|---|---| | how you get it | inferred by default, or `dtype="str"` | `dtype="string"` (opt-in since 1.0) | `dtype=object` or mixed values | | class | `StringDtype(na_value=np.nan)` | `StringDtype(na_value=pd.NA)` | NumPy `dtype('O')` | | missing marker | `NaN` | `<NA>` | `None` or `NaN`, as stored | | accepts | strings or missing | strings or missing | any Python object | | `.str` boolean result | NumPy `bool`, missing as `False` | nullable `boolean` with `<NA>` | depends on the values | | `.str` integer result | `int64`, or `float64` with gaps | nullable `Int64` | depends on the values | | `== "x"` on a missing entry | `False` | `<NA>` | `False` | ## The str dtype (NaN semantics) `str` is what pandas 3.0 infers for strings. It follows the same **NaN semantics** as the other default NumPy-backed dtypes: - missing entries are `NaN`, whatever you passed in (`None`, `np.nan` or `pd.NA`); - boolean string methods such as `.str.contains` or `.str.startswith` return plain NumPy `bool`, with `False` where the input was missing; - integer-returning methods such as `.str.len()` or `.str.count()` return `int64`, switching to `float64` when missing values are present, just as an integer column with gaps does; - comparisons return NumPy `bool`, and a missing value compares unequal, like `NaN`. ## The string dtype (pd.NA semantics) `"string"` has existed since pandas 1.0 and stays opt-in. It uses the **`pd.NA` scalar** and the nullable dtype family: - `.str.len()` returns `Int64`, `.str.contains()` returns `boolean`, each keeping `<NA>` for missing input; - comparisons **propagate** `<NA>` instead of answering `False`. This matters when a pipeline already standardises on `Int64`, `Float64` and `boolean` columns (for example through `convert_dtypes`), because results stay in that family instead of falling back to `float64` and `bool`. ## object dtype `object` is a NumPy array of references to Python objects. It is the only one of the three that accepts **mixed values** (a string next to an integer or a `dict`) and keeps `None` as `None`. The price is that nothing guarantees the content is text: string methods turn non-strings into missing values, and code reading the dtype cannot tell a text column from a column of arbitrary objects. ## The four StringDtype variants Both string aliases also have a **storage**, chosen from PyArrow availability unless you set it: 1. `StringDtype(storage="pyarrow", na_value=np.nan)`: what `"str"` means when PyArrow is installed. 2. `StringDtype(storage="python", na_value=np.nan)`: `"str"` without PyArrow. 3. `StringDtype(storage="pyarrow", na_value=pd.NA)`: `"string"` with PyArrow. 4. `StringDtype(storage="python", na_value=pd.NA)`: `"string"` without PyArrow. Inspect a column with `ser.dtype.storage` and `ser.dtype.na_value`. ## What text-returning methods give back Methods that return text, such as `.str.upper()`, `.str.strip()` or `.str.replace()`, keep the **caller's string dtype**: a `str` column gives a `str` result and a `"string"` column gives a `"string"` result, so the missing-value semantics you chose survive a chain of cleaning steps. The boolean and integer results in the table are where the variants part ways. Two consequences follow: - a chain such as `.str.strip().str.upper().str.len()` stays in one family from start to finish, NaN-based or `pd.NA`-based; - mixing a `"string"` column into a frame of default columns means its comparisons and counts come back nullable while its neighbours' come back as NumPy `bool`, `int64` or `float64`, which is easy to miss in later arithmetic. ## Checking which one you have - `ser.dtype == "str"` is `True` only for the NaN variant. - `ser.dtype == "string"` is `True` for **both** variants in pandas 3.0's implementation, so test `ser.dtype.na_value` when the missing-value semantics are what you care about. - `pd.api.types.is_string_dtype(ser.dtype)` is `True` for all three, which suits code that must also run on pandas 2.x. ## Choosing - Keep the inferred **`str`** for ordinary text; it needs no code and matches the default NaN semantics of the rest of the frame. - Choose **`"string"`** when the frame uses nullable dtypes and you want `<NA>` to flow through comparisons and string-method results. - Fall back to **`object`** only when a column really mixes types, and treat that as a data-quality smell worth fixing upstream.
- Why does .str.len() on a default str column return float64 sometimes?With NaN semantics an integer result has no way to hold a missing value, so when the input has gaps the lengths come back as `float64` with `NaN`, exactly as an integer column with gaps would. Without missing input the result is `int64`. The `"string"` dtype avoids this by returning nullable `Int64` with `<NA>`.
- Does ser.dtype == 'string' tell the two StringDtype variants apart?No. In pandas 3.0 a `StringDtype` compares equal to the text `"string"` whatever its `na_value`, so the default `str` column also passes. Compare with `"str"`, or read `ser.dtype.na_value`, which is `nan` for the default and `<NA>` for the opt-in variant.
- The str and string dtypes are two names for the same thing.
- The opt-in string dtype stores missing values as NaN.
- On a str column .str.contains returns NaN for missing entries.
- Only the object dtype can be backed by PyArrow.
- Comparing a missing entry in a str column with == returns <NA>.
Before 3.0 text defaulted to object and dtype='str' meant object; in pandas 3.0 'str' names the NaN-variant StringDtype, while 'string' (pd.NA) is unchanged since 1.0.