skip to content

Your team's shared pandas helper library must run on both pandas 2.x and 3.0 while services upgrade at different times; what rules do you set for code that touches text columns, and how do you test them?

level: principalimportance: should knowfreq 40%

answer

  1. 2.x infers object, 3.0 infers str
  2. check with is_string_dtype
  3. select object and string together
  4. pd.isna, never is None
  5. test 2.3 with infer_string on

basics

~20 s

Write text code that never depends on which dtype pandas inferred: test dtypes with is_string_dtype, select with ['object', 'string'], check missing values with pd.isna, cast to object before storing non-strings, and run the tests on 2.x, on 2.3 with infer_string enabled and on 3.0.

solid answer

~40 s

The breaking change is inference: pandas 2.x stores text as `object`, pandas 3.0 infers the `str` dtype, which holds only strings and uses `NaN` for missing values. A library used by both populations needs rules that hold on either side. Dtype checks use `pd.api.types.is_string_dtype(col.dtype)`, never `dtype == object`. Column selection uses `select_dtypes(include=['object', 'string'])`, because `include='str'` raises on 2.x and `include='object'` warns on 3.0. Missing checks use `pd.isna`, since 3.0 turns `None` into `NaN`. Code that stores numbers or objects into a text column calls `astype('object')` first, because the 3.0 `str` dtype raises `TypeError` on a non-string write. `astype(str)` is avoided where values can be missing: 2.x turns `NaN` into the string `'nan'`. The test matrix runs 2.x with its defaults, 2.3 with `pd.options.future.infer_string = True`, and 3.0.

code

python · 10 lines
python
import pandas as pd
from pandas.api.types import is_string_dtype

df = pd.DataFrame({"sku": ["A-1", "B-2", None], "qty": [3, 1, 2]})

print(pd.__version__, df["sku"].dtype)
print(df["sku"].dtype == object)          # True on 2.x, False on 3.0
print(is_string_dtype(df["sku"].dtype))   # True on both
print(list(df.select_dtypes(include=["object", "string"]).columns))
print(pd.isna(df.loc[2, "sku"]))          # True on both

go deeper

for a junior

Know that pandas 3.0 stores text in the str dtype and 2.x in object, and that pd.isna finds missing values on both.

for a middle

Replace dtype == object with is_string_dtype, select text columns in a version-safe way, and know why a non-string write raises on 3.0.

for a senior

Audit a helper module for every version-dependent text behaviour, including astype(str) and .values, and write tests that catch silent changes.

for a principal

Set the dual-version rules, the CI matrix with the 2.3 preview leg, and the date when the compatibility spellings and the 2.x leg are removed.

## Why this is a lead decision During an upgrade, one helper library is imported by services on both sides of the pandas 3.0 line. The same `read_csv` call hands it an `object` column in one service and a `str` column in the next. Nobody notices until a branch that tested `dtype == object` silently skips a column in production. The lead's job is to set rules that make the library's behaviour independent of the inferred dtype, and a test setup that proves it, so each service can upgrade on its own schedule. ## What actually differs | Concern | pandas 2.x default | pandas 3.0 default | |---|---|---| | Inferred dtype of text | `object` | `str` (a `StringDtype` using `NaN`) | | `col.dtype == object` | `True` | `False` | | `None` in the data | kept as `None` | coerced to `NaN` | | Writing `2.5` into a text column | allowed, column stays `object` | `TypeError` | | `astype(str)` on a column with `NaN` | `NaN` becomes the string `'nan'` | `NaN` stays missing | | `select_dtypes(include='str')` | raises `TypeError` | selects text columns | | `select_dtypes(include='object')` | selects text columns | still selects them, with a deprecation warning | | `col.values` on text | NumPy array | an `ExtensionArray` | pandas 2.3 can switch on the 3.0 behaviour with `pd.options.future.infer_string = True`, which is what makes a two-sided test setup possible before anyone upgrades. ## The rules I would write down 1. **Check dtypes by meaning, not by name.** `pd.api.types.is_string_dtype(col.dtype)` returns `True` for `object` and for the string dtypes, on both versions. A comparison against `object` or `'str'` is banned in review. 2. **Select text columns with both names.** `df.select_dtypes(include=['object', 'string'])` works on 2.x and on 3.0, and on 3.0 the `'string'` entry also picks up the default `str` dtype without the deprecation warning. 3. **Test missing values with `pd.isna`.** Code that looks for `None` stops matching on 3.0, where the value is `NaN`. `pd.isna` is correct for every dtype and sentinel; pass it a scalar when using it in an `if`. 4. **Make mixed columns explicit.** If a helper really needs to put numbers or objects into a text column, it calls `astype('object')` first. On 2.x that call does nothing; on 3.0 it is what keeps the write legal. 5. **Do not stringify with `astype(str)` where values can be missing.** On 2.x, missing values become the literal text `'nan'`, which then passes every not-null check. Either fill missing values deliberately first or keep the column as it is. 6. **Ask for NumPy arrays explicitly.** Use `to_numpy()`, not `.values`, at hand-offs to NumPy code, because on 3.0 `.values` returns an `ExtensionArray` for text. 7. **Declare text in constructors with `dtype='str'` when it must be text.** In 2.x that is treated as `object`, in 3.0 as the new dtype, so one spelling works on both. Hard-coded `dtype='object'` keeps text out of the new dtype on 3.0 and should be a deliberate exception. ## How I would test it - A CI matrix with three legs: the oldest supported 2.x release with defaults, pandas 2.3 with `pd.options.future.infer_string = True` set in the test configuration, and the current 3.0 release. - Golden-output tests that assert the values **and** the missing-value positions of key outputs, because the `'nan'` string and a skipped column both look like valid data. - Warnings turned into errors on the 3.0 leg, so `select_dtypes(include='object')` and other deprecated spellings fail the build instead of piling up. ## What the rules cost - Review effort: every text-handling helper has to be read once against the list, and new code needs the same check until the 2.x leg is dropped. - CI time: three legs instead of one for the duration of the upgrade window, and a failing 2.3-preview leg has to block merges, or it is ignored within a week. - A temporary escape hatch exists, `pd.options.future.infer_string = False` on 3.0, but the option's own description says it will be deprecated, so a library that relies on it only moves the problem. Once the last service is on 3.0, the 2.x leg and the compatibility spellings can go; the dtype and missing-value rules are worth keeping because they are correct on any version.

  • Why not pin every text column to dtype='object' so the library behaves like 2.x everywhere?
    It works, but it opts the library out of the 3.0 string dtype for every caller: no PyArrow-backed storage, no guarantee that a text column holds only strings, and columns that come back as `object` to services that have upgraded. It also has to be undone later. Writing dtype-independent code is the same effort once and does not need reverting.
  • A helper uses s.astype(str) to build lookup keys; what goes wrong on 2.x that the tests may miss?
    On 2.x every missing value becomes the text `'nan'`, so rows with no key all share the key `'nan'` and join to each other or to a real `'nan'` entry. On 3.0 they stay missing. A test that only compares row counts can pass on both; the golden output has to assert where the missing values are.

  • Checking for text columns with dtype == object in shared code
  • Using select_dtypes(include='str') in code that must still run on 2.x
  • Testing for missing text values with `is None`
  • Assuming astype(str) keeps missing values missing on every version
  • Relying on infer_string = False as the permanent fix for 3.0

Written for the pandas 2.x to 3.0 upgrade window: 3.0 infers the str dtype for text, 2.x infers object, and pandas 2.3 previews the 3.0 behaviour with pd.options.future.infer_string = True.