A pandas frame converted with convert_dtypes(dtype_backend='pyarrow') now reports dtypes like 'int64[pyarrow]' — what are these ArrowDtype columns, and what should downstream code check?
answer
- a column backed by PyArrow memory
- type name plus a bracketed suffix
- parametrised types need the class
- string[pyarrow] is not what it seems
- dtype equality checks break
basics
~20 sArrowDtype columns hold their data in PyArrow arrays instead of NumPy, so every type can carry missing values and gains Arrow types such as decimals and lists. Downstream, replace dtype string equality checks and convert explicitly at NumPy hand-offs.
solid answer
~30 sAn `ArrowDtype` column is backed by a PyArrow `ChunkedArray` rather than a NumPy array. You get one with a `'<type>[pyarrow]'` string such as `'int64[pyarrow]'`, with `pd.ArrowDtype(pa.decimal128(5, 2))` for parametrised types, or with `dtype_backend='pyarrow'` on `convert_dtypes`, `to_numeric` and readers. Every Arrow type can hold missing values, so integers with gaps stay integers, and many operations run on PyArrow compute functions. Downstream, check three things: `dtype == 'int64'` is now `False`, so use `pd.api.types.is_integer_dtype`; the alias `'string[pyarrow]'` gives `StringDtype`, not `ArrowDtype(pa.string())`, and the two compare unequal; and NumPy-only code needs an explicit `to_numpy(dtype=..., na_value=...)`. The docs still label `ArrowDtype` experimental.
code
python · 16 linesimport numpy as np
import pandas as pd
import pyarrow as pa
df = pd.DataFrame({"qty": [3.0, np.nan], "name": ["cap", None]})
arrow = df.convert_dtypes(dtype_backend="pyarrow")
print(arrow.dtypes.to_dict())
print(arrow["qty"].dtype == "int64")
print(pd.api.types.is_integer_dtype(arrow["qty"].dtype))
alias = pd.Series(["cap"], dtype="string[pyarrow]").dtype
print(alias == arrow["name"].dtype, type(alias).__name__)
price = pd.Series(["3.19", None]).astype(pd.ArrowDtype(pa.decimal128(5, 2)))
print(price.dtype)go deeper
Recall that a '[pyarrow]' suffix means the column's data lives in PyArrow arrays, and that every such type can hold missing values.
Explain the creation routes: dtype strings, pd.ArrowDtype for parametrised types, astype, and dtype_backend='pyarrow' on convert_dtypes, to_numeric and readers.
Show what breaks after conversion: dtype equality checks, the string[pyarrow] alias mismatch, and NumPy hand-offs that need to_numpy with an explicit na_value.
Weigh the richer types and missing-value support against an experimental dtype and a hard PyArrow dependency before letting Arrow columns spread across shared code.
## What an `ArrowDtype` column is pandas columns normally store values in a NumPy array. An **`ArrowDtype`** column instead stores them in a PyArrow `ChunkedArray`, through pandas' extension-array interface. The dtype object wraps a PyArrow data type, available as `dtype.pyarrow_dtype`, and prints as that type followed by `[pyarrow]`: `int64[pyarrow]`, `double[pyarrow]`, `bool[pyarrow]`, `string[pyarrow]`. What changes compared with NumPy-backed columns: - **Missing values for every type.** Arrow keeps a validity mask per array, so an integer column with gaps stays an integer column instead of becoming `float64`. - **More types.** Decimals, lists, structs, maps, dates, times and time-zone-aware timestamps exist as real column types. - **Arrow compute.** Numeric aggregations and arithmetic, comparisons, string methods and datetime methods are accelerated with PyArrow compute functions where available. - **A dependency.** PyArrow must be installed. pandas declares it as an optional dependency, not a required one, so nothing installs it for you. The `ArrowDtype` docstring carries a warning that it is "considered experimental" and that "parts of the API may change without warning". ## How a column becomes one | Route | Example | |---|---| | dtype string | `pd.Series([1, 2, None], dtype='int64[pyarrow]')` | | parametrised type | `pd.ArrowDtype(pa.decimal128(5, 2))`, `pd.ArrowDtype(pa.list_(pa.string()))` | | cast | `s.astype('int64[pyarrow]')` | | whole frame | `df.convert_dtypes(dtype_backend='pyarrow')` | | parsing text | `pd.to_numeric(s, errors='coerce', dtype_backend='pyarrow')` | | from Arrow | `table.to_pandas(types_mapper=pd.ArrowDtype)` | Readers such as `read_csv` and `read_parquet` accept the same `dtype_backend='pyarrow'`; the I/O lessons cover their other arguments. ## The string trap Three similar spellings produce two different dtypes: 1. `dtype='string[pyarrow]'` gives `pd.StringDtype('pyarrow')`, a pandas string dtype that happens to use PyArrow storage. Its methods can return NumPy-backed nullable results: `str.contains` returns `boolean`. 2. `dtype=pd.ArrowDtype(pa.string())` gives a true `ArrowDtype`, and `str.contains` returns `bool[pyarrow]`. 3. `convert_dtypes(dtype_backend='pyarrow')` produces the second kind, which prints as `string[pyarrow]` even though the string alias of that name produces the first kind. The two compare unequal, so code that checks `dtype == 'string[pyarrow]'` or builds an empty frame with that alias can disagree with a converted frame. The lesson on text columns covers pandas' string dtypes themselves. ## What downstream code should check 1. **Dtype comparisons.** `pd.ArrowDtype(pa.int64()) == 'int64'` is `False`. Replace equality checks with `pd.api.types.is_integer_dtype`, `is_float_dtype` or `is_numeric_dtype`, which recognise Arrow types. `select_dtypes(include='number')` also matches them. 2. **Mixed frames.** Arithmetic between an `int64[pyarrow]` Series and a NumPy `int64` Series returns `int64[pyarrow]` in either order, and a merge on keys of those two dtypes matches values. Results keep the Arrow type, so a frame can drift towards Arrow one column at a time; decide whether that is intended. 3. **Hand-offs to NumPy-only code.** A model library or a hand-written NumPy routine needs a NumPy array. Convert explicitly with `to_numpy(dtype='float64', na_value=np.nan)` so you choose what a missing value becomes. 4. **Parametrised types.** A decimal column must be declared with `pd.ArrowDtype(pa.decimal128(precision, scale))`; the string `'decimal128(5, 2)[pyarrow]'` raises `NotImplementedError`, because type parameters cannot be passed in the string. 5. **Tests.** Assert dtypes in the test suite against the backend the pipeline uses, so a change of backend fails visibly. ## When the conversion is worth it - Integer identifiers, counts and flags with gaps, which NumPy dtypes would turn into floats or `object`. - Money held as exact decimals rather than binary floats. - Nested data, such as a list of tags per row, that would otherwise sit in an `object` column of Python lists. - Frames that are read from and written back to Parquet, where Arrow types carry over without remapping. It is less attractive when the next step is NumPy-only numerical code, or when some runtime in the deployment cannot install PyArrow. ## A worked example ```python import numpy as np import pandas as pd import pyarrow as pa s = pd.Series([1, 2, None], dtype="int64[pyarrow]") print(s.dtype == "int64", pd.api.types.is_integer_dtype(s.dtype)) print((s + pd.Series([1, 1, 1])).dtype) print(s.to_numpy(dtype="float64", na_value=np.nan)) ``` The first line prints `False True`, the second `int64[pyarrow]`, and the third a NumPy float array with `nan` in the gap. ## Mistakes interviewers listen for - Treating `'string[pyarrow]'` and `pd.ArrowDtype(pa.string())` as the same dtype. - Keeping `dtype == 'int64'` checks after switching backend. - Assuming PyArrow-backed and NumPy-backed columns cannot be combined. - Forgetting the PyArrow dependency on every machine that runs the code.
- Why would you build a column with pd.ArrowDtype rather than a '[pyarrow]' string?A string alias works for simple types such as `int64` or `bool`, but pandas refuses type parameters inside the string: `'decimal128(5, 2)[pyarrow]'` raises `NotImplementedError`. Decimals with precision and scale, or a `list_` of strings, have to be built with `pd.ArrowDtype(<pyarrow type>)`. The class also avoids the `'string[pyarrow]'` alias, which yields `StringDtype` rather than `ArrowDtype`.
- What happens if code using 'int64[pyarrow]' runs where PyArrow is not installed?Constructing the dtype fails, because an ArrowDtype cannot exist without PyArrow, and pandas lists PyArrow as an optional extra rather than a required dependency. Code that works on a laptop with PyArrow can therefore fail on a slimmer production image, so pin PyArrow in the project's own dependencies.
- dtype='string[pyarrow]' and pd.ArrowDtype(pa.string()) create the same dtype
- A check like dtype == 'int64' still matches an int64[pyarrow] column
- PyArrow-backed and NumPy-backed columns cannot be used in one expression
- An int64[pyarrow] column with a gap is upcast to float like NumPy int64
- ArrowDtype works without PyArrow installed because pandas bundles it