In pandas 3.0, how do DataFrame.from_arrow and __arrow_c_stream__ move a frame to and from an Arrow-based tool, and what gets copied?
answer
- a protocol, not a file
- dunder method for export
- classmethod for import
- PyArrow does the conversion
- the index becomes a column
basics
~10 sIn pandas 3.0 a DataFrame exports itself through arrow_c_stream, the Arrow PyCapsule Interface, and DataFrame.from_arrow imports any object exposing that interface. Both go through PyArrow, and neither promises zero-copy.
solid answer
~40 spandas 3.0 supports the **Arrow PyCapsule Interface**, a protocol that lets libraries hand each other Arrow data in memory without knowing each other's classes. For export, `DataFrame` and `Series` implement `__arrow_c_stream__`, so any Arrow-aware tool can accept the frame directly. For import, `pd.DataFrame.from_arrow(obj)` and `pd.Series.from_arrow(obj)` accept a PyArrow object or anything with `__arrow_c_stream__` or `__arrow_c_array__`, and raise `TypeError` for anything else. Both directions rely on PyArrow: export uses `pyarrow.Table.from_pandas`, which stores a non-default index as a column, and import uses `Table.to_pandas` with its defaults, so an Arrow integer column with nulls arrives as `float64`. The docstring says export is not necessarily zero-copy, so plan for a copy.
code
python · 23 linesimport pandas as pd
import pyarrow as pa
prices = pd.DataFrame(
{"sku": ["A1", "B2", "C3"], "price": [9.99, 4.5, 12.0]},
index=pd.Index([101, 102, 103], name="item_id"),
)
# Export: an Arrow consumer reads the frame through __arrow_c_stream__
table = pa.RecordBatchReader.from_stream(prices).read_all()
print(table.schema.names)
# Import: a pyarrow Table or any object with __arrow_c_stream__
back = pd.DataFrame.from_arrow(table)
print(back.index.name, back.dtypes.astype(str).tolist())
counts = pa.array([1, None, 3])
print(counts.type, pd.Series.from_arrow(counts).dtype)
try:
pd.DataFrame.from_arrow({"sku": ["A1"]})
except TypeError as err:
print("TypeError:", err)go deeper
Know that pandas 3.0 can hand a DataFrame to Arrow-aware tools directly and read their results back with DataFrame.from_arrow, and that PyArrow must be installed.
Explain the two sides: arrow_c_stream for export, from_arrow for import of anything exposing the interface, and what TypeError a non-Arrow input raises.
Know what does not survive unchanged: a non-default index becomes a column, nulls in integers turn the column into float64, and export is not promised zero-copy.
The call for a lead is narrow: adopt the PyCapsule route over the deprecated interchange protocol and ad hoc file hand-offs, and pin a PyArrow version both sides accept.
## The problem the interface solves A data pipeline often moves one table between libraries: a pandas DataFrame goes to an Arrow-based query engine, and its result comes back to pandas. **Apache Arrow** is a columnar in-memory format, and the **Arrow PyCapsule Interface** is a small protocol on top of it. An object that can export Arrow data defines a dunder method that returns a `PyCapsule`, a Python wrapper around a C pointer to Arrow structures. A consumer calls the method and reads the data, without importing the producer's library. pandas 3.0 implements both sides. ## Export: `__arrow_c_stream__` `DataFrame.__arrow_c_stream__(requested_schema=None)` returns an Arrow C stream capsule. You rarely call it yourself; you pass the DataFrame to a function that accepts "any Arrow stream" and that function calls it. What happens inside: - pandas imports **PyArrow** (an `ImportError` if it is missing) and calls `pyarrow.Table.from_pandas(df)`; - the **index** follows PyArrow's default: a `RangeIndex` is not stored as a column, any other index is, under its name (PyArrow calls an unnamed one `__index_level_0__`); - `requested_schema`, when a consumer passes one, makes pandas cast to that schema during the conversion; - the docstring states plainly: "This conversion is not necessarily zero-copy." `Series.__arrow_c_stream__` does the same for one column and ignores the index. ## Import: `from_arrow` `pd.DataFrame.from_arrow(data)` is a classmethod added in pandas 3.0. It accepts a `pyarrow.Table` or **any** object with `__arrow_c_stream__` or `__arrow_c_array__`, which covers Arrow-based dataframe libraries and query engines without pandas knowing them. Anything else, such as a plain dict, raises `TypeError: Expected an Arrow-compatible tabular object ...`. `pd.Series.from_arrow` is the one-column counterpart for array-like objects. The conversion is PyArrow's `Table.to_pandas()` with its **default** settings, and `from_arrow` takes no options: - an Arrow `int64` column with nulls becomes `float64` with `NaN`, because the default conversion targets NumPy dtypes; - Arrow strings become the pandas `str` dtype; - pandas metadata stored in the schema is honoured, so a frame exported by pandas comes back with its index restored. If you want Arrow-backed pandas columns (`ArrowDtype`) instead, convert with PyArrow directly: `table.to_pandas(types_mapper=pd.ArrowDtype)`. Choosing dtype backends in general is covered with pandas casting and Arrow backends. ## What gets copied | Step | Copies? | |---|---| | Export through `__arrow_c_stream__` | not guaranteed zero-copy; pandas builds a PyArrow Table first | | Import with `from_arrow` | usually, since columns are converted to NumPy-backed pandas dtypes | | `int64` with nulls on import | yes, the values are rewritten as `float64` | | `to_numpy()` on the way to a NumPy-only library | depends on dtype, and the column names are lost | The interface removes the need for a file or a bespoke converter between libraries; it does not make pandas columnar underneath, so a NumPy-backed frame still has to be converted to Arrow somewhere. ## A round trip, step by step A typical exchange with an Arrow-based query engine looks like this: 1. **Hand over the frame.** Pass `prices` directly to the engine's function that accepts Arrow data; it calls `prices.__arrow_c_stream__()`, and PyArrow converts the frame, including the `item_id` index as a column. 2. **Run the query on the other side.** The engine works on its own Arrow memory; pandas is not involved. 3. **Bring the result back.** The engine returns an object of its own type. If that object implements `__arrow_c_stream__`, `pd.DataFrame.from_arrow(result)` converts it, again through PyArrow, into a pandas DataFrame. 4. **Check what changed.** Look at `back.dtypes` and `back.index`: nullable integers may have become `float64`, and an index is restored only if pandas' schema metadata travelled with the result; without it, `item_id` comes back as an ordinary column and the frame gets a default `RangeIndex`. Each crossing is a PyArrow conversion, so a pipeline that bounces the same table back and forth pays for it every time. ## Versus the older routes 1. **`__dataframe__`**, the Dataframe Interchange Protocol, is deprecated in pandas 3.0 and warns with `Pandas4Warning`; its warning text points to the Arrow PyCapsule Interface. 2. **`pa.Table.from_pandas(df)` and `table.to_pandas()`** still work and expose more options; `from_arrow` and `__arrow_c_stream__` are the library-neutral spelling. 3. **Writing Parquet or Feather** to hand data over is a file round trip, covered with pandas file formats. ## Before relying on it - Install PyArrow; both directions need it. - Decide whether the index should travel: `reset_index()` before export if the consumer must see it as an ordinary column with a clear name. - Check dtypes after `from_arrow`, especially nullable integers that arrive as `float64`.
- Why does an Arrow int64 column with nulls come back from from_arrow as float64?`from_arrow` calls PyArrow's `Table.to_pandas()` with default settings, which targets NumPy dtypes, and NumPy `int64` has no missing value, so the column becomes `float64` with `NaN`. To keep integers with `pd.NA`, convert with `table.to_pandas(types_mapper=...)` yourself, for example `pd.ArrowDtype`, or cast to `Int64` afterwards.
- What replaces df.__dataframe__() for handing a pandas frame to another dataframe library?The Dataframe Interchange Protocol behind `__dataframe__` is deprecated in pandas 3.0 and warns with `Pandas4Warning`. The replacement is the Arrow PyCapsule Interface: pass the DataFrame to a consumer that reads `__arrow_c_stream__`, and use `from_arrow` in the other direction.
- __arrow_c_stream__ always hands over the frame without copying
- from_arrow works without PyArrow installed
- The index is never part of the exported Arrow stream
- from_arrow keeps nullable integers as int64
- from_arrow accepts a plain dict of lists
Assumes pandas 3.0: 2.x had no from_arrow, so imports went through PyArrow's Table.to_pandas; 3.0 adds DataFrame.from_arrow and Series.from_arrow, documents __arrow_c_stream__ export and deprecates __dataframe__.