skip to content

In pandas, what do infer_objects() and convert_dtypes() each do to a DataFrame's columns, and why does neither turn the string '42' into a number?

level: middleimportance: should knowfreq 40%

answer

  1. soft conversion versus nullable conversion
  2. object columns only, or every column
  3. inference is not parsing
  4. whole floats become Int64

basics

~20 s

infer_objects() re-infers object columns from the Python objects they hold; convert_dtypes() moves each column it can to a pd.NA-capable dtype such as Int64, Float64, boolean or string. Neither parses text: '42' stays a string, so use pd.to_numeric.

solid answer

~30 s

`infer_objects()` is a soft conversion: it looks only at `object` columns and re-runs the constructor's inference on the objects already there, so Python ints become `int64`, floats with `None` become `float64`, and in pandas 3.0 all-string columns become the `str` dtype; other columns are untouched. `convert_dtypes()` goes further and converts each column it can, not only `object` ones, to a dtype that supports `pd.NA`: `Int64` (including whole-valued floats with gaps), `Float64`, `boolean` and `string`, or ArrowDtype columns with `dtype_backend='pyarrow'`. Both inspect types, not text: `'42'` is a `str` object, so it stays text. Turning text into numbers is `pd.to_numeric`'s job.

code

python · 13 lines
python
import numpy as np
import pandas as pd

df = pd.DataFrame({
    "ints": pd.Series([1, 2, 3], dtype=object),
    "digits": pd.Series(["1", "2", "3"], dtype=object),
    "counts": [10.0, np.nan, 20.0],
})

for name, frame in [("before", df),
                    ("infer_objects", df.infer_objects()),
                    ("convert_dtypes", df.convert_dtypes())]:
    print(name, {c: str(t) for c, t in frame.dtypes.items()})

go deeper

for a junior

Recall that infer_objects() fixes object columns using the objects they hold, while convert_dtypes() moves columns to nullable dtypes; neither parses text.

for a middle

Explain the scope difference, object columns only versus every column, and the conversion table: whole floats to Int64, text to string, booleans to boolean.

for a senior

Show you would not rely on either to clean digit strings, and that convert_dtypes() changes missing markers and dtypes in ways that can break later dtype checks.

for a principal

Decide whether a pipeline normalises frames with convert_dtypes() at its boundary, and with which dtype_backend, so every stage sees one consistent set of dtypes.

## The problem both methods address A pandas column has `object` dtype when it holds arbitrary Python objects instead of a typed array. That happens after transposing a mixed frame, after building a Series from a list with mixed contents and then slicing away the odd values, or after `apply` returns Python objects. The data may already be clean integers, but the dtype does not say so, and numeric operations on `object` columns are slow and unpredictable. pandas offers two methods that improve such dtypes without being told the target, plus one function that actually parses text. ## `infer_objects()`: soft conversion of object columns `DataFrame.infer_objects()` (and `Series.infer_objects()`) attempts a **soft conversion** of `object`-dtype columns, "leaving non-object and unconvertible columns unchanged". It applies the same inference rules a constructor would apply to the objects already in the column: - Python `int` objects become `int64`. - Python `float` objects, with or without `None`, become `float64`. - Python `bool` objects become `bool`. - In pandas 3.0, Python `str` objects become the `str` dtype. In 2.x they stayed `object`. - A genuinely mixed column, say ints and strings, stays `object`. Columns that already have a real dtype are left alone, so a `float64` column stays `float64`. In pandas 3.0 the method's `copy` keyword is deprecated and ignored: it always returns a new object, using a lazy copy that defers copying until necessary. ## `convert_dtypes()`: move to dtypes that support `pd.NA` `DataFrame.convert_dtypes()` considers **each** column, not only `object` ones, and converts it where possible to "the best possible dtypes that support `pd.NA`". With the default `dtype_backend='numpy_nullable'`: | Column before | After `convert_dtypes()` | |---|---| | `int64` or `int32` | `Int64` or `Int32` | | `float64` holding only whole values and `NaN` | `Int64` | | `float64` with fractional values | `Float64` | | `object` of `True`/`False`/`None` | `boolean` | | `object` or `str` text | `string` (missing value `pd.NA`) | | `object` mixing ints and strings | stays `object` | Its keyword switches turn individual conversions off: `convert_integer=False` keeps whole-valued floats as `Float64`, and `convert_string`, `convert_boolean`, `convert_floating` and `infer_objects` do the same for their families. With `dtype_backend='pyarrow'` the targets become ArrowDtype columns instead, such as `int64[pyarrow]`, `double[pyarrow]` and `string[pyarrow]`. What `pd.NA` means for comparisons and arithmetic is taught in the lesson on NA-aware dtypes. ## Why `'42'` stays a string Both methods ask "what type are these objects?", never "what does this text say?". The string `'42'` is a Python `str`, and its type is text whether or not it looks numeric. So: 1. `infer_objects()` turns an all-string `object` column into `str` in pandas 3.0, not into numbers. 2. `convert_dtypes()` turns it into `string`, not `Int64`. 3. Only a parser converts it: `pd.to_numeric(col)` for numbers, `astype('int64')` when the column is known to be clean. This is the confusion interviewers probe: candidates expect "convert dtypes" to clean a CSV-like column of digit strings, and it does not. ## Using them in practice ```python import pandas as pd df = pd.DataFrame({ "a": pd.Series([1, 2, 3], dtype=object), "b": pd.Series(["1", "2", "3"], dtype=object), }) print(df.infer_objects().dtypes.to_dict()) print(df.convert_dtypes().dtypes.to_dict()) ``` The first line shows `a` as `int64` and `b` as the `str` dtype; the second shows `a` as `Int64` and `b` as `string`. Typical uses: - **`infer_objects()`** after an operation left typed data in `object` columns, when you want ordinary NumPy dtypes back. - **`convert_dtypes()`** when a team has chosen nullable dtypes and wants a frame normalised to them in one call. - **`pd.to_numeric`** first, when a column holds numeric text; afterwards `convert_dtypes()` can turn the parsed `float64` result into `Int64` if every value is whole. - **`select_dtypes(include='number')`** afterwards to pick the numeric columns; it matches `Int64`, `Float64` and pyarrow numeric columns as well as NumPy ones. ## Mistakes interviewers listen for - Expecting `convert_dtypes()` or `infer_objects()` to parse numeric text. - Assuming `infer_objects()` touches `float64` or `int64` columns. - Being surprised that `convert_dtypes()` turns a `float64` column of whole numbers into `Int64`. - Forgetting that `convert_dtypes()` changes the missing-value marker of text columns from `NaN` to `pd.NA`, which can alter later comparisons.

  • How do you stop convert_dtypes() from turning a whole-valued float column into Int64?
    Pass `convert_integer=False`. The column then becomes `Float64`, because `convert_floating` is still on. The docstring states that when both are enabled, integer dtypes are preferred whenever the floats can be cast to integers faithfully.
  • Which method would you use to get back ordinary NumPy dtypes after a transpose left everything as object?
    `infer_objects()`. It re-runs constructor inference on the object columns and returns `int64`, `float64`, `bool` and so on, without switching to the nullable extension dtypes that `convert_dtypes()` would introduce.

  • convert_dtypes() parses numeric strings such as '42' into integers
  • infer_objects() also converts float64 columns into nullable dtypes
  • convert_dtypes() only touches object columns
  • infer_objects() and convert_dtypes() are two names for the same conversion
  • In pandas 3.0, infer_objects() leaves an all-string object column as object

Assumes pandas 3.0: infer_objects() turns an all-string object column into the str dtype; in 2.x such a column stayed object.