skip to content

In pandas, what happens when df.astype({'qty': int, 'price': 'float64'}) meets a column with gaps or an unparseable value, and what does errors= change?

level: middleimportance: must knowfreq 60%

answer

  1. one mapping, one cast per column
  2. integers have no slot for NaN
  3. capital-I nullable integer
  4. errors='ignore' works column by column

basics

~10 s

astype with a dict casts only the named columns. A gap makes int raise IntCastingNaNError; use 'Int64'. A bad string raises ValueError naming the column, unless errors='ignore', which silently leaves just that column uncast.

solid answer

~30 s

`df.astype({'qty': int, 'price': 'float64'})` casts each named column to its own dtype and leaves the others alone; a key that is not a column raises `KeyError`. With the default `errors='raise'`, the first failing column aborts the call: `NaN` cast to `int` raises `IntCastingNaNError` (a `ValueError` subclass), and `'$1,200'` cast to `float64` raises `ValueError` with "Error while type casting for column 'price'" appended. For integers with gaps, ask for `'Int64'`, the nullable integer dtype, which still refuses non-whole values like `1.5`. `errors='ignore'` makes each failing column come back unchanged while the others are cast, so check `df.dtypes` afterwards.

code

python · 20 lines
python
import numpy as np
import pandas as pd

df = pd.DataFrame({
    "qty": [3.0, np.nan, 7.0],
    "price": ["$1,200", "85", "40"],
})

try:
    df.astype({"qty": int})
except ValueError as exc:
    print(type(exc).__name__)

try:
    df.astype({"qty": "Int64", "price": "float64"})
except ValueError as exc:
    print(exc)

out = df.astype({"qty": "Int64", "price": "float64"}, errors="ignore")
print(out["qty"].dtype, out["price"].tolist())

go deeper

for a junior

Remember that astype accepts a dict of column to dtype, returns a new frame, and that int cannot hold missing values while 'Int64' can.

for a middle

Explain the per-column loop: which exception each failure raises, that errors='ignore' returns a failing column unchanged, and that 'Int64' refuses non-whole floats.

for a senior

Show how you keep casts loud in production: default errors='raise', explicit dtype assertions after any errors='ignore', and no zero-filling just to make an integer cast pass.

for a principal

Weigh a shared schema mapping applied with astype at pipeline boundaries against ad hoc casts, and decide where integer columns with gaps should be nullable by convention.

## What a dict passed to `astype` does `DataFrame.astype(dtype, copy=..., errors='raise')` accepts either one dtype for the whole frame or a **mapping of column label to dtype**. With a mapping: - Only the columns named in the mapping are cast; every other column keeps its dtype. - Each key must be an existing column. A key such as `'nope'` raises `KeyError: Only a column name can be used for the key in a dtype mappings argument.` - pandas walks the columns one at a time and calls `Series.astype` on each named one, so a failure is reported per column. - The result is a **new** DataFrame. `astype` never changes `df` in place; you must assign the result (`df = df.astype({...})`). The dtype values can be Python types (`int`, `float`, `str`), NumPy dtypes or dtype strings (`'float64'`, `'Int64'`, `'category'`). ## Gaps: why `int` fails and `'Int64'` works A column of counts that has a missing value is usually stored as `float64`, with `NaN` marking the gap. Casting it with `int` asks for a NumPy integer dtype, and NumPy integers have no representation for a missing value. pandas refuses rather than inventing a number: `IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer.` The error is a subclass of `ValueError`, and its message itself suggests the fix: an integer type that supports missing values, such as `'Int64'`. | Requested dtype | `[3.0, NaN, 7.0]` | `[1.5, NaN]` | |---|---|---| | `int` / `'int64'` | raises `IntCastingNaNError` | raises `IntCastingNaNError` | | `'Int64'` | `3, <NA>, 7` as `Int64` | raises `TypeError` (1.5 is not a whole number) | | `'float64'` | unchanged | unchanged | Note the second column: `'Int64'` is **not** a rounding cast. It accepts only floats that are exactly whole and raises `TypeError: cannot safely cast non-equivalent object to int64` otherwise. If rounding is intended, round explicitly first. How `<NA>` then behaves in comparisons and arithmetic is the subject of the lesson on NA-aware dtypes; the point here is which cast succeeds. ## Bad values: what `errors` changes `astype`'s `errors` argument takes two values: - **`'raise'`** (default): the first column that fails stops the call. The error keeps the underlying message and adds the column name, for example `could not convert string to float: '$1,200': Error while type casting for column 'price'`. Nothing is returned, so `df` is exactly as before. - **`'ignore'`**: a column whose cast fails is returned **unchanged**, and the remaining columns are still cast. No warning is emitted. Because pandas walks the frame's columns in order, the order matters only for which error you see: with two bad columns, `errors='raise'` reports the first one in the frame's column order and never reaches the second. Passing a single dtype instead of a mapping, `df.astype('float64')`, casts **every** column, text columns included. With `errors='ignore'` each column that fails, such as a product name, comes back unchanged while the rest are cast, which makes a frame-wide cast even harder to audit. `errors='ignore'` is not a per-value option. It never produces `NaN` for the bad entries; it gives back the whole column in its old dtype. If you want bad entries turned into missing values, that is `pd.to_numeric(..., errors='coerce')` on that column, not `astype`. ## A safe way to cast several columns 1. Fix what is fixable first: strip formatting, coerce known-dirty numeric columns with `pd.to_numeric`. 2. Cast with a mapping and the default `errors='raise'`, so a surprise fails the job. 3. Use `'Int64'` for integer columns that may have gaps, and `int` only where gaps are impossible. 4. If you do use `errors='ignore'`, assert the resulting dtypes immediately, for example `assert df.dtypes['price'] == 'float64'`, because the call itself will not tell you which columns were skipped. ```python import pandas as pd import numpy as np df = pd.DataFrame({"qty": [3.0, np.nan, 7.0], "price": ["$1,200", "85", "40"]}) out = df.astype({"qty": "Int64", "price": "float64"}, errors="ignore") print(out.dtypes) ``` Here `qty` becomes `Int64` and `price` stays text, without any message. ## Mistakes interviewers listen for - Expecting `astype` to modify the frame in place. - Reaching for `astype(int)` on a column with gaps, then filling the gaps with `0` just to make the cast pass, which turns "unknown" into "zero". - Believing `errors='ignore'` sets unconvertible values to `NaN`. - Believing `'Int64'` rounds or truncates fractional values. - Treating `int` and `'Int64'` as spellings of the same dtype: the first is NumPy's default integer (`int64` on common platforms), the second is pandas' nullable integer extension dtype. A strong answer names the exception for each failure, says which dtype handles gaps, and treats `errors='ignore'` as a tool that needs an explicit dtype check after it.

  • Does astype have an errors='coerce' option like to_numeric?
    No. `astype`'s `errors` accepts only `'raise'` and `'ignore'`, and both act on whole columns. To turn individual bad entries into missing values, run `pd.to_numeric(df['price'], errors='coerce')` on that column, check how many values became `NaN`, and then cast the clean result with `astype` if you need a specific dtype.
  • What happens if the mapping names a column that does not exist?
    The call raises `KeyError` with the message that only a column name can be used as a key in a dtype mapping, and no column is cast. That makes a misspelt column name fail loudly, which is useful in pipelines whose input schema may drift.
  • Why might filling gaps with 0 before astype(int) be the wrong fix?
    It changes the meaning of the data: a missing quantity becomes a real zero, so sums, means and counts of zero all shift. Casting to `'Int64'` keeps the gap as missing while giving the column an integer dtype, so the decision about how to treat gaps stays explicit.

  • astype changes the DataFrame in place, so the result need not be assigned
  • errors='ignore' on astype turns the values it cannot convert into NaN
  • astype(int) on a float column with NaN casts the NaN to zero
  • astype('Int64') rounds 1.5 to the nearest whole number
  • int and 'Int64' are two spellings of the same dtype