In pandas 3.0, why does `df.loc[mask, 'fee'] = 2.5` raise TypeError on an int64 fee column, and how do you fix it?
answer
- the column will not widen itself
- an in-place write keeps the dtype
- lossless values still pass
- decide the dtype, then write
basics
~10 sIn pandas 3.0 an in-place write whose value the column's dtype cannot hold raises TypeError instead of upcasting the column. Cast first, for example df['fee'] = df['fee'].astype('float64'), then repeat the loc write.
solid answer
~40 sIn pandas 3.0, setitem-like writes keep the target column's dtype. If the value cannot be stored losslessly, pandas raises `TypeError: Invalid value '2.5' for dtype 'int64'` instead of widening the column to `float64`. That applies to `loc`, `iloc`, `at`, `iat`, Series `[]=` and in-place methods such as `where(..., inplace=True)`, and even to a full-column `df.loc[:, 'fee'] = 2.5`. Lossless values still pass: `30.0` is stored as the integer 30. Two cases widen rather than raise: a scalar NaN written into an integer column, and replacing the whole column with `df['fee'] = ...`, which is not an in-place write. The fix is to choose the dtype on purpose: `df['fee'] = df['fee'].astype('float64')` and then the masked write, or round the values if fees really are whole units.
code
python · 20 linesimport numpy as np
import pandas as pd
payments = pd.DataFrame(
{"amount": [120.0, 9800.0, 45.5, 15000.0], "fee": [3, 25, 1, 40]},
index=pd.Index([101, 102, 103, 104], name="payment_id"),
)
large = payments["amount"] > 5000
try:
payments.loc[large, "fee"] = 2.5
except TypeError as err:
print(err) # Invalid value '2.5' for dtype 'int64'
payments.loc[large, "fee"] = 30.0 # lossless: stays int64
print(payments["fee"].dtype) # int64
payments["fee"] = payments["fee"].astype("float64")
payments.loc[large, "fee"] = 2.5 # now allowed
print(payments["fee"].tolist()) # [3.0, 2.5, 1.0, 2.5]go deeper
Recall that in pandas 3.0 a column keeps its dtype on a write, so storing 2.5 into an integer column raises TypeError until the column is cast.
Explain which writes are covered, why integral floats pass, why a scalar NaN widens instead, and why df['fee'] = ... is not an in-place write.
Treat each new TypeError during a 2.x to 3.0 migration as a possible data bug before casting, and put the cast where the column is created, not before every write.
Use the stricter rule as a schema contract: declare column dtypes at ingestion so writes can never change them, and review casts as deliberate schema changes.
## What changed Every pandas column has one **dtype**, the type of all its values: `int64`, `float64`, `bool`, `str`, and so on. Older pandas let a write change that dtype silently. If you stored `2.5` into an `int64` column, pandas **upcast** the whole column to `float64` to make room, and the code carried on. pandas 2.1 deprecated that behaviour with a `FutureWarning` ("Setting an item of incompatible dtype is deprecated..."), and **pandas 3.0 enforces it**: an in-place write that would need a wider dtype raises ```plaintext TypeError: Invalid value '2.5' for dtype 'int64' ``` The change is pandas Enhancement Proposal 6, "ban upcasting". The reasoning is that a write is meant to change **values**, not the **schema**. A silent upcast could turn an ID column into floats or a numeric column into `object`, and the cause surfaced many steps later. ## Which writes are covered The rule applies to **setitem-like** operations, the ones that write into existing storage: - `df.loc[mask, "fee"] = 2.5` and `df.iloc[rows, 1] = 2.5` - `df.at[102, "fee"] = 2.5` and `df.iat[1, 1] = 2.5` - `s[label] = value` on a Series - a **full-column** `df.loc[:, "fee"] = 2.5`, which pandas also treats as an in-place write - in-place methods such as `where(..., inplace=True)` and `clip(..., inplace=True)` with a value the column cannot hold The same rule protects 3.0's default `str` text dtype: writing the integer `5` into a `str` column raises `TypeError`, because that dtype holds only strings or missing values. ## What still succeeds | Write into `int64` `fee` | Outcome in pandas 3.0 | |---|---| | `df.loc[mask, "fee"] = 30.0` | Succeeds; stored as integer 30, still `int64` | | `df.loc[mask, "fee"] = 2.5` | `TypeError` | | `df.loc[mask, "fee"] = "waived"` | `TypeError` | | `df.loc[mask, "fee"] = np.nan` | Succeeds; column becomes `float64` | | `df["fee"] = 2.5` | Succeeds; whole column replaced, now `float64` | Three points hide in that table: 1. **Lossless is fine.** pandas checks whether the value fits. `30.0` equals the integer 30, so it is stored as an integer; an array of integral floats such as `[2.0, 3.0]` passes too. 2. **A scalar NaN is a deliberate exception.** The source carves it out, so writing `np.nan` into an integer column still widens it to `float64`. The same happens when a right-hand Series introduces NaN through alignment. 3. **`df["fee"] = ...` is not an in-place write.** It replaces the column with a new array, so the new array's dtype wins. The same goes for a method that returns a new object: `df["fee"].where(~mask, 2.5)` returns a `float64` Series you can assign back. ## How to fix it Decide what the column **should** be, then write: - **Fees can hold fractions:** cast once, then write. ```python df["fee"] = df["fee"].astype("float64") df.loc[mask, "fee"] = 2.5 ``` - **Fees are whole units by contract:** the 2.5 is the bug. Round or reject the incoming value instead of widening the column. - **The column needs missing values but must stay integer:** cast to the nullable `Int64` dtype; it holds `pd.NA` but still refuses `2.5`. What does **not** fix it: switching the deprecated `mode.copy_on_write` option (it no longer has any effect and is unrelated to dtypes), or calling `df["fee"].astype("float64")` without assigning the result back. ## How pandas decides whether a value fits The check runs on the value being written, not on the column as a whole: - **Integers** fit an integer column when they lie inside the dtype's range; a value too large for `int8`, say, is refused like 2.5. - **Floats** fit an integer column only when they are whole numbers, so `30.0` passes and `30.5` does not. - **Strings** never fit a numeric column, and non-strings never fit a `str` column. - **Booleans** are checked the same way: writing `1` into a `bool` column raises `TypeError` too. This is stricter than a raw NumPy array, which truncates `2.5` to `2` when it is stored into an integer array. pandas refuses rather than lose the fraction. ## Diagnosing it during a migration Code that ran on pandas 2.x may now raise on a line that looks innocent: 1. Read the message: it names the **value** and the **target dtype**. 2. Check whether the value is legitimate. Often the upcast was hiding a real data bug, such as a fractional fee. 3. If it is legitimate, add the explicit cast **where the column is created** (`astype`, or a `dtype=` argument on the reader) so every later write is consistent. 4. Search for in-place `where`, `mask` and `clip` calls on integer columns with float arguments; they raise for the same reason.
- Why does writing `np.nan` into an int64 column not raise in pandas 3.0, when writing 2.5 does?The source makes a scalar NaN an explicit exception to the no-upcast rule, so the column is widened to `float64` instead. It is the only way a NumPy integer column can record a missing value. If the column must stay integer with gaps, cast it to the nullable `Int64` dtype first.
- Why does `df['fee'] = 2.5` work on pandas 3.0 when `df.loc[:, 'fee'] = 2.5` raises?`df['fee'] = ...` replaces the column with a new array, so the new value's dtype becomes the column's dtype. `df.loc[:, 'fee'] = ...` is treated as a write into the existing column, so pandas keeps `int64` and raises because 2.5 does not fit.
- Does casting to `Int64` let the masked write of 2.5 succeed?No. `Int64` is the nullable integer dtype: it adds `pd.NA` for missing values but still holds only integers, so writing 2.5 raises as well. Fractional fees need a float dtype.
- Expects pandas 3.0 to upcast int64 to float64 on a masked write
- Thinks df.loc[:, 'fee'] = 2.5 replaces the column like df['fee'] = 2.5
- Believes writing 30.0 into an int64 column raises TypeError
- Tries to silence the error with the mode.copy_on_write option
- Calls df['fee'].astype('float64') without assigning the result back
Assumes pandas 3.0: pandas 2.1 to 2.3 upcast the column with a FutureWarning (earlier releases did so silently); pandas 3.0 raises TypeError.