skip to content

In pandas 3.0, what can DataFrame.fillna take as its value, and why fill a mixed-type frame with a per-column dict?

level: middleimportance: should knowfreq 50%

answer

  1. value is required, method is gone
  2. scalar, dict, Series or DataFrame
  3. unlisted columns stay untouched
  4. a number in a str column
  5. None means the dtype's own NA

basics

~20 s

fillna takes a scalar, a dict or Series keyed by column, or an aligned DataFrame. A dict fills only the columns it names, so a text column never receives a numeric 0, which in pandas 3.0 turns a str column into object.

solid answer

~40 s

`value` is the one required argument of `fillna` in pandas 3.0. A scalar fills every gap in every column; a dict or Series maps column labels to fill values and leaves unlisted columns alone, which is why `df.fillna(df.mean(numeric_only=True))` fills each numeric column with its own mean; a DataFrame fills cell by cell where index and columns align; a list raises `TypeError`. On a mixed frame the dict is the safe form: in pandas 3.0 text columns infer the `str` dtype, which holds only strings, so `df.fillna(0)` silently turns a gappy text column into `object`. pandas 3.0 also removed `fillna(method=...)` (use `ffill()`/`bfill()`), now accepts `value=None` to mean each dtype's own NA, and returns `self` with `inplace=True`.

code

python · 23 lines
python
import numpy as np
import pandas as pd

sensors = pd.DataFrame(
    {
        "device_id": ["d1", None, "d3"],
        "temp_c": [21.5, np.nan, 22.0],
        "humidity": [40.0, 42.0, np.nan],
    }
)
print(sensors["device_id"].dtype)                      # str in pandas 3.0
print(sensors.fillna(0)["device_id"].dtype)            # object: 0 is not a string

fill_values = {"device_id": "unknown", "temp_c": 0.0}
means = sensors.mean(numeric_only=True)                # Series keyed by column
cleaned = sensors.fillna(fill_values).fillna({"humidity": means["humidity"]})
print(cleaned["device_id"].dtype)                      # still str
print(cleaned.to_dict("list"))

try:
    sensors.fillna(method="ffill")                     # removed in pandas 3.0
except TypeError as err:
    print(type(err).__name__, err)

go deeper

for a junior

Recall that fillna needs a value and that a dict like {'temp_c': 0.0} fills only the column it names, leaving the rest alone.

for a middle

Explain the four accepted forms of value, why a Series of column means fills each column with its own mean, and that method= is gone in pandas 3.0.

for a senior

Show why a scalar fill on a mixed pandas 3.0 frame silently turns a str column into object, or raises midway with inplace=True, and check dtypes after every fill.

for a principal

Argue for explicit per-column fill dicts as a review convention, so adding a column to a source never inherits a fill value nobody chose.

## The signature in pandas 3.0 In pandas 3.0 `DataFrame.fillna` is declared as `fillna(value, *, axis=None, inplace=False, limit=None)`. Two things follow from that line: - **`value` is required.** Calling `df.fillna()` with nothing raises `TypeError` for a missing positional argument. - **There is no `method` keyword.** `fillna(method='ffill')` was deprecated in pandas 2.1 and removed in pandas 3.0; it now raises `TypeError` for an unexpected keyword. Directional filling lives in the separate `ffill()` and `bfill()` methods. Everything else about filling by value depends on what you pass as `value`. ## The four accepted forms of value | `value` passed | What gets filled | Typical use | |---|---|---| | scalar (`0`, `-1.0`, `"unknown"`) | every gap in every column | a frame with one dtype | | dict `{column: value}` | gaps in the named columns only | a mixed frame with per-column rules | | Series indexed by column label | each column with the value under its label | `df.fillna(df.mean(numeric_only=True))` | | DataFrame | each cell from the cell at the same index and column | patching from a reference table | Details that matter in practice: - A **dict or Series** leaves any column it does not name untouched; the docstring says plainly that values not in the mapping are not filled. - A **DataFrame** aligns on both axes, so a column of `df` absent from the filler keeps its gaps. - A **list or tuple** raises `TypeError` ("value parameter must be a scalar or dict"); a positional list of per-column values is not supported. - **`limit`** caps how many gaps are filled along the axis, so `limit=1` fills only the first gap in each column. - A dict value can be any scalar the column can hold: a `pd.Timestamp` for a datetime column, a string for a text column, a float for a measurement. Each column is filled with its own value in a single call. - Keys in the dict that are not columns of the frame are ignored rather than raising, so a misspelt column name silently fills nothing; re-measuring the gaps afterwards catches that. ## Why the dict is the safe form on a mixed frame Consider an IoT table with `device_id` (text), `temp_c` and `humidity` (floats). A scalar fill applies to all three columns, and that causes two different problems. 1. **Semantics.** A 0 written into a text or ID column is not a reading of anything. A dict such as `{"temp_c": 0.0, "humidity": 0.0, "device_id": "unknown"}` says exactly what each column receives. 2. **Dtype, new in pandas 3.0.** Text columns now infer the `str` dtype, which can hold only strings or missing values. Filling a gappy `str` column with the integer 0 cannot keep that dtype, so the default call (which returns a new frame) changes the column to `object` without a warning. With `inplace=True` the same fill raises `TypeError` ("Invalid value '0' for dtype 'str'"), and columns processed before the text column may already have been filled when it does. Downstream `.str` methods and string comparisons then meet a mix of strings and integers. In pandas 2.x the column was already `object`, so the same call only mixed types inside it. A dict also documents intent in code review: the reader sees which columns are filled and with what, and a column added to the source later is not filled by accident. ## value=None and the dtype's own NA marker pandas 3.0 accepts `value=None`. For a non-object column it means "use this dtype's own missing marker", so a `float64` column keeps its `NaN` and the call changes nothing there. For an `object` column it replaces `NaN` and `pd.NA` with `None`. Earlier releases did not accept `None` as a fill value. The other addition in 3.0 is support for `axis=1` with a dict or Series argument. ## Things fillna does not do - It does **not** carry values forward or backward; that is `ffill()` and `bfill()`, which take their own `limit` and `limit_area`. - It does **not** estimate values from neighbours; that is `interpolate()`. - It does **not** fill per group; filling inside groups is a group-transform task covered with group transforms. - It does **not** modify the original unless you reassign the result. With `inplace=True`, pandas 3.0 returns the modified object itself instead of `None`, but reassignment remains the idiomatic form. ## A worked pattern A cleaning step for the sensor frame usually reads like this: 1. Measure the gaps with `df.isna().sum()`. 2. Build one dict with an explicit value per column you intend to fill. 3. Call `df = df.fillna(fill_values)` and re-measure; columns you did not list still show their gaps, which is the point. 4. Check `df.dtypes` so a text column has not silently become `object`.

  • What does df.fillna(df.mean(numeric_only=True)) do to a text column in the same frame?
    Nothing. `mean(numeric_only=True)` returns a Series indexed only by the numeric column labels, and `fillna` fills a column only when the Series has an entry for it, so the text column keeps its gaps. Without `numeric_only=True`, `mean()` raises on a `str` column in pandas 3.0, so the argument is needed for a mixed frame anyway.
  • Why does df.fillna(df.mean(axis=1, numeric_only=True)) fill nothing?
    A row-mean Series is indexed by row labels, but `fillna` with a Series matches its index against column labels on the default axis. No row label equals a column label, so no column finds a fill value and every gap stays. Filling by row needs a different call, such as `num.where(num.notna(), row_means, axis=0)` on the numeric columns.
  • Old code calls df.fillna(method='bfill', limit=2); what is the pandas 3.0 replacement?
    `df.bfill(limit=2)`. The `method` keyword was removed from `fillna` in pandas 3.0, so the call raises `TypeError`. `bfill` takes `limit` with the same meaning, the maximum number of consecutive gaps filled, plus `limit_area` to restrict filling to inside or outside the valid values.

  • Passing a list of per-column values to fillna instead of a dict.
  • Assuming a dict fill also fills the columns the dict leaves out.
  • Still writing fillna(method='ffill'), removed in pandas 3.0.
  • Filling a whole mixed frame with 0, text and ID columns included.
  • Expecting a str column to keep its dtype after a numeric fill.

Assumes pandas 3.0: fillna(method=) is removed (deprecated in 2.1), value=None is newly accepted, and text infers the str dtype, so a numeric fill turns it into object.