skip to content

In pandas 3.0, why can `orders.groupby('customer_id')['amount'].agg(np.std)` return different numbers from `.agg('std')`, and which should you pass?

level: middleimportance: should knowfreq 45%

answer

  1. no more quiet substitution
  2. the alias calls the groupby method
  3. NumPy's ddof default
  4. one Python call per group

basics

~20 s

In pandas 3.0 agg calls a NumPy or builtin function as given instead of swapping in the pandas method. np.std then gives the population deviation (ddof=0) while 'std' uses ddof=1, and np.median returns NaN for groups with NaN. Pass string aliases.

solid answer

~40 s

Up to pandas 2.x, `agg`, `apply` and `transform` quietly replaced NumPy functions such as `np.sum`, `np.std` and `np.median`, and builtins such as `sum` and `max`, with the matching groupby method; 2.1 deprecated that. In pandas 3.0 the function you pass is called on each group's Series. `np.std` on a Series defers to `Series.std` with NumPy's default `ddof=0`, so it returns the population standard deviation, while `'std'` uses pandas' `ddof=1`. `np.median` and the builtin `sum` do not skip NaN, so a group with a missing amount yields NaN. Even `np.sum`, which gives the same totals, runs as a Python call per group instead of the optimised groupby code, which is far slower with many groups. Pass string aliases, `'sum'`, `'std'`, `'median'`, or call the methods directly.

code

python · 12 lines
python
import numpy as np
import pandas as pd

orders = pd.DataFrame({
    "customer_id": ["c1", "c1", "c1", "c2", "c2"],
    "amount": [20.0, np.nan, 10.0, 35.0, 15.0],
})
gb = orders.groupby("customer_id")["amount"]

for func in ["sum", np.sum, sum, "std", np.std, "median", np.median]:
    label = func if isinstance(func, str) else f"{func.__module__}.{func.__name__}"
    print(f"{label:<16}", gb.agg(func).round(2).tolist())

go deeper

for a junior

Recall the rule for pandas 3.0: pass string aliases such as 'sum' and 'std' to agg, not np.sum or the builtin sum.

for a middle

Explain that 3.0 calls the given callable per group, that np.std then uses ddof=0, and that np.median and builtin sum propagate NaN.

for a senior

Show how you audit a 2.x codebase for substituted callables in lists, dicts and named aggregation, and why np.sum is a speed regression even when totals match.

for a principal

Make string aliases the house rule for aggregations and add a lint or review check, so a version upgrade cannot silently change reported statistics.

## What changed in pandas 3.0 `agg` (with `apply` and `transform`) accepts either a **string alias**, such as `'sum'`, or a **callable**, such as `np.sum`. For years pandas treated a set of well-known callables specially: - In pandas 2.x and earlier, passing `np.sum`, `np.mean`, `np.std`, `np.median` and their relatives, or the builtins `sum`, `min` and `max`, made pandas run its own groupby method instead, exactly as if you had written the string alias. - pandas 2.1 deprecated this substitution and recommended string aliases. - **In pandas 3.0 the substitution is gone.** A callable is called as given, once per group, on that group's Series. Nothing warns. Code that was correct on 2.x can therefore return different numbers on 3.0 without any change to the code. ## What each function does now For a customer whose amounts are `20.0, NaN, 10.0`: | Passed to `agg` | What runs per group | Result | |---|---|---| | `'sum'` | groupby `sum`, skips NaN | 30.0 | | `np.sum` | `np.sum(series)`, which defers to `Series.sum` | 30.0, but slower | | `sum` (builtin) | Python adds the values one by one | NaN | | `'std'` | groupby `std`, `ddof=1` | 7.07 | | `np.std` | `np.std(series)`, which defers to `Series.std(ddof=0)` | 5.0 | | `'median'` | groupby `median`, skips NaN | 15.0 | | `np.median` | NumPy median over the raw values | NaN | The pattern has two causes, one per layer: 1. **NumPy's own rules.** When given a pandas object, `np.sum`, `np.mean` and `np.std` look for a method of the same name and call it, passing NumPy's defaults; NumPy's `ddof` default is 0, so `np.std` yields the **population** standard deviation, while pandas' `std` defaults to the **sample** one, `ddof=1`. `np.median` has no such method lookup: it converts to an array, and a NaN in the array makes the median NaN. 2. **Python's own rules.** The builtin `sum` adds values in order, and any NaN makes the total NaN. The builtins `max` and `min` compare values with NaN, so their result depends on where the NaN sits in the group. ## Why string aliases are also faster - A string alias dispatches to pandas' groupby implementation, which computes all groups in one pass over the data. - A callable is called once per group, through a Python-level loop. With tens of thousands of groups, that is tens of thousands of function calls and Series constructions, often an order of magnitude slower or worse. - So `np.sum`, which still produces the right totals, is a performance regression on 3.0 even where it is not a correctness one. ## Where the callables hide The change applies wherever `agg` takes a function, and some spellings are easy to miss in review: - in lists: `agg([np.sum, 'mean'])`, where the output column is still labelled `sum`, so the label does not reveal which code ran; - in dicts: `agg({'amount': np.median})`; - in named aggregation: `total=('amount', np.sum)`; - in builtins passed without `np.`: `agg(max)`, `agg(sum)`. ## Custom functions still have their place The lesson is not that callables are wrong, only that a callable now means exactly what it says: - Use a **string alias** whenever one exists. The built-in groupby aggregations include `sum`, `mean`, `median`, `min`, `max`, `std`, `var`, `sem`, `prod`, `count`, `size`, `nunique`, `first`, `last`, `quantile`, `idxmin` and `idxmax`. - Use a **callable** for logic that has no alias, such as a range `lambda s: s.max() - s.min()`, accepting that it runs per group. - When a NumPy function's own semantics are wanted, call it on purpose and say so in a comment, so a reviewer does not "fix" it back to an alias. ## Fixing it 1. Replace each NumPy or builtin callable with its string alias: `'sum'`, `'mean'`, `'std'`, `'median'`, `'min'`, `'max'`. 2. Where NumPy's semantics were wanted, say so explicitly: `.agg('std', ddof=0)` or `.std(ddof=0)` for the population deviation. 3. Keep genuine custom functions as callables, but know they run per group in Python. 4. On a 2.1 to 2.3 install, the deprecation warning pandas emits for each substituted callable is a ready-made list of the call sites to change before upgrading. Applying the same rule in `transform` and `apply` is covered with group transforms.

  • If NumPy's population standard deviation is what you want, how do you ask for it cleanly?
    Say so in pandas terms: `.agg('std', ddof=0)` or `.std(ddof=0)`, because the alias form passes extra keyword arguments to the groupby method. In named aggregation, which does not forward arguments, use `lambda s: s.std(ddof=0)` or a `functools.partial`.
  • Does the change reach callables inside named aggregation, such as `total=('amount', np.sum)`?
    Yes. Named aggregation hands the callable to the same machinery, so in pandas 3.0 `np.sum` runs once per group on that group's Series. The totals match, but it is slower than `total=('amount', 'sum')`, and `np.median` or `np.std` there change the numbers just as they do elsewhere.
  • How do you find these call sites before upgrading a codebase to pandas 3.0?
    Run the test suite on pandas 2.1 to 2.3, where each substituted NumPy or builtin callable triggers a deprecation warning, and treat those warnings as errors. Then search for `agg(np.`, `agg([np.` and builtins such as `agg(sum)` that tests may not exercise.

  • np.std in agg gives the same sample deviation as 'std'.
  • pandas 3.0 still maps np.median to the groupby median method.
  • The builtin sum skips NaN inside agg like the pandas method.
  • A result column labelled sum proves pandas' own sum ran.
  • String aliases are only a style choice with no runtime effect.

Assumes pandas 3.0: 2.x replaced NumPy and builtin callables in agg, apply and transform with pandas methods (deprecated in 2.1); 3.0 calls them as given.