In pandas, what does groupby filter return when dropping stores with fewer than five sales, and when is a transform mask better?
answer
- whole groups in or out
- one boolean per group
- original rows, labels and order
- dropna=False keeps the shape
- size broadcast as a mask
basics
~20 sdf.groupby('store').filter(lambda g: len(g) >= 5) returns the original rows of every store whose function returned True, with their original labels, order and columns. A mask from transform('size') >= 5 selects the same rows without a Python call per store.
solid answer
~40 s`DataFrameGroupBy.filter(func)` calls `func` on each store's sub-frame, which includes the `store` column, and keeps or drops the **whole** group depending on the single boolean it returns. The result is a subset of the original rows with their original index and order, not sorted by store and not keyed by it; `as_index` and `sort` do not change that. If `func` returns a Series, such as `g['amount'] > 100`, pandas raises `TypeError`. With `dropna=False` the failing groups stay in place filled with `NaN`, which upcasts integer columns to `float64`. Because `filter` makes a Python call per group, the vectorized form `df[df.groupby('store')['amount'].transform('size') >= 5]` is usually preferred for simple conditions; `filter` reads better when the condition is complex.
code
python · 13 linesimport pandas as pd
sales = pd.DataFrame({
"store": ["S1", "S2", "S1", "S3", "S1", "S2"],
"amount": [120, 50, 80, 30, 200, 150],
})
kept = sales.groupby("store").filter(lambda g: len(g) >= 2)
print(kept)
mask = sales.groupby("store")["amount"].transform("size") >= 2
print(sales[mask].equals(kept))
print(sales.groupby("store").filter(lambda g: len(g) >= 2, dropna=False))go deeper
Recall that groupby filter keeps or drops whole groups and returns the original rows, labels and columns of the groups that pass.
Explain why the function must return one boolean per group, what dropna=False does to the frame, and how transform('size') builds the same selection as a mask.
Show you pick a transform mask for simple conditions on large frames, keep the statistic as a column for auditing, and avoid size versus count mistakes.
Decide when group-level exclusions belong in shared pipeline code with a recorded threshold, so analyses dropping small stores stay comparable.
## What filter does A **filtration** is a grouped operation that returns a subset of the original rows. `DataFrameGroupBy.filter(func, dropna=True)` is the general one: pandas splits the frame by the key, calls `func` on each group's sub-frame, and keeps **all** rows of the groups for which `func` returned `True` and **none** of the others. That is the difference from an ordinary boolean mask on a column, which judges row by row. In a sales table, `sales.groupby('store').filter(lambda g: len(g) >= 5)` removes every store with fewer than five sales and leaves the rest untouched. ```python kept = sales.groupby("store").filter(lambda g: len(g) >= 5) big = sales.groupby("store").filter(lambda g: g["amount"].sum() > 10_000) ``` ## What comes back - **Rows**: the original rows of the passing groups, in their original order. pandas collects the positions of the kept groups, sorts them and takes them from the frame, so the output is not grouped by store. - **Index**: the original labels. Nothing is reset and no store level is added. - **Columns**: every column of the grouped object, including the grouping column `store`. Filtrations respect a column selection, so `sales.groupby('store')[['amount']].filter(...)` returns only `amount`. - **Arguments that do nothing here**: `as_index=False` and `sort=True` on `groupby` do not change a filtration's output, because no key index is built. Inside `func`, each sub-frame carries a `name` attribute holding its key, so `lambda g: g.name not in closed_stores` works. ## The function must return one boolean `func` is evaluated once per group and must return a **scalar boolean**. pandas squeezes a one-element result to a scalar, and treats a missing scalar as `False`, but anything longer is rejected: ```python sales.groupby("store").filter(lambda g: g["amount"] > 100) # TypeError: filter function returned a Series, but expected a scalar bool ``` The usual fix is to reduce the row condition to a group answer: `.any()`, `.all()`, `.sum() > n` or `.mean() > x`. ## Keeping the frame's shape with dropna=False By default the failing groups are dropped. `filter(func, dropna=False)` instead returns a frame of the **original shape** in which the rows of failing groups are replaced by missing values, the grouping column included. Two consequences: 1. Integer columns become `float64`, since a NumPy integer column cannot hold `NaN`. 2. The store name on those rows is gone too, so you cannot tell afterwards which store was blanked out. This mode suits plotting code that expects aligned rows; for most pipelines, a separate boolean column is clearer. ## filter or a transform mask The same selection can be written with a **broadcast group statistic** and ordinary boolean indexing: ```python n_sales = sales.groupby("store")["amount"].transform("size") kept = sales[n_sales >= 5] ``` | aspect | `groupby.filter(func)` | `df[g.transform(stat) >= x]` | |---|---|---| | work per group | one Python call | none: the statistic is computed in bulk | | speed with many small groups | slows down | stays fast | | conditions it can express | anything over the sub-frame | one broadcast statistic per condition | | keeps the condition for later | no | yes, the mask or the statistic can be a column | | result | original rows, labels and order | original rows, labels and order | The pandas user guide makes the same point: filtering with a UDF is often slower than chaining built-in methods. Reach for `filter` when the rule genuinely needs the whole sub-frame at once, for example a condition that combines several columns and a count. ## Missing keys and empty results Rows whose key is missing are not part of any group under the default `dropna=True` on `groupby`, so `filter` never calls the function for them and they are **absent from the result** even when every real group passes. A mask built from `transform` behaves the same way in effect, because the broadcast statistic on those rows is `NaN` and any comparison with it is `False`. If unknown stores must survive, group with `dropna=False` or fill the key first. When no group passes, `filter` returns an empty frame that still has all the columns, which keeps a later `concat` or write step from breaking. ## Pitfalls - Writing a row condition inside `filter` and expecting row-level filtering; it raises instead. - Expecting the result sorted or indexed by store. - Using `transform('count')` for a row count on a column with gaps: it counts non-missing values, while `'size'` counts rows. - Forgetting that `dropna=False` blanks the key column as well. The general difference between dropping whole groups and dropping individual rows before aggregating is covered in the library-neutral lesson on aggregate, transform and filter.
- How do you keep stores whose total sales exceed 10,000 without filter?Broadcast the total and mask: `sales[sales.groupby('store')['amount'].transform('sum') > 10_000]`. It returns the same rows as `filter(lambda g: g['amount'].sum() > 10_000)`, with the original labels and order, but computes the totals in one grouped pass rather than one Python call per store.
- How do you write a filter rule that needs two columns, such as enough sales and enough distinct days?Inside `filter`, combine two group-level answers with `and`: `lambda g: len(g) >= 5 and g['day'].nunique() >= 3`. As a mask, broadcast each statistic with `transform` and combine them with `&`: `(g['amount'].transform('size') >= 5) & (g['day'].transform('nunique') >= 3)`, where `g` is `sales.groupby('store')`.
- groupby filter keeps individual rows that meet a row-level condition
- filter returns its result sorted and indexed by the group key
- A filter function may return a boolean Series per row
- dropna=False keeps the failing groups' values unchanged
- transform('count') and transform('size') always give the same row count
- filter is the fastest way to drop small groups