skip to content

Applying Functions

Series.map, DataFrame.map and apply call Python per element or row, while np.where, np.select and column arithmetic run vectorized. Interviewers ask you to replace an apply.

on this pageshow

questions

5

In pandas, what does `Series.map` return when you pass it a dict, and what happens to values the dict has no key for?

level: juniorimportance: must knowfreq 62%

answer

  1. same index, new values
  2. lookup, not a Python call
  3. missing key means missing value
  4. defaultdict or fillna afterwards

basics

~20 s

Series.map with a dict returns a new Series on the same index, each value replaced by its dict lookup. Any value without a key becomes NaN, unless the dict defines missing, which then supplies the default.

solid answer

~30 s

`s.map(mapping)` looks every value up in the dict and returns a new Series aligned to the original index; the source is untouched. A value with no key, including a near miss like `"ca"` against a key `"CA"`, comes back as NaN, and an integer result turns float64 once a NaN appears. I fix the gaps with `.fillna("Unknown")`, or pass a dict subclass with `__missing__` such as `defaultdict`, which pandas calls for the default. If I want unmapped values kept as they were, `replace(mapping)` is the right call instead. Then I count the NaNs, because a silent miss is the real risk.

code

python · 19 lines
python
from collections import defaultdict

import pandas as pd

loans = pd.DataFrame({
    "state": ["CA", "TX", "NY", "PR", "ca"],
    "amount": [12000, 5000, 30000, 8000, 7000],
})
regions = {"CA": "West", "TX": "South", "NY": "Northeast"}

raw = loans["state"].map(regions)
print(raw.isna().sum())  # 2: "PR" has no key, "ca" differs in case

filled = loans["state"].map(regions).fillna("Unknown")
with_default = loans["state"].map(defaultdict(lambda: "Unknown", regions))
kept = loans["state"].replace(regions)  # unmatched values survive

print(pd.DataFrame({"raw": raw, "filled": filled,
                    "with_default": with_default, "kept": kept}))

go deeper

for a junior

Recall that map with a dict returns a new Series on the same index and that any value the dict lacks becomes NaN.

for a middle

Explain the remedies, fillna, a missing dict or replace, and why an incomplete integer mapping turns the column float64.

for a senior

Show the validation habit: compare NaN counts before and after the map, normalise keys first, and know that a missing dict falls back to per-element calls.

for a principal

Frame code mappings as reference data: keep the lookup table versioned beside the pipeline and make an unmapped code fail a check rather than become a silent NaN.

## What `Series.map` does with a dict `Series.map(func, na_action=None)` substitutes each value of a Series with another value. Its `func` argument can be a **function**, a **mapping** (a `dict` or any `collections.abc.Mapping`), or another **Series**. With a dict, each value of the Series is treated as a **key**, and the result holds the dict's **value** for that key. The result is always a **new Series with the same index** as the caller. Nothing is written back into the original column; you assign the result yourself, for example `loans["region"] = loans["state"].map(regions)`. Take a loan-application table whose `state` column holds `"CA"`, `"TX"`, `"NY"`, `"PR"` and `"ca"`, and a dict `{"CA": "West", "TX": "South", "NY": "Northeast"}`. The first three rows map cleanly. `"PR"` has no key at all, and `"ca"` differs only in case, but a dict lookup is exact, so both come back as **NaN**. ## Unmapped keys become NaN The `Series.map` docstring states the rule: values not found in the dict as keys are converted to NaN. Nothing warns you. Three consequences follow: - **Silent data loss.** A typo, a new code, extra whitespace or a different case in the source turns into a missing region with no error. - **A dtype change.** Mapping integer codes to integer values gives `int64` when every key matches, but `float64` as soon as one value is unmapped, because NaN cannot live in a NumPy integer column. - **Downstream surprises.** A grouping or a count on the new column quietly treats those rows as missing. Missing values already in the source behave like any other key. A NaN in `state` is looked up too: if the dict has a NaN key, that value is used, otherwise the row stays NaN. With `na_action="ignore"`, NaN is never looked up and always stays NaN, so the NaN count after the map can only grow from rows the dict did not cover. The habit worth showing in an interview is to check `result.isna().sum()` against `source.isna().sum()` right after the map, so an unexpected miss is caught where it happens. ## Ways to handle the missing keys | Approach | What unmapped values become | When to use it | |---|---|---| | `s.map(d)` | NaN | You want misses to stay visible | | `s.map(d).fillna("Unknown")` | the fill value | One default for every miss | | `s.map(defaultdict(lambda: "Unknown", d))` | the `__missing__` default | The default must come from the mapping itself | | `s.replace(d)` | their original value | Partial recodes, where unmatched values should survive | The `defaultdict` route works because pandas checks for a **dict subclass that defines `__missing__`** and then uses that default instead of NaN. The docstring says exactly that. ## How pandas performs the lookup For a plain dict, pandas does not call Python once per value. It turns the dict into a Series indexed by its keys, finds the position of every source value in that index with one hash lookup pass (`get_indexer`), and takes the matching values. That is why a dict map over a large column is fast. A dict that defines `__missing__` changes the route: pandas wraps it in a function and calls it **per element**, because only a real `__getitem__` call can trigger `__missing__`. On a column of millions of rows, `map(d).fillna(default)` is usually the faster way to the same result. Passing a **Series** works the same way as a plain dict: the Series' index plays the role of the keys. That is handy when the lookup table already lives in a DataFrame, for example `codes.set_index("state")["region"]`. ## `map` with a function, and `na_action` With a function, `map` calls it once per element, so it is Python-speed. The `na_action` argument decides what happens to missing values: 1. `na_action=None` (the default) passes NaN to the function too, so a function such as `str.lower` raises `TypeError` on a float NaN. 2. `na_action="ignore"` propagates NaN without calling the function or looking it up. For string work, the vectorized `.str` accessor methods are the usual choice over a per-element function; the text-columns lesson covers them. ## What to say in an interview - `map` with a dict is a **lookup**: same index, new values, original untouched. - **Unmapped values become NaN** silently; integer results turn float. - Fix with `fillna`, a `__missing__` dict, or `replace` when unmatched values should be kept. - Normalise keys before the map when the source is messy, and **count the NaNs** afterwards.

  • Why does `codes.map({1: 10, 2: 20})` come back as float64 when `codes` is int64?
    At least one code had no key, so pandas put NaN in that row, and a NumPy-backed integer column cannot hold NaN, so the result is float64. If every value matches, the result stays int64. Fill the gap and cast back, or map onto a nullable integer dtype if a missing value is genuinely expected.
  • Is `map` with a `defaultdict` as fast as `map` with a plain dict?
    No. For a plain dict pandas builds a lookup Series and resolves every value with one vectorized index lookup. A dict that defines `__missing__` has to be called per element so the default can fire, which is Python-speed. On a large column, `map(d).fillna(default)` gives the same result faster.
  • When would you choose `replace` over `map` for recoding a column?
    When only some values should change and the rest must stay as they were, for example fixing three misspelt codes in a column of hundreds. `replace(d)` leaves unmatched values untouched, while `map(d)` turns every unmatched value into NaN.

A mail sorter with a pigeonhole per known postcode: a letter with an unknown or misspelt code is not forced into the nearest slot, it lands in the unsorted tray, and only someone who checks that tray notices it.

  • Believes values missing from the dict are left unchanged by map
  • Expects map to raise a KeyError when a value has no key
  • Thinks map modifies the original column in place
  • Never checks how many NaNs the map introduced
  • Assumes the integer dtype survives an incomplete mapping
open as a page

A pandas loan table assigns risk tiers with `df.apply(tier, axis=1)`; how do you replace that apply with `np.where` or `np.select`?

level: middleimportance: must knowfreq 74%

basics

~20 s

Build each condition as a boolean Series over whole columns, then use np.where for two outcomes or np.select(conditions, choices, default=...) for several, where the first true condition wins. Handle NaN explicitly, since it fails every comparison.

open as a page

In pandas, when do you bin a credit-score column with `pd.cut` and when with `pd.qcut`, and what do their defaults do to edge values?

level: middleimportance: should knowfreq 48%

basics

~20 s

pd.cut uses edges you choose, such as policy thresholds; pd.qcut picks edges at quantiles so bins hold about equal counts. cut's intervals are right-closed, so the lowest edge is excluded unless include_lowest=True, and values outside the edges become NaN.

open as a page

After upgrading a loan-scoring script to pandas 3.0, `df.applymap(fmt)` raises AttributeError; how do you migrate the call safely?

level: middleimportance: should knowfreq 44%

basics

~10 s

pandas 3.0 removed DataFrame.applymap; rename it to DataFrame.map, which calls the function on every cell. Add na_action="ignore" if the function cannot take NaN, and prefer a vectorized method where one exists.

open as a page

In pandas, a `df.apply(func, axis=1)` whose function returns a tuple yields one column of tuples; how do you get two real columns instead?

level: seniorimportance: should knowfreq 38%

basics

~20 s

With the default result_type=None, a function returning a tuple or list gives a Series of those objects. Pass result_type="expand" to spread them into columns, or return a Series whose index becomes the column names; better still, compute each column vectorized.

open as a page