In pandas, what does `Series.map` return when you pass it a dict, and what happens to values the dict has no key for?
answer
- same index, new values
- lookup, not a Python call
- missing key means missing value
- defaultdict or fillna afterwards
basics
~20 sSeries.map with a dict returns a new Series on the same index, each value replaced by its dict lookup. Any value without a key becomes NaN, unless the dict defines missing, which then supplies the default.
solid answer
~30 s`s.map(mapping)` looks every value up in the dict and returns a new Series aligned to the original index; the source is untouched. A value with no key, including a near miss like `"ca"` against a key `"CA"`, comes back as NaN, and an integer result turns float64 once a NaN appears. I fix the gaps with `.fillna("Unknown")`, or pass a dict subclass with `__missing__` such as `defaultdict`, which pandas calls for the default. If I want unmapped values kept as they were, `replace(mapping)` is the right call instead. Then I count the NaNs, because a silent miss is the real risk.
code
python · 19 linesfrom collections import defaultdict
import pandas as pd
loans = pd.DataFrame({
"state": ["CA", "TX", "NY", "PR", "ca"],
"amount": [12000, 5000, 30000, 8000, 7000],
})
regions = {"CA": "West", "TX": "South", "NY": "Northeast"}
raw = loans["state"].map(regions)
print(raw.isna().sum()) # 2: "PR" has no key, "ca" differs in case
filled = loans["state"].map(regions).fillna("Unknown")
with_default = loans["state"].map(defaultdict(lambda: "Unknown", regions))
kept = loans["state"].replace(regions) # unmatched values survive
print(pd.DataFrame({"raw": raw, "filled": filled,
"with_default": with_default, "kept": kept}))go deeper
Recall that map with a dict returns a new Series on the same index and that any value the dict lacks becomes NaN.
Explain the remedies, fillna, a missing dict or replace, and why an incomplete integer mapping turns the column float64.
Show the validation habit: compare NaN counts before and after the map, normalise keys first, and know that a missing dict falls back to per-element calls.
Frame code mappings as reference data: keep the lookup table versioned beside the pipeline and make an unmapped code fail a check rather than become a silent NaN.
## What `Series.map` does with a dict `Series.map(func, na_action=None)` substitutes each value of a Series with another value. Its `func` argument can be a **function**, a **mapping** (a `dict` or any `collections.abc.Mapping`), or another **Series**. With a dict, each value of the Series is treated as a **key**, and the result holds the dict's **value** for that key. The result is always a **new Series with the same index** as the caller. Nothing is written back into the original column; you assign the result yourself, for example `loans["region"] = loans["state"].map(regions)`. Take a loan-application table whose `state` column holds `"CA"`, `"TX"`, `"NY"`, `"PR"` and `"ca"`, and a dict `{"CA": "West", "TX": "South", "NY": "Northeast"}`. The first three rows map cleanly. `"PR"` has no key at all, and `"ca"` differs only in case, but a dict lookup is exact, so both come back as **NaN**. ## Unmapped keys become NaN The `Series.map` docstring states the rule: values not found in the dict as keys are converted to NaN. Nothing warns you. Three consequences follow: - **Silent data loss.** A typo, a new code, extra whitespace or a different case in the source turns into a missing region with no error. - **A dtype change.** Mapping integer codes to integer values gives `int64` when every key matches, but `float64` as soon as one value is unmapped, because NaN cannot live in a NumPy integer column. - **Downstream surprises.** A grouping or a count on the new column quietly treats those rows as missing. Missing values already in the source behave like any other key. A NaN in `state` is looked up too: if the dict has a NaN key, that value is used, otherwise the row stays NaN. With `na_action="ignore"`, NaN is never looked up and always stays NaN, so the NaN count after the map can only grow from rows the dict did not cover. The habit worth showing in an interview is to check `result.isna().sum()` against `source.isna().sum()` right after the map, so an unexpected miss is caught where it happens. ## Ways to handle the missing keys | Approach | What unmapped values become | When to use it | |---|---|---| | `s.map(d)` | NaN | You want misses to stay visible | | `s.map(d).fillna("Unknown")` | the fill value | One default for every miss | | `s.map(defaultdict(lambda: "Unknown", d))` | the `__missing__` default | The default must come from the mapping itself | | `s.replace(d)` | their original value | Partial recodes, where unmatched values should survive | The `defaultdict` route works because pandas checks for a **dict subclass that defines `__missing__`** and then uses that default instead of NaN. The docstring says exactly that. ## How pandas performs the lookup For a plain dict, pandas does not call Python once per value. It turns the dict into a Series indexed by its keys, finds the position of every source value in that index with one hash lookup pass (`get_indexer`), and takes the matching values. That is why a dict map over a large column is fast. A dict that defines `__missing__` changes the route: pandas wraps it in a function and calls it **per element**, because only a real `__getitem__` call can trigger `__missing__`. On a column of millions of rows, `map(d).fillna(default)` is usually the faster way to the same result. Passing a **Series** works the same way as a plain dict: the Series' index plays the role of the keys. That is handy when the lookup table already lives in a DataFrame, for example `codes.set_index("state")["region"]`. ## `map` with a function, and `na_action` With a function, `map` calls it once per element, so it is Python-speed. The `na_action` argument decides what happens to missing values: 1. `na_action=None` (the default) passes NaN to the function too, so a function such as `str.lower` raises `TypeError` on a float NaN. 2. `na_action="ignore"` propagates NaN without calling the function or looking it up. For string work, the vectorized `.str` accessor methods are the usual choice over a per-element function; the text-columns lesson covers them. ## What to say in an interview - `map` with a dict is a **lookup**: same index, new values, original untouched. - **Unmapped values become NaN** silently; integer results turn float. - Fix with `fillna`, a `__missing__` dict, or `replace` when unmatched values should be kept. - Normalise keys before the map when the source is messy, and **count the NaNs** afterwards.
- Why does `codes.map({1: 10, 2: 20})` come back as float64 when `codes` is int64?At least one code had no key, so pandas put NaN in that row, and a NumPy-backed integer column cannot hold NaN, so the result is float64. If every value matches, the result stays int64. Fill the gap and cast back, or map onto a nullable integer dtype if a missing value is genuinely expected.
- Is `map` with a `defaultdict` as fast as `map` with a plain dict?No. For a plain dict pandas builds a lookup Series and resolves every value with one vectorized index lookup. A dict that defines `__missing__` has to be called per element so the default can fire, which is Python-speed. On a large column, `map(d).fillna(default)` gives the same result faster.
- When would you choose `replace` over `map` for recoding a column?When only some values should change and the rest must stay as they were, for example fixing three misspelt codes in a column of hundreds. `replace(d)` leaves unmatched values untouched, while `map(d)` turns every unmatched value into NaN.
A mail sorter with a pigeonhole per known postcode: a letter with an unknown or misspelt code is not forced into the nearest slot, it lands in the unsorted tray, and only someone who checks that tray notices it.
- Believes values missing from the dict are left unchanged by map
- Expects map to raise a KeyError when a value has no key
- Thinks map modifies the original column in place
- Never checks how many NaNs the map introduced
- Assumes the integer dtype survives an incomplete mapping