In pandas, why can `pd.get_dummies(batch, columns=["plan"], drop_first=True)` on a scoring batch silently disagree with the columns built from the training frame?
answer
- levels come from the data present
- first level in sorted order
- a batch with fewer levels
- fix the categories up front
basics
~20 sget_dummies builds one column per level present in the data it is given, and drop_first drops the first of those levels. A batch missing some plans gets different columns, so fix the levels with a CategoricalDtype before encoding.
solid answer
~40 s`pd.get_dummies` learns its levels from the values in front of it: a plain column's distinct values in sorted order, or a categorical column's declared categories. `drop_first=True` then removes the first of those levels. On a training frame with `basic`, `pro` and `team` it drops `basic` and keeps `plan_pro` and `plan_team`; on a batch that only holds `pro`, `pro` is the first level, so it is dropped and no plan column comes out at all. Reindexing to the training columns with `fill_value=False` then encodes every row as `basic` without an error. The fix is to cast `plan` to a `pd.CategoricalDtype(["basic", "pro", "team"])` before both calls, so the columns and the dropped level never depend on the batch. Indicators are `bool` by default; `pd.from_dummies` with `default_category` decodes them back.
code
python · 15 linesimport pandas as pd
plan_type = pd.CategoricalDtype(["basic", "pro", "team"])
batch = pd.DataFrame({"plan": ["pro", "pro"], "seats": [2, 4]})
Xb = pd.get_dummies(batch.astype({"plan": plan_type}),
columns=["plan"], drop_first=True)
print(Xb)
# seats plan_pro plan_team
# 0 2 True False
# 1 4 True False
back = pd.from_dummies(Xb[["plan_pro", "plan_team"]], sep="_",
default_category="basic")
print(back["plan"].tolist()) # ['pro', 'pro']go deeper
Recall that get_dummies makes one True/False column per level, and that drop_first removes one of them.
Explain how levels are chosen, sorted values versus declared categories, and which level drop_first removes.
Spot a batch encoding that silently differs from training, and fix it with a CategoricalDtype plus a check for unseen values.
Decide whether a team encodes with get_dummies on fixed categories or with a fitted encoder that stores its levels.
## What `get_dummies` produces `pd.get_dummies` turns a categorical column into **indicator columns**, one per level, holding `True` where the row has that level. On a DataFrame, the arguments that shape the output are: | Argument | Default | Effect | |---|---|---| | `columns` | `None` | Columns to encode; `None` encodes every `object`, `string` or `category` column, which covers text held as `object` or as `str` | | `prefix` / `prefix_sep` | column name / `"_"` | Output names such as `plan_pro` | | `drop_first` | `False` | Drops the first level, giving k-1 columns for k levels | | `dummy_na` | `False` | Adds a `NaN` indicator column; otherwise a missing value is all `False` | | `dtype` | `bool` | The indicator dtype; `dtype=int` or `float` for 0/1 numbers | Columns that are not encoded, such as a numeric `seats` column, are kept unchanged and placed before the indicators. ## Where the levels come from This is the part that bites in production. `get_dummies` does not know the full set of plans a business sells; it knows only the values in the frame it is given. - For a **plain** column (`str` or `object`), the levels are the distinct values present, in **sorted** order. - For a **categorical** column, the levels are its declared **categories**, in their declared order, whether or not every category occurs in this frame. `drop_first=True` drops the **first** of those levels. It exists so that a linear model does not see k columns that always sum to one; the dropped level becomes the baseline, the row where every indicator is `False`. ## The silent mismatch ```python import pandas as pd train = pd.DataFrame({"plan": ["basic", "pro", "team", "pro"], "seats": [1, 5, 12, 3]}) batch = pd.DataFrame({"plan": ["pro", "pro"], "seats": [2, 4]}) X = pd.get_dummies(train, columns=["plan"], drop_first=True) # columns: seats, plan_pro, plan_team Xb = pd.get_dummies(batch, columns=["plan"], drop_first=True) # columns: seats (pro was the only level, so it was the one dropped) Xb = Xb.reindex(columns=X.columns, fill_value=False) # both rows now read plan_pro=False, plan_team=False: every row looks like "basic" ``` Nothing raises. A model scoring `Xb` treats two `pro` customers as `basic` customers. Without `drop_first`, a `reindex` with `fill_value=False` does repair a level that is merely absent from the batch, but the reverse case still bites: a level training never saw adds an extra column, the `reindex` silently drops it, and that row reads all `False`. ## The fix: declare the levels 1. Define the levels once: `plan_type = pd.CategoricalDtype(["basic", "pro", "team"])`. 2. Cast before encoding, in training and in scoring: `batch.astype({"plan": plan_type})`. 3. Encode as before. The columns now come from the categories, so `plan_pro` and `plan_team` are produced for the batch, with the right values. 4. Values outside the declared categories become `NaN` on the cast, so check `isna()` after it to catch a new plan instead of encoding it as the baseline. ## Missing values and the baseline By default a missing `plan` is encoded as **all `False`**, the same pattern `drop_first` gives the baseline level. With `drop_first=True`, a customer with no plan and a `basic` customer therefore look identical to the model. - `dummy_na=True` adds a separate `NaN` indicator column, so the two cases stay apart. - Alternatively, fill the missing value with an explicit level such as `"unknown"` that is declared among the categories. ## Going back with `from_dummies` `pd.from_dummies` inverts the encoding. It takes a frame of 0/1 or boolean indicator columns and returns one column per variable: - `sep="_"` splits `plan_pro` into the variable `plan` and the level `pro`; without `sep` every column is treated as a level of one variable. - `default_category="basic"` names the level for rows where every indicator is zero, which is exactly what `drop_first` produced. Without it, such a row raises `ValueError: Dummy DataFrame contains unassigned value(s)`. - A row with two indicators set raises `ValueError: Dummy DataFrame contains multi-assignment(s)`, and NA values in the input raise as well. The decoded column holds the level labels, typed like the indicator column labels, not a `category` dtype; cast it if you need one. ## Review checklist - Is the encoded column categorical with fixed categories, or are levels inferred from each batch? - Does every run produce the same column names in the same order? - Is the baseline level a deliberate choice, made by ordering the categories, rather than whatever sorts first? - Are unseen values detected rather than silently treated as the baseline?
- What dtype are `get_dummies` indicator columns by default, and how do you get 0/1 integers?They are `bool` by default; before pandas 2.0 the default was `uint8`. Pass `dtype=int` or `dtype=float` for numeric indicators. With a nullable `"string"` or an Arrow-backed input, the default follows that family and yields `boolean` or Arrow bool columns.
- How do you choose which level `drop_first` treats as the baseline?Order the categories. With a `CategoricalDtype`, the first declared category is the one dropped, so `pd.CategoricalDtype(["pro", "basic", "team"])` makes `pro` the baseline. On a plain column the first level is whichever value sorts first.
- get_dummies remembers the levels it saw in training
- drop_first always drops the same level in every batch
- Reindexing a drop_first batch to the training columns makes it correct
- get_dummies returns uint8 columns by default in current pandas
- from_dummies works on drop_first output without any extra argument