In pandas 3.0, when does astype('category') shrink a text column's memory, and when does it make the column larger?
answer
- distinct values against row count
- one code per row
- smallest signed integer that fits
- the str baseline in 3.0
basics
~20 sA category column stores each distinct value once plus one small integer code per row, so it shrinks low-cardinality text sharply. When nearly every value is unique, the categories duplicate the data and the codes add bytes, so the column grows.
solid answer
~40 sA pandas categorical keeps `.cat.categories` (each distinct value once) and `.cat.codes`, an integer per row sized to the number of categories: `int8` below 127 categories, `int16` below 32,767, then `int32`. With six department names over a million rows the codes take one byte per row, so the column drops to about 1 MB. With a million unique IDs the categories hold every string again and the codes need `int32`, so the column gets bigger. In pandas 3.0 the baseline is no longer object: text infers the `str` dtype, PyArrow-backed when PyArrow is installed, which is already several times smaller than object, so the saving is smaller than old 2.x tutorials show but still large for repeated values. I measure with `memory_usage(deep=True)` before and after.
code
python · 18 linesimport numpy as np
import pandas as pd
rng = np.random.default_rng(0)
names = ["Engineering", "Marketing", "Sales", "Finance", "Support", "Legal"]
dept = pd.Series(rng.choice(names, 1_000_000))
ids = pd.Series([f"user{i:07d}" for i in range(1_000_000)])
for label, s in [("dept", dept), ("ids", ids)]:
cat = s.astype("category")
print(
label,
s.dtype,
s.memory_usage(deep=True),
"->",
cat.memory_usage(deep=True),
cat.cat.codes.dtype,
)go deeper
Recall the two parts of a categorical, categories stored once and a code per row, and that few distinct values is when it saves memory.
Explain how the code width follows the category count, why unique IDs make the column larger, and how to measure with memory_usage(deep=True).
Know that pandas 3.0's str baseline narrows the gap, and catch the operations, such as concat with mismatched categories or a CSV round trip, that silently drop the categorical.
Weigh where a declared categorical belongs in a pipeline's schema against the upkeep of keeping category lists current as new values arrive.
## How a pandas categorical is laid out A categorical column (dtype `category`) is two arrays: - **`categories`** - an Index holding each distinct value exactly once, exposed as `s.cat.categories`; - **`codes`** - one integer per row giving the position of that row's value in `categories`, exposed as `s.cat.codes`. A missing value has code `-1`, which is why the codes are signed. pandas sizes the codes to the number of categories. The helper that builds them picks the smallest signed integer type whose maximum exceeds the category count: | Number of categories | Codes dtype | Bytes per row | |---|---|---| | up to 126 | `int8` | 1 | | up to 32,766 | `int16` | 2 | | up to about 2.1 billion | `int32` | 4 | | more | `int64` | 8 | So the memory of a categorical is roughly **(size of the distinct values) + (rows x code width)**. The pandas user guide puts it as memory proportional to the number of categories plus the length of the data, where an object column is a constant times the length of the data. ## When it shrinks the column The win comes from **low cardinality**: few distinct values repeated over many rows. A department, country, status or size column is the textbook case. Each value is stored once, and every row costs one byte of code instead of its own copy of the text. One measurement on pandas 3.0.6 with PyArrow installed, a million rows drawn from six department names: | Representation | `memory_usage(deep=True)` | |---|---| | object (the 2.x default for text) | about 56 MB | | `str` (the 3.0 default, PyArrow-backed) | about 15 MB | | `category` | about 1 MB | The exact bytes depend on the strings and the build, but the shape is general: the categorical is dominated by its one-byte codes. ## When it makes the column larger When the number of categories approaches the number of rows, the trade reverses: 1. the `categories` Index stores essentially every value again, so the text is not deduplicated at all; 2. the codes widen (a million distinct IDs need `int32`, four bytes per row); 3. the hash table built while factorizing costs time on every cast. In the same measurement, a million unique `user0000000`-style IDs took about 19 MB as `str` and about 23 MB as `category`. Identifiers, free text, email addresses and timestamps rendered as text are the usual bad candidates. ## What pandas 3.0 changed about the comparison Most category-memory examples were written when text columns were **object**: an array of pointers to separate Python string objects, each with its own header. In pandas 3.0 a text column infers the new **`str` dtype**. With PyArrow installed it stores all the characters in one contiguous buffer plus offsets; without PyArrow it falls back to object storage. Against the PyArrow-backed `str` baseline the categorical saving is smaller than against object, yet still an order of magnitude for repeated values, because `str` still keeps one copy of the text per row. ## Keeping the saving Several operations quietly give the memory back: - **concatenating** categoricals with different categories returns a non-categorical result (the dtype is inferred from the values); cast back with `astype` or combine with `union_categoricals`; - **writing to CSV** keeps only the text, so the column comes back as `str` and must be cast again (or read with a `CategoricalDtype` passed to the reader's `dtype=`); - **adding values** first needs `s.cat.add_categories`, since assigning a value outside the categories raises `TypeError`. ## How to decide - Count first: `s.nunique()` against `len(s)`. A ratio of a few percent or less is a strong candidate. - Measure with `s.memory_usage(deep=True)` before and after; without `deep=True` the object baseline is undercounted. - Prefer a declared `CategoricalDtype` when the set of values is known, so every frame shares the same categories and codes. The short version for an interview: categoricals pay off when values repeat, and the payoff is the code width per row; with near-unique values they cost more than the text they replace.
- Why are the codes signed integers rather than unsigned?A missing value in a categorical is not a category; it is stored as code `-1`. The codes therefore need a sign, and pandas picks the smallest signed type (`int8`, `int16`, `int32`, `int64`) whose maximum exceeds the number of categories. That is also why the `int8` limit is near 127 categories (pandas switches to `int16` at 127) rather than 255.
- A categorical column is concatenated with another frame's categorical and memory jumps. What happened?When the two inputs have different categories, `pd.concat` does not union them; the result falls back to a non-categorical dtype inferred from the values, such as `str` for text. Cast the result back with `astype` using a shared `CategoricalDtype`, or build it with `union_categoricals`, which combines the categories and recodes the rows.
- Does converting to category also speed anything up?Often. Grouping, sorting and `value_counts` work on the integer codes instead of comparing strings, and `.str` methods on a categorical run once per category and then expand. The gain depends on the operation, so it is worth timing; the memory effect is the dependable part.
- Converting any text column to category always saves memory
- Categorical codes are always 64-bit integers, whatever the number of categories
- pandas 3.0 text columns are object, so the saving is as large as ever
- The str dtype already deduplicates repeated values, so category adds nothing
- memory_usage() without deep=True gives a fair object-versus-category comparison
pandas 2.x inferred text columns as object, the baseline most category-memory examples compare against; pandas 3.0 infers the str dtype (PyArrow-backed when installed), a smaller baseline.