In pandas, why can df.info() report a DataFrame as far smaller than it really is, and how does memory_usage(deep=True) fix the measurement?
answer
- array buffer vs the objects behind it
- 8-byte pointers per object row
- the plus sign after the total
- deep walks every element
basics
~10 sBy default pandas counts only each column's array buffer, so an object column counts 8-byte pointers, not the Python strings behind them. memory_usage(deep=True) or info(memory_usage='deep') measures those objects too.
solid answer
~40 s`DataFrame.memory_usage()` returns a Series of bytes per column, with an `Index` entry first. With the default `deep=False` it reads each column's array buffer: exact for numeric and datetime columns, but an `object` column counts only its 8-byte pointers, not the Python strings they reference. `info()` prints the same shallow total and marks it with a `+` when object data is present. `deep=True`, or `info(memory_usage='deep')`, adds the size of every referenced object, at the cost of a full scan. In pandas 3.0 text infers the `str` dtype: backed by PyArrow its buffers are counted exactly even without `deep`, while the NumPy-object fallback still needs `deep=True`. So I rank `df.memory_usage(deep=True).sort_values()` before shrinking anything.
code
python · 18 linesimport numpy as np
import pandas as pd
n = 100_000
codes = [f"event-{i % 500:04d}" for i in range(n)]
df = pd.DataFrame({
"user_id": np.arange(n, dtype="int64"),
"amount": np.random.default_rng(0).random(n),
"code_obj": pd.Series(codes, dtype=object),
"code_str": pd.Series(codes, dtype="str"),
})
report = pd.DataFrame({
"shallow": df.memory_usage(),
"deep": df.memory_usage(deep=True),
})
print(report)
df.info(memory_usage="deep")go deeper
Remember that df.info() and memory_usage() count object columns shallowly, and that deep=True or info(memory_usage='deep') gives the real figure for text held as Python objects.
Explain the mechanism: buffer size per column, 8-byte pointers for object, the + marker, and why the deep scan costs time on tens of millions of rows.
Show how you rank columns with deep=True before shrinking, why PyArrow-backed str columns report exactly, and why the frame's figure is not the process peak.
Decide what a team measures as memory: per-column deep reports in reviews, process peak in production monitoring, and how pandas 3.0's str dtype changes the baseline across a migration.
## What `memory_usage` measures `DataFrame.memory_usage(index=True, deep=False)` returns a **Series of byte counts**: one entry per column and, when `index=True`, an `Index` entry first. `Series.memory_usage` returns a single integer, and `DataFrame.info()` prints the sum of the same figures in human-readable units. For each column pandas asks the column's array how big it is. With the default `deep=False`, that is the size of the array's own buffer: - **numeric columns** cost rows times the item size: 8 bytes for `int64` and `float64`, 4 for `int32` and `float32`, 1 for `bool` and `int8` - **datetime64 and timedelta64** columns cost 8 bytes a row - **categorical** columns cost their integer codes plus their categories Index - **object** columns cost 8 bytes a row: the **pointers** only - the **index** costs little when it is a `RangeIndex` (a start, stop and step) and as much as a column once it is materialised For numeric and datetime columns the shallow figure is the real cost; a categorical misses only the text of its categories when they are held as Python objects. For `object` it is a floor. ## Why object columns under-report An `object` column is a NumPy array of references to Python objects scattered across the heap. Each Python string carries its own object header plus its characters, typically several times the 8-byte pointer that refers to it. The shallow count sees only the pointer array, so a 50-million-row column of short event codes reports about 400 MB while the strings behind it take several times that. `info()` uses the same shallow numbers by default and appends a **`+`** to the total (for example `memory usage: 3.1+ GB`). The source calls that marker **best effort**: it is keyed to `object` columns and an object-like index, and it is not guaranteed to catch every case. ## What `deep=True` changes `memory_usage(deep=True)` and `info(memory_usage='deep')` interrogate the objects: for every element of an object array pandas adds `sys.getsizeof` of that element to the buffer size. Three consequences: 1. **It is slow on big frames**, because it visits every element. That is why it is opt-in. 2. **It counts each reference separately.** The loop sums `getsizeof` per element without de-duplicating, so a column whose rows all point at the same few string objects is over-counted. 3. **It is shallow per object.** `getsizeof` of a list or dict does not include the objects it contains, so nested values still under-report. Even with those limits, `deep=True` is the figure to rank columns by, because it is the one that sees the text. ## The `str` dtype in pandas 3.0 Before 3.0, text loaded into `object`, so every text column needed `deep=True`. In pandas 3.0 text columns infer the dedicated **`str`** dtype instead: | storage of the column | `deep=False` counts | `deep=True` adds | |---|---|---| | `object` (explicit, or mixed values) | 8-byte pointers | every Python object | | `str` backed by PyArrow | the Arrow buffers: offsets, character data, validity bitmap | nothing more | | `str` on the NumPy-object fallback (no PyArrow installed) | 8-byte pointers | every Python string | With PyArrow installed, a `str` column's report is exact either way, because the characters live in Arrow buffers rather than in separate Python objects. Note the gap in `info()`: a `str` column on the object fallback is under-reported shallowly, but its dtype name is not `object`, so the `+` marker may not appear. ## Series and Index forms The same switch exists one level down. `Series.memory_usage(index=True, deep=False)` returns a single integer that includes the Series' index unless `index=False`. `df.memory_usage(index=False)` drops the `Index` row from the frame's report, and `df.index.memory_usage(deep=True)` measures the row labels alone, which matters when a frame has been indexed by a text column. ## Reading the result A sound measurement looks like this: 1. Run `df.memory_usage(deep=True).sort_values(ascending=False)` to rank columns. 2. Compare it with the shallow figure; a large gap marks the columns holding Python objects. 3. Sum with `.sum()` and compare against the process's resident memory. The frame is only part of peak usage, since many operations build intermediate copies. Two further cautions: - `memory_usage` reports the **frame's own buffers**. It does not show temporaries that a `merge`, `groupby` or `astype` creates while running. - With Copy-on-Write in pandas 3.0, a derived frame may **share** buffers with its parent until one is written to, so the sum over several frames can double-count shared memory.
- Why can memory_usage(deep=True) over-report a column?For object arrays pandas sums `sys.getsizeof` of every element without checking whether two rows point at the same Python object. A column built from a handful of repeated string objects is therefore counted as if every row owned its own copy, so the deep figure can exceed what the process really holds.
- Does the sum of memory_usage over a whole session equal the process's memory?No. It reports the frame's buffers only, not the temporaries that merges, groupbys or casts allocate while running, nor interpreter overhead. Under Copy-on-Write in pandas 3.0, derived frames can share buffers with their parent, so summing several frames can count the same memory twice. Compare against the process's resident memory for the real peak.
A warehouse ledger that lists every shelf slot but not what sits in the boxes: numeric columns are the slots themselves, while an object column's slot holds only a note saying where the real box is. deep=True walks to every box and weighs it.
- info() shows the true memory of any DataFrame by default.
- deep=True is always cheap, so pandas should make it the default.
- An object column costs 8 bytes per row, like int64.
- In pandas 3.0 every str column still needs deep=True to be measured.
- The + after info()'s total means the frame is over budget.
Assumes pandas 3.0, where text infers the str dtype (PyArrow-backed when installed); in 2.x text loaded as object, so every text column needed deep=True to be measured.