skip to content

In pandas, how do read_csv's chunksize, iterator=True and low_memory differ, and which of them bounds memory on a 20 GB clickstream file?

level: seniorimportance: should knowfreq 55%

answer

  1. one frame or many
  2. a reader object you iterate
  3. an internal setting, not streaming
  4. the mixed-types warning

basics

~10 s

chunksize and iterator=True make read_csv return a TextFileReader that yields DataFrames piece by piece, which bounds memory. low_memory=True only parses in internal pieces and still returns one full DataFrame.

solid answer

~40 s

`pd.read_csv(path, chunksize=500_000)` returns a `TextFileReader`, not a frame; iterating it yields DataFrames of up to 500,000 rows, so memory holds one piece at a time as long as you drop each one after use. `iterator=True` returns the same reader and lets you pull pieces of any size with `get_chunk(n)`. Use the reader as a context manager so the file closes. `low_memory=True` is the C engine's default: it parses the file in internal blocks to lower peak parser memory, but still hands back one complete DataFrame, so it does not help a file that does not fit. Its side effect is the `DtypeWarning` about mixed types. Each chunk infers its own dtypes, so pass `dtype` and `usecols` on the chunked read.

code

python · 18 lines
python
import io

import pandas as pd

raw = "user_id,event\n" + "".join(f"u{i},view\n" for i in range(10))

with pd.read_csv(io.StringIO(raw), chunksize=4, dtype={"user_id": "str"}) as reader:
    print(type(reader).__name__)  # TextFileReader
    for chunk in reader:
        print(len(chunk), chunk.index[0], chunk.index[-1])
# 4 0 3
# 4 4 7
# 2 8 9

with pd.read_csv(io.StringIO(raw), iterator=True) as reader:
    first = reader.get_chunk(3)
    nxt = reader.get_chunk(2)
print(first.index.tolist(), nxt.index.tolist())  # [0, 1, 2] [3, 4]

go deeper

for a junior

Remember that chunksize gives you an object to loop over, each step a normal DataFrame, while low_memory still returns one frame.

for a middle

Explain TextFileReader, get_chunk with iterator=True, the continuing RangeIndex, and why low_memory triggers DtypeWarning.

for a senior

Stream a file larger than memory with chunksize, usecols and a dtype map in a with block, and refuse the list-then-concat pattern in review.

for a principal

Weigh whether repeated chunked passes over a huge CSV are worth keeping versus converting it once to a columnar format the team reads directly.

## Three settings that sound alike `pd.read_csv` has three arguments that mention reading in parts, and only two of them change what you get back. | Argument | What `read_csv` returns | Rows in memory at once | |---|---|---| | `chunksize=n` | a `TextFileReader` that yields DataFrames | up to `n` per chunk | | `iterator=True` | a `TextFileReader`; you call `get_chunk(size)` | whatever `size` you ask for | | `low_memory=True` (default) | one `DataFrame` | the whole file | ## `chunksize`: a reader that yields frames With `chunksize`, `read_csv` returns a **`TextFileReader`** instead of parsing the file. The reader is an iterator: each step parses the next `n` data rows and returns them as an ordinary DataFrame. The idiomatic form uses it as a **context manager**, so the file handle is closed even if your loop raises: ```python with pd.read_csv(path, chunksize=500_000, usecols=["ts", "zip", "page"], dtype={"zip": "str"}) as reader: for chunk in reader: ... # work on one chunk, keep only a small result ``` Details that matter in practice: - **The index keeps counting.** Without `index_col`, each chunk gets a `RangeIndex` that continues from the previous one (0 to 499,999, then 500,000 onward), not one that restarts at 0. - **The last chunk is shorter** when the row count is not a multiple of `chunksize`. - **Dtypes are inferred per chunk.** One chunk may see only digits in a column that a later chunk sees as text. Passing `dtype` for every column you care about makes all chunks agree; the library-neutral lesson on reading more than fits covers why this happens. - **`usecols` applies to every chunk**, which is the cheapest way to cut memory per chunk. - **`nrows` caps the whole pass.** With `chunksize=500_000, nrows=2_000_000` the reader stops after four chunks, which is a cheap way to test the loop on the start of the file before running it on all 20 GB. - **Exhaustion closes the file.** When iteration runs out, the reader closes its handle itself; the `with` block matters for loops that `break` early or raise. - **The PyArrow engine cannot chunk.** `engine='pyarrow'` with `chunksize` or `iterator` raises `ValueError`, so a streamed read stays on the C (or Python) engine. What you do with each chunk (aggregating, filtering, writing it out) is the general pattern of reading in pieces, covered in the library-neutral material. The pandas trap is collecting every chunk in a list and calling `pd.concat` at the end, which rebuilds the full frame you were trying to avoid. ## `iterator=True` and `get_chunk` `iterator=True` returns the same `TextFileReader`, but you control the size of each pull: 1. `reader = pd.read_csv(path, iterator=True)` 2. `head = reader.get_chunk(1000)` parses the first 1,000 rows. 3. `rest = reader.get_chunk(50_000)` continues from where the last pull stopped. It suits inspecting the first rows of a huge file, or a loop whose piece size changes. For a slice without a reader, `nrows=` and `skiprows=` read a fixed window, for example `skiprows=range(1, 1_000_001), nrows=1_000_000` for the second million rows while keeping the header. ## `low_memory`: internal blocks, one result `low_memory=True` is the default and is honoured only by the C engine. The docstring states that the file is processed internally in chunks to lower memory use **while parsing**, and that the entire file is still read into a single DataFrame regardless; it points you to `chunksize` or `iterator` for chunked results. So `low_memory` never makes a 20 GB file fit. Its visible side effect is **`DtypeWarning`**: because each internal block infers types separately, a column can come back with mixed types, and pandas warns with "Columns (...) have mixed types. Specify dtype option on import or set low_memory=False." Both fixes are real, but they differ: - `dtype={'zip': 'str'}` fixes the type and keeps the lower parsing memory. - `low_memory=False` parses in one pass so inference sees the whole column, which uses **more** memory, not less. ## Choosing for the 20 GB file 1. Use `chunksize` with `usecols` and an explicit `dtype` map, inside a `with` block. 2. Keep only a small per-chunk result, never the chunks themselves. 3. Treat a `DtypeWarning` on a full read as a missing `dtype` entry, not as a reason to flip `low_memory`.

  • A full read_csv emits DtypeWarning about mixed types in column 7. What do you change?
    Give that column an explicit type, for example `dtype={'zip': 'str'}`, after checking what the column should hold. The warning comes from the C engine's internal blocks inferring types separately. `low_memory=False` also silences it by parsing in one pass, but it raises peak memory and still lets inference choose the type for you.
  • Why does collecting every chunk in a list and calling pd.concat defeat chunksize?
    Because the list ends up holding every chunk, and `pd.concat` then builds one more frame of the same size, so peak memory is at least the full dataset. Chunking only helps when each chunk is reduced (aggregated, filtered or written out) and dropped before the next is parsed.

low_memory is a copier that scans a book a few pages at a time but still hands you the whole stack; chunksize hands you one chapter, takes it back, then hands you the next.

  • low_memory=True makes read_csv return the file in pieces
  • chunksize makes read_csv return a list of DataFrames
  • Each chunk's index restarts at 0
  • engine='pyarrow' with chunksize streams the file faster
  • low_memory=False is the fix when a read runs out of memory