skip to content

In pandas 3.0, how do read_csv's dtype and parse_dates keep zip codes like 02134 intact and timestamps typed, and what dtypes result?

level: middleimportance: must knowfreq 75%

answer

  1. decide types before parsing
  2. a per-column dict
  3. astype afterwards is too late
  4. the 3.0 text and datetime units

basics

~10 s

Pass dtype={'zip': 'str'} so the parser never turns zip codes into numbers, and parse_dates=['ts'] for timestamps. In pandas 3.0 the zip column gets the str dtype and ISO timestamps become datetime64[us].

solid answer

~40 s

Left alone, `read_csv` sees `02134` as an integer, so the column comes back `int64` (or `float64` once a blank appears) and the zero is gone; `astype(str)` afterwards cannot bring it back. Declaring `dtype={'zip': 'str'}` keeps the raw text, and in pandas 3.0 that column has the new `str` dtype, a `StringDtype` whose missing value is NaN, where 2.x gave `object`. Missing markers still apply first, so a blank zip is NaN, not an empty string. `parse_dates=['ts']` converts the timestamp column at read time; in pandas 3.0 ISO strings without nanoseconds give `datetime64[us]`, not 2.x's `datetime64[ns]`. Add `date_format=` for a non-ISO layout. `parse_dates` now takes only a bool or a list: combining columns was removed in 3.0.

code

python · 17 lines
python
import io

import pandas as pd

raw = (
    "ts,zip,page\n"
    "2026-03-01 08:15:02,02134,/home\n"
    "2026-03-01 08:15:09,,/cart\n"
)

guessed = pd.read_csv(io.StringIO(raw))
print(guessed["zip"].tolist())  # [2134.0, nan]  zero lost, float because of the blank

typed = pd.read_csv(io.StringIO(raw), dtype={"zip": "str"}, parse_dates=["ts"])
print(typed["zip"].tolist())  # ['02134', nan]
print(typed["zip"].dtype)  # str  (object in pandas 2.x)
print(typed["ts"].dtype)  # datetime64[us]  (datetime64[ns] in pandas 2.x)

go deeper

for a junior

Remember that read_csv guesses types, so zip codes and ids need dtype='str' and timestamps need parse_dates at read time.

for a middle

Explain why the zero is lost during parsing, what the 3.0 str dtype is, how missing markers still apply, and why datetimes now come back in microseconds.

for a senior

Audit 2.x code for dtype == 'object' checks, parse_dates column combining and date_parser, all of which change or fail on 3.0.

for a principal

Push explicit dtype maps for identifier columns into shared readers so no consumer depends on inference that shifts between releases.

## Why the zeros disappear A CSV file is text with no type information. `pd.read_csv` therefore **infers** a dtype for each column from the values it sees. A column whose every value looks like an integer becomes `int64`; if one field is blank, the blank becomes NaN and the column becomes `float64`, because a NumPy integer column cannot hold NaN. Either way, the text `02134` has been parsed into the number 2134 and the leading zero no longer exists. That is why `df['zip'].astype(str)` after the read does not help: it turns 2134 into `'2134'` (or 2134.0 into `'2134.0'`). The information was lost **during parsing**, so the fix has to happen **before** parsing. ## Declaring types with `dtype` The `dtype` argument accepts a single dtype for every column or a dict mapping column names to dtypes: ```python df = pd.read_csv(path, dtype={"zip": "str", "device": "category"}) ``` - **`'str'` or the builtin `str`** tells the parser to keep the field as text. In **pandas 3.0** the column gets the `str` dtype: a `StringDtype` that holds only strings or missing values, uses NaN as its missing value, and is backed by PyArrow when PyArrow is installed. In pandas 2.x the same call produced `object` dtype. - **Missing markers run first.** A blank zip field is still read as NaN, because the default missing markers (empty string, `NA`, `NULL` and others) are applied regardless of `dtype`. Control that with `na_values` and `keep_default_na`. - **`converters` win over `dtype`.** The docstring says converters are applied *instead of* dtype conversion, so do not set both for one column. - **A `defaultdict`** as `dtype` sets the dtype for every column you did not list, for example `defaultdict(lambda: 'str', clicks='int64')`. - **A NumPy integer dtype refuses blanks.** `dtype={'clicks': 'int64'}` on a column with an empty field raises `ValueError` ("Integer column has NA values in column ..."). Ask for the nullable `'Int64'` instead, which holds `pd.NA`, or leave the column to infer `float64`. A defensive pattern for files from outside the team is to read **every** column as text with `dtype='str'` and convert the few numeric ones explicitly afterwards. It costs some memory and an extra step, but no identifier is ever silently turned into a number, and a column that drifts from numbers to text in next month's file cannot change the frame's types behind your back. ## Parsing timestamps with `parse_dates` `parse_dates=['ts']` asks the reader to convert the named columns to datetimes once the text is read. ISO 8601 strings take a fast path. For other layouts, pass `date_format='%d/%m/%Y %H:%M'` (or a dict of formats per column), or `date_format='ISO8601'` for ISO strings that vary in shape. If a value cannot be parsed, the column is left unconverted instead of raising, so check the dtype after reading. In **pandas 3.0** the datetime **resolution** is inferred: parsed strings default to microseconds, so the column is `datetime64[us]`, falling back to nanoseconds only when a string carries nanosecond digits. Pandas 2.x always gave `datetime64[ns]`. Most code does not notice, but converting to integers gives values 1000 times smaller than before; the lesson on pandas datetime parsing covers `as_unit` for that. pandas 3.0 also enforced several removals around this argument: | 2.x usage | pandas 3.0 | |---|---| | `parse_dates={'when': ['date', 'time']}` | `TypeError`: only booleans and lists are accepted | | `parse_dates=[['date', 'time']]` to combine | no longer supported; combine the columns yourself, then convert | | `date_parser=` | removed in favour of `date_format=` | | `infer_datetime_format=True` | removed; a strict version is the default | | `keep_date_col=` | removed | ## Reading fewer columns with `usecols` `usecols=['ts', 'zip', 'page']` parses only those columns, which saves time and memory on a wide clickstream file. Two details are worth knowing: 1. **Order is ignored.** `usecols=['zip', 'ts']` returns the columns in file order; select `df[['zip', 'ts']]` afterwards if order matters. 2. **A callable works too.** `usecols=lambda c: c.startswith('utm_')` keeps every column whose name passes the test, which suits files whose column set drifts. ## A worked call for the clickstream file 1. List the columns you need in `usecols`. 2. Give every identifier-like column (`zip`, `session_id`) a `'str'` dtype. 3. Put timestamp columns in `parse_dates` and add `date_format` if they are not ISO. 4. Check `df.dtypes` once: expect `str` for text and `datetime64[us]` for ISO timestamps on 3.0.

  • Your code checks df['zip'].dtype == 'object' after the read. What happens on pandas 3.0?
    The check turns False, because the column now has the `str` dtype rather than `object`. Code that branches on object dtype to find text columns silently stops matching. Use `pd.api.types.is_string_dtype(df['zip'])`, which is true for object and the string dtypes, or select with `select_dtypes(include=['str'])` once 2.x support is dropped.
  • A legacy job used parse_dates=[['date', 'time']]. How do you port it to pandas 3.0?
    Combining columns inside `parse_dates` was removed in 3.0. Read `date` and `time` as text, join them, then convert: `pd.to_datetime(df['date'] + ' ' + df['time'], format='%Y-%m-%d %H:%M:%S')`. The explicit `format` keeps parsing strict and fast. `date_parser=` is gone too, so any custom parser moves into that post-read step.

  • astype(str) after reading brings the leading zeros back
  • dtype={'zip': 'str'} also keeps blank zip fields as empty strings
  • In pandas 3.0 a text column read from CSV is object dtype
  • parse_dates still combines a date and a time column in pandas 3.0
  • Setting converters and dtype for one column applies both
  • usecols=['zip', 'ts'] returns the columns in that order

Assumes pandas 3.0: dtype 'str' gives the str dtype (object in 2.x), parsed timestamps default to microseconds (nanoseconds in 2.x), and parse_dates no longer combines columns.