In pandas 3.0, how do read_csv's dtype and parse_dates keep zip codes like 02134 intact and timestamps typed, and what dtypes result?
answer
- decide types before parsing
- a per-column dict
- astype afterwards is too late
- the 3.0 text and datetime units
basics
~10 sPass dtype={'zip': 'str'} so the parser never turns zip codes into numbers, and parse_dates=['ts'] for timestamps. In pandas 3.0 the zip column gets the str dtype and ISO timestamps become datetime64[us].
solid answer
~40 sLeft alone, `read_csv` sees `02134` as an integer, so the column comes back `int64` (or `float64` once a blank appears) and the zero is gone; `astype(str)` afterwards cannot bring it back. Declaring `dtype={'zip': 'str'}` keeps the raw text, and in pandas 3.0 that column has the new `str` dtype, a `StringDtype` whose missing value is NaN, where 2.x gave `object`. Missing markers still apply first, so a blank zip is NaN, not an empty string. `parse_dates=['ts']` converts the timestamp column at read time; in pandas 3.0 ISO strings without nanoseconds give `datetime64[us]`, not 2.x's `datetime64[ns]`. Add `date_format=` for a non-ISO layout. `parse_dates` now takes only a bool or a list: combining columns was removed in 3.0.
code
python · 17 linesimport io
import pandas as pd
raw = (
"ts,zip,page\n"
"2026-03-01 08:15:02,02134,/home\n"
"2026-03-01 08:15:09,,/cart\n"
)
guessed = pd.read_csv(io.StringIO(raw))
print(guessed["zip"].tolist()) # [2134.0, nan] zero lost, float because of the blank
typed = pd.read_csv(io.StringIO(raw), dtype={"zip": "str"}, parse_dates=["ts"])
print(typed["zip"].tolist()) # ['02134', nan]
print(typed["zip"].dtype) # str (object in pandas 2.x)
print(typed["ts"].dtype) # datetime64[us] (datetime64[ns] in pandas 2.x)go deeper
Remember that read_csv guesses types, so zip codes and ids need dtype='str' and timestamps need parse_dates at read time.
Explain why the zero is lost during parsing, what the 3.0 str dtype is, how missing markers still apply, and why datetimes now come back in microseconds.
Audit 2.x code for dtype == 'object' checks, parse_dates column combining and date_parser, all of which change or fail on 3.0.
Push explicit dtype maps for identifier columns into shared readers so no consumer depends on inference that shifts between releases.
## Why the zeros disappear A CSV file is text with no type information. `pd.read_csv` therefore **infers** a dtype for each column from the values it sees. A column whose every value looks like an integer becomes `int64`; if one field is blank, the blank becomes NaN and the column becomes `float64`, because a NumPy integer column cannot hold NaN. Either way, the text `02134` has been parsed into the number 2134 and the leading zero no longer exists. That is why `df['zip'].astype(str)` after the read does not help: it turns 2134 into `'2134'` (or 2134.0 into `'2134.0'`). The information was lost **during parsing**, so the fix has to happen **before** parsing. ## Declaring types with `dtype` The `dtype` argument accepts a single dtype for every column or a dict mapping column names to dtypes: ```python df = pd.read_csv(path, dtype={"zip": "str", "device": "category"}) ``` - **`'str'` or the builtin `str`** tells the parser to keep the field as text. In **pandas 3.0** the column gets the `str` dtype: a `StringDtype` that holds only strings or missing values, uses NaN as its missing value, and is backed by PyArrow when PyArrow is installed. In pandas 2.x the same call produced `object` dtype. - **Missing markers run first.** A blank zip field is still read as NaN, because the default missing markers (empty string, `NA`, `NULL` and others) are applied regardless of `dtype`. Control that with `na_values` and `keep_default_na`. - **`converters` win over `dtype`.** The docstring says converters are applied *instead of* dtype conversion, so do not set both for one column. - **A `defaultdict`** as `dtype` sets the dtype for every column you did not list, for example `defaultdict(lambda: 'str', clicks='int64')`. - **A NumPy integer dtype refuses blanks.** `dtype={'clicks': 'int64'}` on a column with an empty field raises `ValueError` ("Integer column has NA values in column ..."). Ask for the nullable `'Int64'` instead, which holds `pd.NA`, or leave the column to infer `float64`. A defensive pattern for files from outside the team is to read **every** column as text with `dtype='str'` and convert the few numeric ones explicitly afterwards. It costs some memory and an extra step, but no identifier is ever silently turned into a number, and a column that drifts from numbers to text in next month's file cannot change the frame's types behind your back. ## Parsing timestamps with `parse_dates` `parse_dates=['ts']` asks the reader to convert the named columns to datetimes once the text is read. ISO 8601 strings take a fast path. For other layouts, pass `date_format='%d/%m/%Y %H:%M'` (or a dict of formats per column), or `date_format='ISO8601'` for ISO strings that vary in shape. If a value cannot be parsed, the column is left unconverted instead of raising, so check the dtype after reading. In **pandas 3.0** the datetime **resolution** is inferred: parsed strings default to microseconds, so the column is `datetime64[us]`, falling back to nanoseconds only when a string carries nanosecond digits. Pandas 2.x always gave `datetime64[ns]`. Most code does not notice, but converting to integers gives values 1000 times smaller than before; the lesson on pandas datetime parsing covers `as_unit` for that. pandas 3.0 also enforced several removals around this argument: | 2.x usage | pandas 3.0 | |---|---| | `parse_dates={'when': ['date', 'time']}` | `TypeError`: only booleans and lists are accepted | | `parse_dates=[['date', 'time']]` to combine | no longer supported; combine the columns yourself, then convert | | `date_parser=` | removed in favour of `date_format=` | | `infer_datetime_format=True` | removed; a strict version is the default | | `keep_date_col=` | removed | ## Reading fewer columns with `usecols` `usecols=['ts', 'zip', 'page']` parses only those columns, which saves time and memory on a wide clickstream file. Two details are worth knowing: 1. **Order is ignored.** `usecols=['zip', 'ts']` returns the columns in file order; select `df[['zip', 'ts']]` afterwards if order matters. 2. **A callable works too.** `usecols=lambda c: c.startswith('utm_')` keeps every column whose name passes the test, which suits files whose column set drifts. ## A worked call for the clickstream file 1. List the columns you need in `usecols`. 2. Give every identifier-like column (`zip`, `session_id`) a `'str'` dtype. 3. Put timestamp columns in `parse_dates` and add `date_format` if they are not ISO. 4. Check `df.dtypes` once: expect `str` for text and `datetime64[us]` for ISO timestamps on 3.0.
- Your code checks df['zip'].dtype == 'object' after the read. What happens on pandas 3.0?The check turns False, because the column now has the `str` dtype rather than `object`. Code that branches on object dtype to find text columns silently stops matching. Use `pd.api.types.is_string_dtype(df['zip'])`, which is true for object and the string dtypes, or select with `select_dtypes(include=['str'])` once 2.x support is dropped.
- A legacy job used parse_dates=[['date', 'time']]. How do you port it to pandas 3.0?Combining columns inside `parse_dates` was removed in 3.0. Read `date` and `time` as text, join them, then convert: `pd.to_datetime(df['date'] + ' ' + df['time'], format='%Y-%m-%d %H:%M:%S')`. The explicit `format` keeps parsing strict and fast. `date_parser=` is gone too, so any custom parser moves into that post-read step.
- astype(str) after reading brings the leading zeros back
- dtype={'zip': 'str'} also keeps blank zip fields as empty strings
- In pandas 3.0 a text column read from CSV is object dtype
- parse_dates still combines a date and a time column in pandas 3.0
- Setting converters and dtype for one column applies both
- usecols=['zip', 'ts'] returns the columns in that order
Assumes pandas 3.0: dtype 'str' gives the str dtype (object in 2.x), parsed timestamps default to microseconds (nanoseconds in 2.x), and parse_dates no longer combines columns.