skip to content

Choosing a Dtype

pandas 3.0 stores text in a dedicated str dtype, offers categorical and Arrow-backed types, and casts with astype. Interviewers ask which dtype a column should have and its cost.

on this pageshow

explore

questions

19

In pandas, a scraped price column holds strings like '$1,200' and 'N/A' — why does astype(float) raise, and how do you convert it?

level: juniorimportance: must knowfreq 70%

answer

  1. strict cast versus forgiving parse
  2. one bad value stops the cast
  3. coerce failures to NaN
  4. strip symbols before coercing
  5. count the NaN you created

basics

~10 s

astype(float) is all-or-nothing: one value that does not parse, such as '$1,200' or 'N/A', raises ValueError. Strip the symbols with .str.replace, call pd.to_numeric(errors='coerce') so leftovers become NaN, then count those NaN.

solid answer

~30 s

`astype(float)` converts every value or fails: the first string that is not a plain number, here `'$1,200'`, raises `ValueError` and nothing is converted. `pd.to_numeric` parses the same way but takes `errors='coerce'`, which sets every unparseable entry to `NaN` instead of raising. Coercing the raw column is the trap: `'$1,200'` is a real price and would become `NaN` right beside `'N/A'`. So clean first with `s.str.replace(r'[$,]', '', regex=True)`, then coerce, then compare the missing count before and after so you know how many values were discarded. The result is `float64`, because a NumPy integer column cannot hold `NaN`.

code

python · 15 lines
python
import pandas as pd

raw = pd.Series(["$1,200", "N/A", "$85", " 40 "])

try:
    raw.astype(float)
except ValueError as exc:
    print("astype:", exc)

print(pd.to_numeric(raw, errors="coerce").tolist())

cleaned = raw.str.replace(r"[$,]", "", regex=True)
prices = pd.to_numeric(cleaned, errors="coerce")
print(prices.tolist(), prices.dtype)
print("discarded:", raw[prices.isna() & cleaned.notna()].tolist())

go deeper

for a junior

Recall that astype is strict and raises on the first bad value, while pd.to_numeric with errors='coerce' turns unparseable values into NaN.

for a middle

Explain why currency symbols and thousands separators must be stripped before coercion, and why the result is float64 rather than int64.

for a senior

Show the check: compare missing counts before and after coercion and list the raw values that were discarded, so a cleaning rule cannot silently erase real data.

for a principal

Argue for a team rule that every coercion logs or asserts how many values it discarded, because silent NaN creation corrupts downstream aggregates without failing any job.

## Why `astype` refuses the column `Series.astype(dtype)` asks pandas to convert **every** element to the target dtype. It is a strict cast: if any element cannot be converted, the call raises and returns nothing. A scraped price column usually holds text such as `'$1,200'`, `'85'`, `' 40 '` and `'N/A'`. The first element, `'$1,200'`, is not a number that Python's float parser understands, so `s.astype(float)` raises `ValueError: could not convert string to float: '$1,200'`. That strictness is a feature. `astype` is the right tool when you **know** the data is clean and want a loud failure if it is not. It is the wrong tool for a column you already know is dirty. ## `pd.to_numeric` and its `errors` argument `pd.to_numeric(arg, errors='raise', downcast=None, dtype_backend=...)` is the parsing function for one-dimensional data: a scalar, a list, an array or a Series. Its `errors` argument decides what happens to a value that will not parse: | `errors=` | What an unparseable value does | When to use it | |---|---|---| | `'raise'` (default) | raises `ValueError` naming the value and its position | the data should be clean; a failure is a bug | | `'coerce'` | becomes `NaN` | the column is known to hold junk you are willing to treat as missing | Two details of the parser matter for prices: - It accepts surrounding spaces, so `' 40 '` parses as `40`. - It does **not** understand currency symbols or thousands separators, so `'$1,200'` and `'1,200'` both fail. That second point is why coercion alone is not the fix. ## Clean, then coerce The reliable sequence is: 1. **Strip the formatting** that makes valid numbers unparseable: `s.str.replace(r'[$,]', '', regex=True)`. After this `'$1,200'` is `'1200'`. 2. **Coerce what is left** with `pd.to_numeric(cleaned, errors='coerce')`. Placeholders such as `'N/A'`, `'-'` or `'call'` become `NaN`. 3. **Measure the loss**: compare `cleaned.isna().sum()` before with `result.isna().sum()` after, and look at the raw values that turned into `NaN`. 4. **Decide**: if the newly missing values are all placeholders, keep the result; if some are real prices in a format you did not anticipate (a `'€'` sign, a trailing `'k'`), extend step 1 and run again. Step 3 is what separates a careful answer from a careless one. `errors='coerce'` never tells you what it threw away; a typo in step 1 can silently turn half a column into `NaN`. ```python import pandas as pd raw = pd.Series(["$1,200", "N/A", "$85", " 40 "]) cleaned = raw.str.replace(r"[$,]", "", regex=True) prices = pd.to_numeric(cleaned, errors="coerce") lost = raw[prices.isna() & cleaned.notna()] ``` `lost` holds exactly the raw strings that coercion discarded, here only `'N/A'`. ## What the result looks like - When at least one value was coerced, the result is **`float64`**: `NaN` is a floating-point value, and a NumPy `int64` column has no slot for it. Whole-number prices therefore print as `1200.0`. - When nothing failed and every value is whole, `to_numeric` returns **`int64`**; the docstring says the default result is `float64` or `int64` "depending on the data supplied". - `downcast='integer'`, `'signed'`, `'unsigned'` or `'float'` shrinks a successfully parsed result to the smallest dtype of that family that holds every value. A coerced column that still contains `NaN` cannot become an integer, so `downcast='integer'` leaves it `float64`. - To keep whole prices as integers despite the gaps, cast the result with `astype('Int64')` or pass `dtype_backend='numpy_nullable'` to `to_numeric`; how that missing marker behaves is covered in the lesson on NA-aware dtypes. ## Choosing between the two tools - Use **`astype`** when the column must already be clean, for example one your own code produced; a failure then signals an upstream bug and should stop the job. - Use **`pd.to_numeric(errors='coerce')`** at the edge of the system, where data arrives from scraping, spreadsheets or user input, and always pair it with the discard check. - Use **`pd.to_numeric`** with its default `errors='raise'` when you want its parser but a loud failure: its message names the offending value and its position (`Unable to parse string "$1,200" at position 0`), which `astype`'s message does not. Make the choice per column. A loop that runs the same conversion over every column of a frame treats product names, identifiers and prices alike, and that is where silent damage starts. ## Mistakes interviewers listen for - Calling `astype(float)` and wrapping it in a bare `try/except` that swallows the error, leaving the column as text. - Coercing the raw column and reporting a mean that silently excludes every price written with a currency sign or a thousands separator. - Assuming `to_numeric` reads locale formats; it parses plain numerals (sign, decimal point, exponent), not currency or grouping characters. - Forgetting that `pd.to_numeric` takes one dimension: `pd.to_numeric(df)` raises `TypeError`, so a frame is converted column by column, ideally only on the columns you named. A good interview answer names both tools, says which one fails loudly and which one forgives, and then shows the check that makes forgiveness safe.

  • What does to_numeric's downcast argument do, and when does it have no effect?
    After a successful parse, `downcast='integer'` or `'signed'` picks the smallest signed integer dtype that holds every value (from `int8` up), `'unsigned'` starts at `uint8`, and `'float'` starts at `float32`. It only shrinks; if no smaller dtype fits, the result is unchanged. A column that still holds `NaN` cannot become an integer, so `downcast='integer'` on a coerced column with gaps stays `float64`.
  • Why not simply run pd.to_numeric(s, errors='coerce') on the raw column?
    Because coercion cannot tell junk from formatting. `'$1,200'` is a valid price written in a form the parser does not read, and it becomes `NaN` exactly like `'N/A'`. The column then looks numeric and every statistic on it is quietly wrong. Clean the formatting first, and compare missing counts before and after so the discarded values are visible.
  • How do you keep whole-number prices as integers when some are missing?
    Cast the coerced result with `astype('Int64')`, a nullable integer dtype, or call `pd.to_numeric(cleaned, errors='coerce', dtype_backend='numpy_nullable')`, which returns `Int64` when every parsed value is whole. Plain `astype(int)` raises on the missing entries because NumPy integers cannot represent them.

  • astype(float) skips the values it cannot parse and converts the rest
  • pd.to_numeric understands currency symbols and thousands separators
  • errors='coerce' is safe on the raw column without cleaning first
  • A coerced column containing NaN can still be int64 if the other values are whole
  • Coercion needs no follow-up check because only junk turns into NaN
open as a page

In pandas 3.0, what dtype does a new column of text get by default, and what can that column hold?

level: juniorimportance: must knowfreq 60%

basics

~10 s

In pandas 3.0 text is inferred as the str dtype, a StringDtype whose missing value is NaN, stored in PyArrow when installed and in Python objects otherwise. It holds only strings and missing values.

open as a page

In pandas, what happens when df.astype({'qty': int, 'price': 'float64'}) meets a column with gaps or an unparseable value, and what does errors= change?

level: middleimportance: must knowfreq 60%

basics

~10 s

astype with a dict casts only the named columns. A gap makes int raise IntCastingNaNError; use 'Int64'. A bad string raises ValueError naming the column, unless errors='ignore', which silently leaves just that column uncast.

open as a page

In pandas 3.0, when does astype('category') shrink a text column's memory, and when does it make the column larger?

level: middleimportance: must knowfreq 60%

basics

~20 s

A category column stores each distinct value once plus one small integer code per row, so it shrinks low-cardinality text sharply. When nearly every value is unique, the categories duplicate the data and the codes add bytes, so the column grows.

open as a page

In pandas, how does Series.str.extract turn product SKU codes like 'TSH-L-RED' into separate columns, and what decides the result's shape?

level: middleimportance: must knowfreq 55%

basics

~20 s

Series.str.extract applies a regex to each string and returns one column per capture group, labelled by named groups, from the first match only. Non-matching or missing rows get missing values; expand=False with one group returns a Series.

open as a page

In pandas, how do you make a T-shirt size column sort S, M, L, XL, and why does astype('category') alone not do it?

level: juniorimportance: should knowfreq 50%

basics

~20 s

Cast with an explicit pd.CategoricalDtype(['S', 'M', 'L', 'XL'], ordered=True). Plain astype('category') infers the categories in sorted text order and leaves them unordered, so sorting still gives L, M, S, XL and min, max and < raise TypeError.

open as a page

In pandas, what do infer_objects() and convert_dtypes() each do to a DataFrame's columns, and why does neither turn the string '42' into a number?

level: middleimportance: should knowfreq 40%

basics

~20 s

infer_objects() re-infers object columns from the Python objects they hold; convert_dtypes() moves each column it can to a pd.NA-capable dtype such as Int64, Float64, boolean or string. Neither parses text: '42' stays a string, so use pd.to_numeric.

open as a page

In pandas, why does assigning 'Delighted' into an ordered satisfaction-scale categorical column raise TypeError, and how do you add the new answer properly?

level: middleimportance: should knowfreq 40%

basics

~10 s

A pandas categorical accepts only values already in its categories, so setting 'Delighted' raises TypeError. Add it first with s.cat.add_categories(['Delighted']), which appends it as the highest level, or use set_categories to place it elsewhere.

open as a page

In pandas, why can sku.str.contains('BLK+GRN') miss the value 'CAP-OS-BLK+GRN' while sku.str.replace('+', '/') edits it correctly?

level: middleimportance: should knowfreq 45%

basics

~20 s

Series.str.contains treats its pattern as a regular expression by default (regex=True), so '+' means one or more K. Series.str.replace defaults to regex=False and replaces the literal '+'. Pass regex=False to contains for a literal match.

open as a page

In pandas, what does Series.str.split with expand=True return for SKU codes with different numbers of parts, and how do n and rsplit help?

level: middleimportance: should knowfreq 58%

basics

~20 s

With expand=True, Series.str.split returns a DataFrame with one column per piece, numbered from 0, and pads shorter rows with missing values on the right. n caps the number of splits and rsplit counts them from the end.

open as a page

In pandas 3.0, how do the default 'str' dtype, the opt-in 'string' dtype and object dtype differ for a text column?

level: middleimportance: should knowfreq 42%

basics

~20 s

In pandas 3.0, 'str' is a StringDtype using NaN, 'string' is a StringDtype using pd.NA, and object holds any Python objects. They differ in missing-value semantics and in the dtypes that .str methods and comparisons return.

open as a page

After upgrading to pandas 3.0, a cleaning step df.apply(pd.to_numeric, errors='ignore') raises ValueError, and it had been stripping leading zeros from ID codes — what do you replace it with?

level: seniorimportance: should knowfreq 45%

basics

~20 s

In pandas 3.0, to_numeric no longer accepts errors='ignore', so the call raises. Replace the blanket conversion with an explicit list of numeric columns, coerce those with a NaN check, and keep ID codes as text.

open as a page

After an upgrade to pandas 3.0, a sales report grouped by a categorical T-shirt size column no longer shows the XL row with zero sales; why, and how do you restore it?

level: seniorimportance: should knowfreq 45%

basics

~20 s

In pandas 3.0 groupby defaults to observed=True, so a categorical grouper yields only the categories present in the data. Pass observed=False to get a row for every category, XL included, with 0 for sum, count and size and NaN for mean.

open as a page

In pandas, why does df[df['satisfaction'] >= 'Satisfied'] filter one survey file correctly but raise TypeError on another file with the same column?

level: seniorimportance: should knowfreq 35%

basics

~20 s

An inequality on a pandas categorical needs ordered=True and a scalar that is one of its categories. One file's column was cast with the ordered scale; the other was left unordered by astype('category'), or the label is misspelled, so pandas raises TypeError.

open as a page

After upgrading a product-catalog pipeline to pandas 3.0, the check df['sku'].dtype == object returns False — what changed, and what else breaks with it?

level: seniorimportance: should knowfreq 48%

basics

~20 s

In pandas 3.0 the sku column is inferred as the str dtype, not object, so the equality is False. The same change turns None into NaN and makes writing a non-string into the column raise TypeError.

open as a page

As the lead standardising new pandas 3.0 pipelines, would you default to NumPy dtypes, dtype_backend='numpy_nullable' or dtype_backend='pyarrow', and what does each choice cost?

level: principalimportance: should knowfreq 30%

basics

~20 s

There is no single right default. NumPy dtypes maximise compatibility but turn integer gaps into floats; numpy_nullable keeps integers and booleans with gaps without PyArrow; pyarrow adds decimals, lists and Arrow interchange, costing a dependency and experimental status.

open as a page

As the lead of a team whose pandas pipelines carry many low-cardinality text columns, would you make converting them to category at load time a standard, and what does that rule cost?

level: principalimportance: should knowfreq 40%

basics

~20 s

Not as a blanket rule. Category saves memory and speeds grouping on low-cardinality columns, but new labels raise on assignment, concat and merge with different categories fall back to str, and 3.0's observed=True drops empty groups. I would allow it for listed columns with fixed category sets.

open as a page

Your team's shared pandas helper library must run on both pandas 2.x and 3.0 while services upgrade at different times; what rules do you set for code that touches text columns, and how do you test them?

level: principalimportance: should knowfreq 40%

basics

~20 s

Write text code that never depends on which dtype pandas inferred: test dtypes with is_string_dtype, select with ['object', 'string'], check missing values with pd.isna, cast to object before storing non-strings, and run the tests on 2.x, on 2.3 with infer_string enabled and on 3.0.

open as a page

A pandas frame converted with convert_dtypes(dtype_backend='pyarrow') now reports dtypes like 'int64[pyarrow]' — what are these ArrowDtype columns, and what should downstream code check?

level: seniorimportance: nice to knowfreq 25%

basics

~20 s

ArrowDtype columns hold their data in PyArrow arrays instead of NumPy, so every type can carry missing values and gains Arrow types such as decimals and lists. Downstream, replace dtype string equality checks and convert explicitly at NumPy hand-offs.

open as a page