Why does a pandas int64 column of clinic visit counts turn float64 once a value goes missing, and how does Int64 keep it integer?
answer
- the storage has no spare slot
- the only NumPy marker is a float
- 3 quietly becomes 3.0
- values array plus a mask
- capital I, not lowercase
basics
~20 sNumPy's int64 has no way to store a missing value, so pandas writes the gap as NaN and upcasts the whole column to float64. The nullable Int64 dtype keeps int64 values plus a separate mask and shows the gap as pd.NA.
solid answer
~40 sA NumPy `int64` array must hold an integer in every slot, and pandas' marker for NumPy numbers is `NaN`, which is a float. So when a gap appears, through `reindex`, an outer or left join with no match, `pd.Series([3, None])`, or assigning `np.nan` with `.loc`, pandas upcasts the column to `float64`: 3 becomes 3.0, integers above 2**53 can lose precision, and casting back with `astype("int64")` raises `IntCastingNaNError`. A NumPy `bool` column turns `object` the same way. The nullable dtypes fix this: `Int64` (capital I, also `Int8` to `UInt64`), `Float64` and `boolean` store the values in one array and a boolean mask in another, so the gap is `pd.NA` and the column stays integer. Ask for it explicitly with `dtype="Int64"`; the `Series` constructor never infers it.
code
python · 15 linesimport pandas as pd
visits = pd.Series([3, 1, 4], index=["p01", "p02", "p03"])
print(visits.dtype) # int64
roster = ["p01", "p02", "p03", "p04"] # p04 has no visits recorded
print(visits.reindex(roster)) # float64: 3.0, 1.0, 4.0, NaN
nullable = visits.astype("Int64").reindex(roster)
print(nullable) # Int64: 3, 1, 4, <NA>
print((nullable + 1).dtype) # Int64, the gap stays <NA>
consented = pd.Series([True, False, True], index=["p01", "p02", "p03"])
print(consented.reindex(roster).dtype) # object
print(consented.astype("boolean").reindex(roster).dtype) # booleango deeper
Recall the symptom and the fix: a gap turns an int64 column into float64 (and a bool column into object), and dtype="Int64" with a capital I keeps integers with pd.NA.
Explain the mechanism: int64 has no spare bit pattern, NaN is a float, so pandas upcasts; nullable dtypes keep a values array plus a mask instead.
Trace where a gap enters a pipeline (reindex, an unmatched join, an empty cell on read) and catch the costs: IDs beyond 2**53 rounding, failed casts back, and typed sinks receiving floats.
Decide whether integer columns with possible gaps should be nullable by default across a codebase, weighing schema fidelity against the extension-dtype edge cases.
## The storage problem behind the upcast A pandas column backed by NumPy is one contiguous array of one **dtype**. An `int64` array reserves 8 bytes per slot and every bit pattern in those 8 bytes is a valid integer, so there is no pattern left over to mean "missing". For NumPy-backed numbers pandas uses **`NaN`** as the missing marker, and `NaN` exists only in floating-point types. When a gap has to be stored in an integer column, pandas therefore **upcasts** the column: it picks the smallest dtype that holds both the integers and `NaN`, which is `float64`. The same logic hits booleans harder. A NumPy `bool` column with a gap cannot become a float column of True and False, so it becomes **`object`**: an array of Python objects holding `True`, `False` and `NaN`, slower and no longer a boolean mask. ## Where the gap sneaks in In a clinic-visits table with an integer `visits` column, a gap rarely comes from typing `None`. It usually arrives from an operation: - **`reindex`** to a fuller list of patients, where new labels get `NaN`; - a **left or outer join** in which some patients have no matching visit rows (the lesson on merges covers the join itself); - building a column from Python data with a `None` in it, such as `pd.Series([3, None])`; - **assigning `np.nan`** into one cell with `.loc`, which upcasts the column in place; - **reading a file** whose integer column has empty cells, which arrives as `float64` unless you choose a nullable dtype at read time. The upcast is silent. No warning is raised, and the first symptom is often a report showing `3.0` visits. ## What the upcast costs 1. **Display and type checks.** Counts print as `3.0`, and code that checks for an integer dtype (with `pd.api.types.is_integer_dtype`, for example) now says no. 2. **Precision.** `float64` represents every integer exactly only up to 2**53. A 19-digit patient or order ID stored as `float64` can be rounded to a neighbouring value, so two IDs collide. 3. **No easy way back.** `astype("int64")` on a column that still holds `NaN` raises `IntCastingNaNError` ("Cannot convert non-finite values (NA or inf) to integer"), so you must deal with the gap first or pick a nullable target. 4. **Downstream schemas.** Writing the column to a typed store sends a float where an integer was expected. ## The nullable dtypes: values plus a mask pandas' answer is a family of **nullable extension dtypes**. Instead of one array, each column keeps two: the values in a NumPy array of the real type and a **mask**, a boolean array that is True where the value is missing. The missing slot holds some placeholder value that is never shown; reads return the **`pd.NA`** singleton. | Need | NumPy dtype (gap behaviour) | Nullable dtype | Missing marker | |---|---|---|---| | integers | `int64`, upcast to `float64` | `Int64` (also `Int8` to `Int32`, `UInt8` to `UInt64`) | `pd.NA` | | floats | `float64`, holds `NaN` | `Float64`, `Float32` | `pd.NA` | | True / False | `bool`, upcast to `object` | `boolean` | `pd.NA` | The string alias is case-sensitive: `"Int64"` with a capital I is the pandas nullable dtype, `"int64"` is NumPy's. Writing `pd.Series([3, None], dtype="int64")` raises an error because NumPy's int64 has no way to store `None`. Useful facts about these dtypes: - **They are opt-in.** `pd.Series([3, None])` still infers `float64` for backward compatibility, while `pd.array([3, None])` infers `Int64`. The pandas docs recommend passing the dtype explicitly. - **Operations stay nullable.** `s + 1` keeps `Int64` and the gap stays `<NA>`; `s == 1` returns a `boolean` column with `<NA>` in the gap; `s + 0.01` moves to `Float64`, not to NumPy `float64`. - **A new empty column needs a dtype.** `df["recall_due"] = pd.NA` creates an `object` column; `pd.Series(pd.NA, index=df.index, dtype="Int64")` creates a proper nullable one. - **Reductions skip the gap.** `sum()` and `mean()` ignore `pd.NA` by default, as they ignore `NaN`. ## Interview angle The question checks whether a candidate can explain the upcast instead of just noticing it. A strong answer names the missing bit pattern, the `float64` result, the precision and casting costs, and `Int64` with its mask as the fix. Converting a whole frame in one step is a casting topic; filling the gap with a value is the lesson on filling and dropping.
- Why is pd.Series([3, None]) float64 while pd.array([3, None]) is Int64?The two constructors use different inference rules. `pd.array` infers the nullable `Int64` dtype for integers with a gap. The `Series` constructor keeps the older NumPy rule for backward compatibility and produces `float64` with `NaN`. pandas recommends passing `dtype="Int64"` explicitly so the result does not depend on which constructor you used.
- A float64 ID column holding NaN must become integer again; what fails and what works?`astype("int64")` raises `IntCastingNaNError`, because NumPy's int64 cannot store the gap. `astype("Int64")` works when every non-missing value is a whole number, turning `NaN` into `pd.NA`. If the IDs exceeded 2**53 while they were floats, precision may already be lost, so the safe fix is to read or build the column as `Int64` from the start.
- pandas stores a missing integer as 0 so the column stays int64.
- The column only turns float64 if the data had decimals.
- astype('int64') will turn the NaN back into a missing integer.
- Int64 and int64 are the same dtype written two ways.
- pd.Series([1, None]) infers Int64 in pandas 3.0.