A sensor export built into a pandas 3.0 DataFrame shows temp_c as object and reading_count as float64 in info(); what does each tell you, and how do you fix the construction?
answer
- what object means once text is str
- a stray sentinel among numbers
- integers cannot hold a hole
- non-null count confirms the gap
- type each column before building
basics
~20 sIn pandas 3.0, object marks mixed values, such as an 'n/a' string among temperatures, since clean text infers str. float64 on a count means a missing reading upcast the integers. Clean the sentinel and build typed columns.
solid answer
~40 sIn pandas 3.0 a column of plain text infers `str` and a column of plain numbers infers `int64` or `float64`, so `object` on `temp_c` says the values are **mixed**: almost always a sentinel such as `"n/a"` among floats. The same column also drops out of `describe()`, which covers numeric columns only. `float64` on `reading_count` with a `Non-Null Count` below the entries says a reading is missing: NumPy `int64` has no slot for `NaN`, so the integers were upcast. Confirm by listing the non-float values of `temp_c`. Fix it at construction: turn the sentinel into `None` in the records, then build each column as a typed `Series` (`float64` for the temperature, so a leftover string raises `ValueError` at build time, and nullable `Int64` for the count, so gaps stay `<NA>` without becoming floats).
code
python · 23 linesimport pandas as pd
export = {
"sensor_id": [101, 102, 103],
"temp_c": [21.5, "n/a", 22.1],
"status": ["ok", "ok", None],
"reading_count": [10, None, 12],
}
raw = pd.DataFrame(export)
print(raw.dtypes.to_dict())
print(raw.describe().columns.tolist())
temps = [None if v == "n/a" else v for v in export["temp_c"]]
fixed = pd.DataFrame(
{
"sensor_id": export["sensor_id"],
"temp_c": pd.Series(temps, dtype="float64"),
"status": export["status"],
"reading_count": pd.Series(export["reading_count"], dtype="Int64"),
}
)
print(fixed.dtypes.to_dict())
print(fixed["reading_count"].tolist())go deeper
Recall that a missing value turns an integer column into float64, and that object means the column holds mixed Python values.
Explain why describe() drops temp_c, why the non-null count stays full for a string sentinel, and what Int64 changes.
Walk the diagnosis from info() to the offending values, and fix it where the frame is built so bad values fail loudly.
Consider where a team should enforce column types for raw exports, so that sentinels are rejected once rather than handled per notebook.
## The symptom A data engineer builds a DataFrame from a sensor export (a list of readings per device) and runs `df.info()`. The report shows: - `sensor_id` as `int64` with every value present; - `temp_c` as `object`, even though it should hold temperatures; - `status` as `str` with one value missing; - `reading_count` as `float64`, even though it counts readings, and with a non-null count below the number of entries. Each dtype is pandas telling you what it found in the input. Read correctly, the report points at the two defects without opening the raw file. ## What object means in pandas 3.0 **In pandas 3.0**, text infers the dedicated **`str` dtype**, which can hold only strings or missing values. Numbers infer `int64` or `float64`. So a constructor falls back to the general-purpose `object` dtype (a column of arbitrary Python objects) when the values fit no specific dtype: typically numbers mixed with strings, or nested dicts and lists. For `temp_c`, that means at least one non-numeric value sits among the floats. Sensor exports usually encode a failed read as a sentinel string such as `"n/a"`, `"ERR"` or `"--"`. Two more clues point the same way: - `describe()` leaves `temp_c` out, because by default it summarises numeric columns only; - the non-null count can be **complete**, because `"n/a"` is a real string, not a missing value. In pandas 2.x this reading was weaker: a clean text column was also `object`, so `object` alone did not separate "text" from "mixed". The 3.0 `str` default makes `object` a signal worth acting on. ## What float64 on a count means NumPy's `int64` stores every bit pattern as a number, so it has no marker for "missing". When even one reading lacks `reading_count` (a key absent from a record, or an explicit `None`), pandas stores `NaN`, a float value, and upcasts the whole column to `float64`; `10` then prints as `10.0`. The `Non-Null Count` column of `info()` confirms it: `2 non-null` against `3 entries` is one missing value. This behaviour is the same in pandas 2.x and 3.0. ## Confirming the cause Before fixing anything, find the actual offenders: 1. Compare each `Non-Null Count` with the `RangeIndex` entry count to locate the gaps. 2. List the values of `temp_c` that are not floats, for example with `df.loc[~df["temp_c"].map(lambda v: isinstance(v, float)), "temp_c"]`, or tally their Python types with `df["temp_c"].map(type).value_counts()`. 3. Check whether the sentinel is one value or several, because the fix must cover all of them. ## Fixing it at construction The constructor's own `dtype=` argument cannot do this job: it takes **one dtype for every column**, so `pd.DataFrame(data, dtype="float64")` would also try to turn `status` into floats and fail. The construction-time fix has two steps: 1. **Normalise the sentinel in the source data.** Replace `"n/a"` with `None` in the records before building, so it becomes a genuine missing value rather than a string. 2. **Build typed columns.** Put a typed `Series` per column into a dict: - `pd.Series(temps, dtype="float64")` for `temp_c`, so any sentinel you missed raises `ValueError: could not convert string to float` at build time instead of producing a silent `object` column; - `pd.Series(counts, dtype="Int64")` for `reading_count`, a **nullable integer** dtype (capital I) that keeps integers and marks gaps as `<NA>`; - leave `sensor_id` and `status` to inference, or type them too. | Column | What info() showed | Cause | Construction fix | |---|---|---|---| | `temp_c` | `object` | `"n/a"` among floats | sentinel to `None`, build as `float64` | | `reading_count` | `float64`, fewer non-null | a missing reading | build as `Int64` | | `status` | `str`, fewer non-null | a missing status | none needed | ## Where other lessons take over - Converting an existing column with `to_numeric(errors="coerce")` or `astype` is the lesson on casting. - Reading the export from a file with a reader's `na_values=` and `dtype=` is the CSV lesson. - How `Int64`, `pd.NA` and `NaN` differ is the lesson on NA-aware dtypes. The interview point is the reasoning chain: `object` in pandas 3.0 means mixed values, `float64` on a count means a hole, and both are fixed most cheaply where the frame is built.
- Why not pass dtype='float64' to the DataFrame constructor to catch the bad temperature?The constructor's `dtype=` applies one dtype to every column, so it would also try to convert `status` to floats and raise on `"ok"`. A typed `Series` per column scopes the type to `temp_c`, where a leftover sentinel raises `ValueError` at build time.
- Would the same info() report have pointed at temp_c in pandas 2.x?Less clearly. In 2.x, `status` would also have shown `object`, because text inferred `object` there, so `object` did not distinguish clean text from mixed values. In pandas 3.0 text infers `str`, so `object` stands out as mixed values.
- object on temp_c means the temperatures were stored as strings
- float64 on a count column means the source sent decimals
- An 'n/a' string is counted as missing in info()'s non-null count
- dtype= on the DataFrame constructor can type one column
- In pandas 3.0 every text column still shows as object
Assumes pandas 3.0: in 2.x text and mixed columns both showed object; in 3.0 text infers str, so object marks mixed values.