skip to content

Series, DataFrame and Index

Series, DataFrame and Index are the objects every pandas call returns, and under Copy-on-Write each result acts as a copy. Interviewers start with construction and alignment.

on this pageshow

explore

questions

15

In pandas, how does a DataFrame built from a dict of lists differ from one built from a list of dicts like API records?

level: juniorimportance: must knowfreq 70%

answer

  1. column-oriented versus row-oriented input
  2. keys become column labels both ways
  3. equal lengths versus ragged records
  4. an absent key leaves a hole
  5. integer column quietly turns float64

basics

~20 s

A dict of lists is column-oriented: each key is a column and all lists must share one length. A list of dicts is row-oriented: each dict is a row, columns are the union of keys, and a missing key becomes NaN.

solid answer

~40 s

Both shapes go straight into `pd.DataFrame(...)` and both get a default `RangeIndex` unless you pass `index=`. With a **dict of lists**, each key is a column in insertion order, and lists of different lengths raise `ValueError: All arrays must be of the same length`. With a **list of dicts** (what an orders API usually returns), each dict is a row and the columns are the union of all keys, in the order pandas first sees them, so a key that appears only in a later record is appended at the end. A record that lacks a key gets `NaN` in that cell, and because NumPy `int64` cannot hold `NaN`, an integer field such as `qty` comes back as `float64`. So records with missing keys build without complaint, with holes you should check with `info()`.

code

python · 18 lines
python
import pandas as pd

records = [
    {"order_id": "A1", "qty": 2, "price": 9.5},
    {"order_id": "A2", "qty": 1, "price": 4.0, "coupon": "SPRING"},
    {"order_id": "A3", "price": 12.0},
]
from_rows = pd.DataFrame(records)
print(from_rows.columns.tolist())
print(from_rows["qty"].tolist(), from_rows["qty"].dtype)

from_cols = pd.DataFrame({"order_id": ["A1", "A2"], "qty": [2, 1]})
print(from_cols.shape, from_cols["qty"].dtype)

try:
    pd.DataFrame({"order_id": ["A1", "A2", "A3"], "qty": [2, 1]})
except ValueError as exc:
    print("ValueError:", exc)

go deeper

for a junior

Recall which input is column-oriented and which is row-oriented, and that missing keys in records become NaN rather than an error.

for a middle

Explain the column order rules, the equal-length ValueError and why one absent key turns an integer field into float64.

for a senior

Show that you verify API records after building them: shape for row count, info for non-null counts, and an explicit columns list so page-to-page order cannot drift.

for a principal

Frame the tradeoff of tolerant construction: records that never fail to build move schema checking to the team, so decide where a missing field is caught.

## Two orientations of the same table A pandas **DataFrame** is a two-dimensional table of labelled rows and labelled columns, where every column holds one **dtype** (data type). The `pd.DataFrame` constructor accepts many input shapes, and the two you meet most often describe the same table from opposite directions: - a **dict of lists** (or arrays) is **column-oriented**: one key per column, one list of values per column; - a **list of dicts** is **row-oriented**: one dict per row, one key-value pair per cell. Python data from an HTTP API usually arrives as the second shape, because JSON responses list records. Data you assemble yourself, such as columns computed in a loop, is usually the first. ## Dict of lists: columns first When you pass `{"order_id": [...], "qty": [...]}`, pandas does the following: - each key becomes a **column label**, and the column order follows the dict's insertion order; - each list becomes the values of that column, and pandas infers one dtype per column from them; - every list must have the **same length**; otherwise the constructor raises `ValueError: All arrays must be of the same length`, it never pads or truncates; - the row labels are a default `RangeIndex` (0, 1, 2, ...) unless you pass `index=` with one label per row. The strict length check is a feature: a column that came out one value short is a bug upstream, and the constructor refuses to guess which row is missing. ## List of dicts: rows first When you pass `[{"order_id": "A1", "qty": 2}, {"order_id": "A2", "price": 4.0}]`, pandas treats each dict as one row: - the column labels are the **union** of the keys across all records; - the column order is the order in which keys are **first seen**, so a field that only appears in record 40 becomes the last column; - a record that lacks a key gets a missing value in that cell, shown as `NaN`; - a key present in some records but not others still builds; nothing raises. That last point is the trap. Real APIs omit optional fields (a `coupon` that most orders lack, a `qty` a buggy client dropped), and the frame builds silently with holes. A field of integers with one hole cannot stay `int64`, because NumPy's integer dtype has no slot for a missing value, so pandas upcasts it to `float64` and `2` prints as `2.0`. ## What pandas infers, side by side | Question | Dict of lists | List of dicts | |---|---|---| | What becomes a column? | each key | the union of all keys | | Column order | dict insertion order | order keys are first seen | | Unequal or missing data | `ValueError` on unequal lengths | `NaN` in the missing cell | | Default row labels | `RangeIndex` | `RangeIndex` | | Integer field with a gap | not possible (lengths must match) | becomes `float64` | ## Traps with API records 1. **A nested object stays nested.** If each record carries `"customer": {"id": 7}`, that column holds whole Python dicts in an `object` column. `pd.json_normalize(records)` flattens them into dotted columns such as `customer.id`. 2. **The ID is just another column.** The order ID is not used as the row label unless you ask: pass `index=` with a list of labels, use `DataFrame.from_records(records, index="order_id")`, which moves that field out of the columns, or call `set_index` afterwards (the lesson on axis labels covers that). 3. **`columns=` selects; it does not rename.** Passing `columns=["order_id", "qty"]` with a list of dicts keeps only those keys, in that order, and a name the records lack comes back as an all-NaN column. A typo therefore yields a silent empty column, not an error. 4. **Order can drift between pages.** If page 2 of an API response is built separately and its first record carries keys in a different order, the two frames' column orders differ; fix the order with `columns=` when you build them. ## Checking what you built Right after construction, confirm the result rather than trusting it: - `df.shape` gives `(rows, columns)` and catches a wrong record count; - `df.dtypes` or `df.info()` shows which numeric fields became `float64` because of a gap, and `info()` also prints each column's non-null count, so `2 non-null` out of `3 entries` locates the hole; - `df.head()` shows the first five rows, enough to see whether a nested dict or an unexpected column slipped in. The habit worth stating in an interview: pick the constructor input that matches the data's natural orientation, then verify columns, row count and dtypes before the first transformation, because a list of dicts never complains about missing fields.

  • What happens to a nested object inside each record, such as a customer dict?
    The constructor does not flatten it: the `customer` column holds whole Python dicts in an `object` column, which is awkward to filter or aggregate. `pd.json_normalize(records)` flattens nested keys into dotted columns such as `customer.id` and `customer.tier`, and its `sep` argument changes the separator.
  • How do you make the order ID the row label at construction time?
    Pass `index=` with one label per row, or use `DataFrame.from_records(records, index="order_id")`, which takes the named field out of the columns and uses it as the index. Calling `set_index("order_id")` after construction gives the same result in a separate step.
  • Why does `qty` print as 2.0 instead of 2 when one record lacks it?
    The default integer dtype is NumPy `int64`, which has no representation for a missing value. To store `NaN` for the absent record, pandas upcasts the whole column to `float64`. A nullable dtype such as `Int64` keeps integers with a `pd.NA` marker instead, which the lesson on NA-aware dtypes covers.

  • A list of dicts raises an error when records have different keys
  • Missing keys become None and an integer column stays int64
  • Columns built from records are sorted alphabetically
  • A dict of unequal-length lists is padded with NaN
  • The keys of a dict of lists become row labels by default
open as a page

In pandas, what do shape, dtypes, head(), info() and describe() each tell you about a DataFrame you have just built?

level: juniorimportance: must knowfreq 75%

basics

~20 s

shape gives (rows, columns); dtypes gives each column's type; head() shows the first five rows; info() prints the index, each column's non-null count and dtype; describe() gives count, mean, spread and quartiles of numeric columns unless include='all'.

open as a page

In pandas 3.0, if you take `email = customers['email']` and change a value in `email`, does the `customers` DataFrame change?

level: juniorimportance: must knowfreq 55%

basics

~20 s

No. In pandas 3.0 Copy-on-Write makes every indexing result, including a column taken with [], behave as a copy, so editing it leaves the parent DataFrame untouched; to change the frame, write to the frame itself.

open as a page

In pandas, what do DataFrame.set_index and reset_index do to a product table's row labels, and what does drop change in each?

level: juniorimportance: must knowfreq 60%

basics

~20 s

set_index moves one or more columns into the row labels and, by default, removes them from the columns; reset_index moves the labels back into columns and installs a fresh RangeIndex, or discards them when drop=True.

open as a page

In pandas, why does adding two stores' monthly revenue Series with s1 + s2 return NaN for some months, and what does add(fill_value=0) change?

level: middleimportance: must knowfreq 70%

basics

~20 s

pandas pairs values by index label and returns the union of both indexes, so a month only one Series has gets NaN. add(fill_value=0) treats the missing side as 0, except where both sides are missing.

open as a page

In pandas 3.0, why does `customers[customers['balance'] < 0]['status'] = 'review'` leave `customers` unchanged, and what is ChainedAssignmentError?

level: seniorimportance: must knowfreq 60%

basics

~20 s

The first indexing step returns a temporary that behaves as a copy; the value lands in it and is discarded. pandas 3.0 flags this with ChainedAssignmentError, a warning, not an exception; assign in one loc step instead.

open as a page

When do you reach for pandas DataFrame.from_records or DataFrame.from_dict(orient='index') instead of the plain DataFrame constructor?

level: middleimportance: should knowfreq 35%

basics

~20 s

Use from_records for tuples, structured arrays or cursor rows when you want a field as the index, fields excluded, Decimals coerced or only n rows read; use from_dict(orient='index') when a dict's keys are row IDs.

open as a page

In pandas, what do the index, columns and dtype arguments of the Series and DataFrame constructors do when the data already carries labels?

level: middleimportance: should knowfreq 45%

basics

~20 s

With labelled data (dict keys, record keys, Series indexes), index= and columns= select and reorder by label, and an absent label gives NaN; with unlabelled arrays and lists they name positions. dtype= forces one dtype on every column.

open as a page

In pandas 3.0, when does Copy-on-Write actually copy data, and why does `df = df.reset_index(drop=True)` avoid that copy?

level: middleimportance: should knowfreq 40%

basics

~20 s

pandas 3.0 shares data between a result and its source and copies only when one of them is written while both still reference it. Rebinding to the same name drops the old frame, so no copy is needed.

open as a page

In pandas, after pd.concat stacks two monthly product extracts, how do you detect duplicate index labels, and how do you stop them arising?

level: middleimportance: should knowfreq 40%

basics

~10 s

Check df.index.is_unique and list repeats with df.index.duplicated(keep=False). Prevent them with ignore_index=True on the concat, or guard a frame with set_flags(allows_duplicate_labels=False), which raises DuplicateLabelError when a repeat appears.

open as a page

In pandas, what does DataFrame.reindex do when a product table indexed by SKU is conformed to a full catalogue list of SKUs?

level: middleimportance: should knowfreq 50%

basics

~20 s

reindex returns a frame with exactly the listed labels, in list order: existing SKUs keep their rows, new SKUs get NaN (or fill_value), and SKUs absent from the list are dropped. It matches labels; it never renames them.

open as a page

A sensor export built into a pandas 3.0 DataFrame shows temp_c as object and reading_count as float64 in info(); what does each tell you, and how do you fix the construction?

level: seniorimportance: should knowfreq 40%

basics

~20 s

In pandas 3.0, object marks mixed values, such as an 'n/a' string among temperatures, since clean text infers str. float64 on a count means a missing reading upcast the integers. Clean the sentinel and build typed columns.

open as a page

In pandas 3.0, why does writing into the array from `df.to_numpy()` raise 'assignment destination is read-only', and how do you fix it?

level: seniorimportance: should knowfreq 35%

basics

~10 s

In pandas 3.0, when the array shares memory with the frame, pandas returns a NumPy view with its writeable flag off so writes cannot bypass Copy-on-Write. Use to_numpy(copy=True) for a writeable copy.

open as a page

When dividing one store's monthly revenue DataFrame by another's in pandas, why call DataFrame.align with join='inner' first, and what does it return?

level: seniorimportance: should knowfreq 35%

basics

~20 s

The / operator aligns on the union of rows and columns, filling unmatched cells with NaN. a.align(b, join="inner") returns both frames cut to their shared labels on both axes, so the ratio covers only comparable cells.

open as a page

Your team is moving a pandas 2.x data-cleaning codebase to pandas 3.0; how do you plan the Copy-on-Write migration and catch the breakage that stays silent?

level: principalimportance: should knowfreq 30%

basics

~20 s

Upgrade through pandas 2.3, make ChainedAssignmentError fail tests, read code for helpers and named subsets that expected writes to reach a parent, diff pipeline outputs across versions, then remove defensive copies, copy= arguments and the option.

open as a page