skip to content

Pandas

Python's DataFrame library: labelled Series and frames, selection under Copy-on-Write, dtypes, groupby, merges, reshaping and time series. Interviewers watch for column thinking.

on this pageshow

explore

questions

143 · 13 sections

In pandas, how does a DataFrame built from a dict of lists differ from one built from a list of dicts like API records?

level: juniorimportance: must knowfreq 70%
basics
~20 s

A dict of lists is column-oriented: each key is a column and all lists must share one length. A list of dicts is row-oriented: each dict is a row, columns are the union of keys, and a missing key becomes NaN.

open as a page

In pandas, what do shape, dtypes, head(), info() and describe() each tell you about a DataFrame you have just built?

level: juniorimportance: must knowfreq 75%
basics
~20 s

shape gives (rows, columns); dtypes gives each column's type; head() shows the first five rows; info() prints the index, each column's non-null count and dtype; describe() gives count, mean, spread and quartiles of numeric columns unless include='all'.

open as a page

In pandas 3.0, if you take `email = customers['email']` and change a value in `email`, does the `customers` DataFrame change?

level: juniorimportance: must knowfreq 55%
basics
~20 s

No. In pandas 3.0 Copy-on-Write makes every indexing result, including a column taken with [], behave as a copy, so editing it leaves the parent DataFrame untouched; to change the frame, write to the frame itself.

open as a page

In pandas, what do DataFrame.set_index and reset_index do to a product table's row labels, and what does drop change in each?

level: juniorimportance: must knowfreq 60%
basics
~20 s

set_index moves one or more columns into the row labels and, by default, removes them from the columns; reset_index moves the labels back into columns and installs a fresh RangeIndex, or discards them when drop=True.

open as a page

In pandas, why does adding two stores' monthly revenue Series with s1 + s2 return NaN for some months, and what does add(fill_value=0) change?

level: middleimportance: must knowfreq 70%
basics
~20 s

pandas pairs values by index label and returns the union of both indexes, so a month only one Series has gets NaN. add(fill_value=0) treats the missing side as 0, except where both sides are missing.

open as a page

In pandas, how do you keep only the DataFrame rows that satisfy two conditions at once, and why does each comparison need its own parentheses?

level: juniorimportance: must knowfreq 80%
basics
~20 s

Build one boolean Series per condition and combine them with & (and), | (or) and ~ (not), wrapping each comparison in parentheses: Python binds & tighter than > or ==, so an unparenthesised filter parses wrongly and raises.

open as a page

In pandas, what do emp['salary'], emp[['salary']] and emp[0:2] return, and why does emp[0] raise KeyError?

level: juniorimportance: must knowfreq 70%
basics
~20 s

On a DataFrame, [] with one name returns that column as a Series, a list returns a DataFrame, and an integer slice returns rows by position. emp[0] raises KeyError because a scalar key looks for a column named 0.

open as a page

In pandas, how do df.loc and df.iloc differ on a frame indexed by non-sequential employee IDs, and what does each slice include?

level: juniorimportance: must knowfreq 85%
basics
~20 s

df.loc selects by index label and its slices include both endpoints; df.iloc selects by 0-based position and its slices exclude the stop. On an ID index, loc[1007] is employee 1007 and iloc[1] is the second row.

open as a page

In pandas, how do you set a column's value only on the rows that match a condition, such as flagging suspicious payments?

level: juniorimportance: must knowfreq 75%
basics
~20 s

Use one loc call with the boolean mask as the row selector and the column label as the column selector: df.loc[df['amount'] > 5000, 'is_fraud'] = True. Initialise a new column first so its dtype stays clean.

open as a page

In pandas 3.0, what triggers 'ValueError: The truth value of a Series is ambiguous', and how do you fix each trigger?

level: middleimportance: must knowfreq 65%
basics
~20 s

Anything that asks a whole Series for one bool raises it: and, or, not, if, a chained comparison like 10 < s < 50, or comparisons joined by & without parentheses. Use &, |, ~ element-wise, or reduce with .any(), .all(), .empty or .item().

open as a page

In pandas, a scraped price column holds strings like '$1,200' and 'N/A' — why does astype(float) raise, and how do you convert it?

level: juniorimportance: must knowfreq 70%
basics
~10 s

astype(float) is all-or-nothing: one value that does not parse, such as '$1,200' or 'N/A', raises ValueError. Strip the symbols with .str.replace, call pd.to_numeric(errors='coerce') so leftovers become NaN, then count those NaN.

open as a page

In pandas 3.0, what dtype does a new column of text get by default, and what can that column hold?

level: juniorimportance: must knowfreq 60%
basics
~10 s

In pandas 3.0 text is inferred as the str dtype, a StringDtype whose missing value is NaN, stored in PyArrow when installed and in Python objects otherwise. It holds only strings and missing values.

open as a page

In pandas, what happens when df.astype({'qty': int, 'price': 'float64'}) meets a column with gaps or an unparseable value, and what does errors= change?

level: middleimportance: must knowfreq 60%
basics
~10 s

astype with a dict casts only the named columns. A gap makes int raise IntCastingNaNError; use 'Int64'. A bad string raises ValueError naming the column, unless errors='ignore', which silently leaves just that column uncast.

open as a page

In pandas 3.0, when does astype('category') shrink a text column's memory, and when does it make the column larger?

level: middleimportance: must knowfreq 60%
basics
~20 s

A category column stores each distinct value once plus one small integer code per row, so it shrinks low-cardinality text sharply. When nearly every value is unique, the categories duplicate the data and the codes add bytes, so the column grows.

open as a page

In pandas, how does Series.str.extract turn product SKU codes like 'TSH-L-RED' into separate columns, and what decides the result's shape?

level: middleimportance: must knowfreq 55%
basics
~20 s

Series.str.extract applies a regex to each string and returns one column per capture group, labelled by named groups, from the first match only. Non-matching or missing rows get missing values; expand=False with one group returns a Series.

open as a page

In pandas, how do you count the missing values in each DataFrame column, and how do dropna() and fillna(0) treat them differently?

level: juniorimportance: must knowfreq 80%
basics
~10 s

df.isna().sum() counts the missing cells in each column and df.isna().mean() gives their share. dropna() removes every row holding any missing value, while fillna(0) keeps every row and writes 0 into each gap.

open as a page

In pandas 3.0, which missing marker (NaN, NaT, None or pd.NA) does each dtype hold, and why test for them with isna rather than ==?

level: juniorimportance: must knowfreq 62%
basics
~20 s

NumPy float64 and the pandas 3.0 str dtype hold NaN, datetime64 and timedelta64 hold NaT, object holds what was stored (often None), and Int64, Float64, boolean and string hold pd.NA. pd.isna detects all four; == finds none.

open as a page

A pandas Series of minute-level sensor temperatures has three-minute dropouts and one three-hour outage; what does ffill(limit=5) do to each?

level: middleimportance: must knowfreq 60%
basics
~20 s

ffill copies the last valid reading forward, so each three-minute dropout is filled completely. limit=5 caps each gap at five filled rows, so the outage gets five copies at its start and its remaining rows stay NaN.

open as a page

Why does a pandas int64 column of clinic visit counts turn float64 once a value goes missing, and how does Int64 keep it integer?

level: middleimportance: must knowfreq 68%
basics
~20 s

NumPy's int64 has no way to store a missing value, so pandas writes the gap as NaN and upcasts the whole column to float64. The nullable Int64 dtype keeps int64 values plus a separate mask and shows the gap as pd.NA.

open as a page

In pandas, how do DataFrame.dropna's how, thresh, subset and axis arguments decide which rows or columns are removed?

level: middleimportance: should knowfreq 55%
basics
~20 s

dropna removes rows by default when any cell is missing. how='all' drops only fully empty rows, thresh=n keeps rows with at least n non-missing values, subset limits which columns are checked, and axis='columns' drops columns instead.

open as a page

In pandas, what does `Series.map` return when you pass it a dict, and what happens to values the dict has no key for?

level: juniorimportance: must knowfreq 62%
basics
~20 s

Series.map with a dict returns a new Series on the same index, each value replaced by its dict lookup. Any value without a key becomes NaN, unless the dict defines missing, which then supplies the default.

open as a page

A pandas loan table assigns risk tiers with `df.apply(tier, axis=1)`; how do you replace that apply with `np.where` or `np.select`?

level: middleimportance: must knowfreq 74%
basics
~20 s

Build each condition as a boolean Series over whole columns, then use np.where for two outcomes or np.select(conditions, choices, default=...) for several, where the first true condition wins. Handle NaN explicitly, since it fails every comparison.

open as a page

In a pandas method chain, why should DataFrame.assign receive a lambda rather than a column taken from the original frame?

level: middleimportance: must knowfreq 58%
basics
~20 s

DataFrame.assign calls a lambda with the intermediate frame at that step, while trips['fare'] reads the original frame: columns created earlier in the chain are missing (KeyError), changed values are stale, and relabelled rows silently misalign.

open as a page

Why does calling rename(..., inplace=True) in the middle of a pandas method chain break the chain?

level: juniorimportance: should knowfreq 52%
basics
~10 s

DataFrame.rename with inplace=True modifies the frame and returns None, so the next method is called on None and raises AttributeError; in a chain, let every method return a new frame instead.

open as a page

In pandas, when do you bin a credit-score column with `pd.cut` and when with `pd.qcut`, and what do their defaults do to edge values?

level: middleimportance: should knowfreq 48%
basics
~20 s

pd.cut uses edges you choose, such as policy thresholds; pd.qcut picks edges at quantiles so bins hold about equal counts. cut's intervals are right-closed, so the lowest edge is excluded unless include_lowest=True, and values outside the edges become NaN.

open as a page

In pandas, how do DataFrame.duplicated and drop_duplicates differ, and which copy of a repeated row counts as the original by default?

level: juniorimportance: must knowfreq 72%
basics
~20 s

DataFrame.duplicated returns a boolean Series flagging repeated rows; drop_duplicates returns a new DataFrame without them. Both compare all columns unless subset is given, ignore the index, and by default keep='first' treats the first occurrence as the original.

open as a page

In pandas, how do you sort a DataFrame by one column in descending order, and where do missing values end up?

level: juniorimportance: must knowfreq 78%
basics
~10 s

Call df.sort_values('revenue', ascending=False), which returns a new DataFrame; missing values go last in either direction unless na_position='first', and every row keeps its original index label.

open as a page

In pandas, how do you keep only each contact's most recent sign-up when a CRM export repeats contacts across sign-ups?

level: middleimportance: must knowfreq 66%
basics
~10 s

drop_duplicates keeps a copy by row position, not by date, so sort first: crm.sort_values('signup', ascending=False).drop_duplicates('contact_id') keeps each contact's latest dated row; ignore_index=True renumbers the result.

open as a page

In pandas, how do Series.value_counts(normalize=True) shares of churn reasons change when some reasons are missing and you pass dropna=False?

level: middleimportance: must knowfreq 64%
basics
~20 s

By default value_counts drops missing values from both the counts and the denominator, so the shares of known reasons sum to 1. dropna=False adds a NaN row and divides by every row, so each known share shrinks.

open as a page

With pandas sort_values, how do you sort by several columns with a different direction for each, and what constraints apply?

level: middleimportance: must knowfreq 62%
basics
~10 s

Pass lists to both arguments: df.sort_values(by=['revenue', 'product'], ascending=[False, True]). The ascending list must match by's length or pandas raises ValueError, and na_position is a single setting for every key.

open as a page

In pandas, what does `orders.groupby('customer_id')['amount'].sum()` return, and how do you get `customer_id` back as a column?

level: juniorimportance: must knowfreq 80%
basics
~20 s

A Series of totals, one per customer, whose index holds the sorted customer_id values and whose name is amount. Pass as_index=False to groupby, or call reset_index() on the result, to get a DataFrame with customer_id as a column.

open as a page

In pandas, how do you compute several differently named summary columns per customer with `groupby().agg`, and why prefer named aggregation?

level: middleimportance: must knowfreq 70%
basics
~20 s

Pass keywords to agg, each naming an output column and pairing a source column with a function: agg(total=('amount', 'sum'), orders=('amount', 'size')). Lists and dicts of functions produce two-level column labels to flatten; named aggregation gives flat, chosen names.

open as a page

In pandas, how do `groupby().size()`, `groupby().count()` and `groupby().nunique()` differ when the `amount` column has missing values?

level: middleimportance: must knowfreq 65%
basics
~20 s

size() counts every row in each group, missing values included, and returns one number per group. count() counts non-missing values per column, so a NaN amount lowers it. nunique() counts distinct non-missing values per column.

open as a page

In pandas, how does groupby transform('sum') put each store's total on every row, and how does transform with a function differ?

level: middleimportance: must knowfreq 70%
basics
~20 s

transform returns a result with the grouped frame's index and length. transform('sum') broadcasts each store's total onto its rows for a share column; a function runs per group in Python and must return a scalar or a group-length result.

open as a page

In pandas, why is a per-department z-score computed with groupby apply slow when there are many departments, and what do you replace it with?

level: seniorimportance: must knowfreq 65%
basics
~20 s

groupby apply calls a Python function once per group and then concatenates the pieces and infers the result's shape, so its cost grows with the number of groups. (x - g.transform('mean')) / g.transform('std') computes the same z-score with grouped built-ins.

open as a page

In pandas, when do you combine two DataFrames with merge, with DataFrame.join, or with pd.concat?

level: juniorimportance: must knowfreq 78%
basics
~20 s

Use merge to match rows on key columns (default how='inner'), DataFrame.join to match against the other frame's index (default how='left'), and pd.concat to stack frames along an axis while aligning the labels of the other axis (default join='outer').

open as a page

In pandas, how do you find customers who never placed an order with merge, and what did pandas 3.0 add for it?

level: middleimportance: must knowfreq 62%
basics
~10 s

Left-merge customers to orders with indicator=True and keep rows whose _merge is 'left_only'. In pandas 3.0, merge(how='left_anti') returns those customers directly.

open as a page

In pandas, how does merge's validate argument stop an orders-to-customers merge from silently multiplying rows?

level: middleimportance: must knowfreq 55%
basics
~10 s

validate declares the expected key relationship; with validate='many_to_one', merge checks that customer_id is unique in the customers frame and raises MergeError instead of returning extra rows when it is not.

open as a page

In pandas, after pd.concat of twelve monthly DataFrames, why does df.loc[0] return twelve rows, and what do ignore_index and keys change?

level: seniorimportance: should knowfreq 45%
basics
~20 s

pd.concat keeps each piece's index, so twelve frames that each start at 0 stack into twelve rows labelled 0. ignore_index=True renumbers 0..n-1; keys=[...] adds an outer index level naming each month. In pandas 3.0, passing both raises ValueError.

open as a page

Should a data team make merge's validate argument and a row-count check a standing review rule on every pandas merge in its pipelines?

level: principalimportance: should knowfreq 30%
basics
~20 s

Usually yes for pipeline code: require explicit how= and validate= on every merge, plus a row-count or match-rate assertion where the business has an expectation. Keep notebooks exempt, and accept the cost of failing runs and an extra uniqueness pass.

open as a page

In pandas, how do you use `melt` to turn a wide survey export with one column per question back into one row per answer?

level: juniorimportance: must knowfreq 60%
basics
~10 s

Call df.melt(id_vars="respondent", value_vars=["q1", "q2", "q3"], var_name="question", value_name="score"). Each listed column becomes rows: the old header goes into the var_name column and its cell into the value_name column.

open as a page

In pandas, why does `pivot` raise on a long station, date and reading weather table when `pivot_table` succeeds, and what does `pivot_table` add?

level: middleimportance: must knowfreq 70%
basics
~10 s

DataFrame.pivot only reshapes, so two rows sharing an index and column pair raise ValueError: Index contains duplicate entries, cannot reshape. pivot_table aggregates those duplicates with aggfunc (default mean) and adds fill_value and margins.

open as a page

In pandas, what does `pd.crosstab(obs["station"], obs["condition"], normalize="index")` return, and how do margins change it?

level: middleimportance: should knowfreq 40%
basics
~20 s

pd.crosstab counts rows per station and condition pair; normalize="index" divides each count by its row total so every station's shares sum to 1. With margins=True it also adds an All row holding the overall shares.

open as a page

In pandas, what does `DataFrame.explode` do to the index, to empty lists and to the dtype when a list-valued sensors column is exploded?

level: middleimportance: should knowfreq 40%
basics
~20 s

DataFrame.explode gives each list element its own row, copying the other columns and repeating the original index label. An empty list becomes one row with NaN, and the exploded column keeps object dtype for numbers.

open as a page

In pandas, why can `pd.get_dummies(batch, columns=["plan"], drop_first=True)` on a scoring batch silently disagree with the columns built from the training frame?

level: seniorimportance: should knowfreq 42%
basics
~20 s

get_dummies builds one column per level present in the data it is given, and drop_first drops the first of those levels. A batch missing some plans gets different columns, so fix the levels with a CategoricalDtype before encoding.

open as a page

In pandas, how do you turn a text date column into real datetimes and then pull out its year and weekday?

level: juniorimportance: must knowfreq 70%
basics
~10 s

Convert the column with pd.to_datetime and assign it back, then read its parts through the .dt accessor: .dt.year for the year and .dt.dayofweek (Monday is 0) or .dt.day_name() for the weekday.

open as a page

In pandas, how do you roll minute-level web traffic up to hourly and daily totals with resample, and what does the result look like?

level: juniorimportance: must knowfreq 70%
basics
~10 s

Call resample with a rule on a frame that has a DatetimeIndex, then aggregate: traffic.resample('h').sum() and traffic.resample('D').sum(). The result has one row per hour or day, stamped with each bucket's start time.

open as a page

With pd.to_datetime in pandas 3.0, how do format, errors='coerce' and unit='s' handle an event log of mixed date strings and epoch seconds?

level: middleimportance: must knowfreq 60%
basics
~10 s

pd.to_datetime applies one format to the whole column, guessed from the first non-missing value unless format= is given; errors='coerce' turns non-matching rows into NaT, and unit='s' reads integers as seconds since 1970.

open as a page

In pandas, what is the difference between Series.dt.tz_localize and Series.dt.tz_convert, and when do you call each one?

level: middleimportance: must knowfreq 55%
basics
~10 s

tz_localize attaches a zone to naive datetimes and keeps their clock reading; tz_convert takes already-aware datetimes and shows the same instants in another zone. Localize once at ingest, convert whenever you need local display.

open as a page

In pandas, how do `rolling(7)` and `rolling('7D')` differ for a 7-day moving average of daily traffic that has missing days?

level: middleimportance: must knowfreq 65%
basics
~10 s

rolling(7) averages the last seven rows, however many days they span; rolling('7D') averages every row in the seven days ending at each timestamp. With missing days, only the offset window really means a week.

open as a page

In pandas, why does a DataFrame saved with df.to_csv('out.csv') come back from pd.read_csv with an extra 'Unnamed: 0' column?

level: juniorimportance: must knowfreq 70%
basics
~10 s

DataFrame.to_csv writes the row index as the first column by default (index=True), under an empty header when the index has no name. read_csv names that empty header 'Unnamed: 0' and builds a fresh RangeIndex.

open as a page

In pandas 3.0, how do read_csv's dtype and parse_dates keep zip codes like 02134 intact and timestamps typed, and what dtypes result?

level: middleimportance: must knowfreq 75%
basics
~10 s

Pass dtype={'zip': 'str'} so the parser never turns zip codes into numbers, and parse_dates=['ts'] for timestamps. In pandas 3.0 the zip column gets the str dtype and ISO timestamps become datetime64[us].

open as a page

When pandas DataFrame.to_sql loads a nightly result into an existing database table, what do if_exists and index default to, and what goes wrong?

level: middleimportance: must knowfreq 60%
basics
~10 s

if_exists defaults to 'fail', so an existing table raises ValueError; index defaults to True, so the DataFrame index is written as an extra column. Use 'append', or in pandas 3.0 'delete_rows', with index=False.

open as a page

In pandas read_csv, how do na_values and keep_default_na decide which strings become NaN, and when must you turn the defaults off?

level: middleimportance: should knowfreq 50%
basics
~10 s

read_csv treats a built-in list of strings ('', 'NA', 'NULL', 'null', 'None', 'nan' and others) as missing, and na_values adds to that list. Set keep_default_na=False when real values such as 'NA' must survive.

open as a page

In pandas, how do you read every sheet of an Excel workbook, what does read_excel return, and how do you write several sheets back?

level: middleimportance: should knowfreq 50%
basics
~10 s

pd.read_excel(path, sheet_name=None) returns a dict of DataFrames keyed by sheet name; the default sheet_name=0 reads only the first sheet. To write several sheets, open one pd.ExcelWriter and call to_excel(writer, sheet_name=...) per frame.

open as a page

In pandas, why is a loop over df.iterrows() slow, and what does switching to df.itertuples() change?

level: juniorimportance: must knowfreq 70%
basics
~20 s

iterrows() builds a full pandas Series for every row, which is slow and coerces the row's values to one dtype. itertuples() yields lightweight namedtuples of the values, is much faster and keeps each value's type, but is still a Python loop.

open as a page

In pandas, why can df.info() report a DataFrame as far smaller than it really is, and how does memory_usage(deep=True) fix the measurement?

level: middleimportance: must knowfreq 55%
basics
~10 s

By default pandas counts only each column's array buffer, so an object column counts 8-byte pointers, not the Python strings behind them. memory_usage(deep=True) or info(memory_usage='deep') measures those objects too.

open as a page

When does pandas' DataFrame.eval() with the numexpr engine make a column expression faster, and when does it make it slower?

level: middleimportance: should knowfreq 28%
basics
~20 s

DataFrame.eval() helps when a compound arithmetic or boolean expression runs over large frames: numexpr evaluates it in one chunked, multi-core pass without full-size temporaries. On small frames, simple expressions or text, parsing overhead makes it slower than plain operators.

open as a page

A 50-million-row pandas events DataFrame takes 12 GB in memory; which dtype and read-time changes shrink it, and how do you confirm each one paid off?

level: seniorimportance: should knowfreq 50%
basics
~10 s

Rank columns with memory_usage(deep=True), then read only the needed columns with usecols and declared dtypes, downcast numbers with to_numeric(downcast=), and turn repeated labels into category; re-measure after each step and watch the load's peak.

open as a page

When should a team stop tuning a pandas pipeline and move the workload to a different engine, and what does that move cost?

level: principalimportance: should knowfreq 35%
basics
~20 s

Move when, after measured dtype, read, vectorisation and compiled-engine work, the job still exceeds one machine's memory or its time budget, especially when its core steps resist chunking. The move costs a rewrite, semantic differences, new skills and new operations.

open as a page

In pandas, why does df.to_numpy() on a feature frame with an Int64 column return an object array, and how do dtype= and na_value= fix it?

level: middleimportance: must knowfreq 50%
basics
~20 s

DataFrame.to_numpy() builds one array with one common dtype, and a nullable Int64 column mixed with other columns resolves to object, holding pd.NA. Passing dtype='float64' gives a numeric array with NaN; na_value= fills missing values with a number you choose.

open as a page

In pandas, what does DataFrame.plot draw by default, what does it return, and how do you pick another kind such as bar or hist?

level: juniorimportance: should knowfreq 50%
basics
~10 s

By default DataFrame.plot draws one line per numeric column against the index, through Matplotlib, and returns the Matplotlib Axes it drew on. kind='bar', 'hist', 'box' or 'scatter', or df.plot.bar() and friends, picks another chart.

open as a page

In pandas, what does DataFrame.to_dict() return by default, and with orient='records', 'list' or 'index'?

level: juniorimportance: should knowfreq 40%
basics
~10 s

By default to_dict() returns {column: {index label: value}}. orient='records' gives a list with one {column: value} dict per row, 'list' gives {column: [values]}, and 'index' gives {index label: {column: value}}.

open as a page

In pandas 3.0, how do Series.values, Series.array and Series.to_numpy() differ on a text, Int64 or category column, and which should library code use?

level: seniorimportance: should knowfreq 35%
basics
~20 s

Series.array returns the backing ExtensionArray without converting; to_numpy() always returns a NumPy ndarray, converting or copying when needed. .values is legacy: an ndarray for NumPy dtypes, the ExtensionArray otherwise, so in pandas 3.0 text columns give an ArrowStringArray.

open as a page

In pandas, why do df.plot.hist() and df.hist() draw the same price and quantity columns so differently, and what does each return?

level: middleimportance: nice to knowfreq 25%
basics
~20 s

df.plot.hist() overlays all numeric columns on one Axes with bin edges computed from every column pooled, so a narrow column collapses into a bin or two. df.hist() bins each column in its own subplot and returns an array of Axes.

open as a page