Pandas
Python's DataFrame library: labelled Series and frames, selection under Copy-on-Write, dtypes, groupby, merges, reshaping and time series. Interviewers watch for column thinking.
on this pageshowhide
explore
- Series, DataFrame and Index15 questions
- Building and Inspecting Frames5 questions
- Axis Labels and Alignment5 questions
- Copy-on-Write5 questions
- Selectors and Setters17 questions
- Label and Position Access6 questions
- Boolean Masks5 questions
- Assigning Values6 questions
- Choosing a Dtype19 questions
- Text Columns7 questions
- Categoricals6 questions
- Casting and Arrow Backends6 questions
- Missing Data10 questions
- NA-Aware Dtypes4 questions
- Filling and Dropping6 questions
- Deriving Columns11 questions
- Applying Functions5 questions
- Method Chaining6 questions
- Order and Uniqueness12 questions
- Sorting and Top N6 questions
- Duplicates and Counting6 questions
- GroupBy11 questions
- Grouping and Aggregating5 questions
- Group Transforms and Filters6 questions
- Merge, Join, and Concat6 questions
- Reshaping and Pivoting6 questions
- Temporal Data12 questions
- Datetime Parsing and Zones6 questions
- Resampling and Windows6 questions
- Readers and Writers12 questions
- CSV Import and Export6 questions
- Parquet, SQL and Excel6 questions
- Vectorized Speed and Memory6 questions
- Plotting and Interop6 questions
questions
143 · 13 sectionsIn pandas, how does a DataFrame built from a dict of lists differ from one built from a list of dicts like API records?
basics
~20 sA dict of lists is column-oriented: each key is a column and all lists must share one length. A list of dicts is row-oriented: each dict is a row, columns are the union of keys, and a missing key becomes NaN.
In pandas, what do shape, dtypes, head(), info() and describe() each tell you about a DataFrame you have just built?
basics
~20 sshape gives (rows, columns); dtypes gives each column's type; head() shows the first five rows; info() prints the index, each column's non-null count and dtype; describe() gives count, mean, spread and quartiles of numeric columns unless include='all'.
In pandas 3.0, if you take `email = customers['email']` and change a value in `email`, does the `customers` DataFrame change?
basics
~20 sNo. In pandas 3.0 Copy-on-Write makes every indexing result, including a column taken with [], behave as a copy, so editing it leaves the parent DataFrame untouched; to change the frame, write to the frame itself.
In pandas, what do DataFrame.set_index and reset_index do to a product table's row labels, and what does drop change in each?
basics
~20 sset_index moves one or more columns into the row labels and, by default, removes them from the columns; reset_index moves the labels back into columns and installs a fresh RangeIndex, or discards them when drop=True.
In pandas, why does adding two stores' monthly revenue Series with s1 + s2 return NaN for some months, and what does add(fill_value=0) change?
basics
~20 spandas pairs values by index label and returns the union of both indexes, so a month only one Series has gets NaN. add(fill_value=0) treats the missing side as 0, except where both sides are missing.
In pandas, how do you keep only the DataFrame rows that satisfy two conditions at once, and why does each comparison need its own parentheses?
basics
~20 sBuild one boolean Series per condition and combine them with & (and), | (or) and ~ (not), wrapping each comparison in parentheses: Python binds & tighter than > or ==, so an unparenthesised filter parses wrongly and raises.
In pandas, what do emp['salary'], emp[['salary']] and emp[0:2] return, and why does emp[0] raise KeyError?
basics
~20 sOn a DataFrame, [] with one name returns that column as a Series, a list returns a DataFrame, and an integer slice returns rows by position. emp[0] raises KeyError because a scalar key looks for a column named 0.
In pandas, how do df.loc and df.iloc differ on a frame indexed by non-sequential employee IDs, and what does each slice include?
basics
~20 sdf.loc selects by index label and its slices include both endpoints; df.iloc selects by 0-based position and its slices exclude the stop. On an ID index, loc[1007] is employee 1007 and iloc[1] is the second row.
In pandas, how do you set a column's value only on the rows that match a condition, such as flagging suspicious payments?
basics
~20 sUse one loc call with the boolean mask as the row selector and the column label as the column selector: df.loc[df['amount'] > 5000, 'is_fraud'] = True. Initialise a new column first so its dtype stays clean.
In pandas 3.0, what triggers 'ValueError: The truth value of a Series is ambiguous', and how do you fix each trigger?
basics
~20 sAnything that asks a whole Series for one bool raises it: and, or, not, if, a chained comparison like 10 < s < 50, or comparisons joined by & without parentheses. Use &, |, ~ element-wise, or reduce with .any(), .all(), .empty or .item().
In pandas, a scraped price column holds strings like '$1,200' and 'N/A' — why does astype(float) raise, and how do you convert it?
basics
~10 sastype(float) is all-or-nothing: one value that does not parse, such as '$1,200' or 'N/A', raises ValueError. Strip the symbols with .str.replace, call pd.to_numeric(errors='coerce') so leftovers become NaN, then count those NaN.
In pandas 3.0, what dtype does a new column of text get by default, and what can that column hold?
basics
~10 sIn pandas 3.0 text is inferred as the str dtype, a StringDtype whose missing value is NaN, stored in PyArrow when installed and in Python objects otherwise. It holds only strings and missing values.
In pandas, what happens when df.astype({'qty': int, 'price': 'float64'}) meets a column with gaps or an unparseable value, and what does errors= change?
basics
~10 sastype with a dict casts only the named columns. A gap makes int raise IntCastingNaNError; use 'Int64'. A bad string raises ValueError naming the column, unless errors='ignore', which silently leaves just that column uncast.
In pandas 3.0, when does astype('category') shrink a text column's memory, and when does it make the column larger?
basics
~20 sA category column stores each distinct value once plus one small integer code per row, so it shrinks low-cardinality text sharply. When nearly every value is unique, the categories duplicate the data and the codes add bytes, so the column grows.
In pandas, how does Series.str.extract turn product SKU codes like 'TSH-L-RED' into separate columns, and what decides the result's shape?
basics
~20 sSeries.str.extract applies a regex to each string and returns one column per capture group, labelled by named groups, from the first match only. Non-matching or missing rows get missing values; expand=False with one group returns a Series.
In pandas, how do you count the missing values in each DataFrame column, and how do dropna() and fillna(0) treat them differently?
basics
~10 sdf.isna().sum() counts the missing cells in each column and df.isna().mean() gives their share. dropna() removes every row holding any missing value, while fillna(0) keeps every row and writes 0 into each gap.
In pandas 3.0, which missing marker (NaN, NaT, None or pd.NA) does each dtype hold, and why test for them with isna rather than ==?
basics
~20 sNumPy float64 and the pandas 3.0 str dtype hold NaN, datetime64 and timedelta64 hold NaT, object holds what was stored (often None), and Int64, Float64, boolean and string hold pd.NA. pd.isna detects all four; == finds none.
A pandas Series of minute-level sensor temperatures has three-minute dropouts and one three-hour outage; what does ffill(limit=5) do to each?
basics
~20 sffill copies the last valid reading forward, so each three-minute dropout is filled completely. limit=5 caps each gap at five filled rows, so the outage gets five copies at its start and its remaining rows stay NaN.
Why does a pandas int64 column of clinic visit counts turn float64 once a value goes missing, and how does Int64 keep it integer?
basics
~20 sNumPy's int64 has no way to store a missing value, so pandas writes the gap as NaN and upcasts the whole column to float64. The nullable Int64 dtype keeps int64 values plus a separate mask and shows the gap as pd.NA.
In pandas, how do DataFrame.dropna's how, thresh, subset and axis arguments decide which rows or columns are removed?
basics
~20 sdropna removes rows by default when any cell is missing. how='all' drops only fully empty rows, thresh=n keeps rows with at least n non-missing values, subset limits which columns are checked, and axis='columns' drops columns instead.
In pandas, what does `Series.map` return when you pass it a dict, and what happens to values the dict has no key for?
basics
~20 sSeries.map with a dict returns a new Series on the same index, each value replaced by its dict lookup. Any value without a key becomes NaN, unless the dict defines missing, which then supplies the default.
A pandas loan table assigns risk tiers with `df.apply(tier, axis=1)`; how do you replace that apply with `np.where` or `np.select`?
basics
~20 sBuild each condition as a boolean Series over whole columns, then use np.where for two outcomes or np.select(conditions, choices, default=...) for several, where the first true condition wins. Handle NaN explicitly, since it fails every comparison.
In a pandas method chain, why should DataFrame.assign receive a lambda rather than a column taken from the original frame?
basics
~20 sDataFrame.assign calls a lambda with the intermediate frame at that step, while trips['fare'] reads the original frame: columns created earlier in the chain are missing (KeyError), changed values are stale, and relabelled rows silently misalign.
Why does calling rename(..., inplace=True) in the middle of a pandas method chain break the chain?
basics
~10 sDataFrame.rename with inplace=True modifies the frame and returns None, so the next method is called on None and raises AttributeError; in a chain, let every method return a new frame instead.
In pandas, when do you bin a credit-score column with `pd.cut` and when with `pd.qcut`, and what do their defaults do to edge values?
basics
~20 spd.cut uses edges you choose, such as policy thresholds; pd.qcut picks edges at quantiles so bins hold about equal counts. cut's intervals are right-closed, so the lowest edge is excluded unless include_lowest=True, and values outside the edges become NaN.
In pandas, how do DataFrame.duplicated and drop_duplicates differ, and which copy of a repeated row counts as the original by default?
basics
~20 sDataFrame.duplicated returns a boolean Series flagging repeated rows; drop_duplicates returns a new DataFrame without them. Both compare all columns unless subset is given, ignore the index, and by default keep='first' treats the first occurrence as the original.
In pandas, how do you sort a DataFrame by one column in descending order, and where do missing values end up?
basics
~10 sCall df.sort_values('revenue', ascending=False), which returns a new DataFrame; missing values go last in either direction unless na_position='first', and every row keeps its original index label.
In pandas, how do you keep only each contact's most recent sign-up when a CRM export repeats contacts across sign-ups?
basics
~10 sdrop_duplicates keeps a copy by row position, not by date, so sort first: crm.sort_values('signup', ascending=False).drop_duplicates('contact_id') keeps each contact's latest dated row; ignore_index=True renumbers the result.
In pandas, how do Series.value_counts(normalize=True) shares of churn reasons change when some reasons are missing and you pass dropna=False?
basics
~20 sBy default value_counts drops missing values from both the counts and the denominator, so the shares of known reasons sum to 1. dropna=False adds a NaN row and divides by every row, so each known share shrinks.
With pandas sort_values, how do you sort by several columns with a different direction for each, and what constraints apply?
basics
~10 sPass lists to both arguments: df.sort_values(by=['revenue', 'product'], ascending=[False, True]). The ascending list must match by's length or pandas raises ValueError, and na_position is a single setting for every key.
In pandas, what does `orders.groupby('customer_id')['amount'].sum()` return, and how do you get `customer_id` back as a column?
basics
~20 sA Series of totals, one per customer, whose index holds the sorted customer_id values and whose name is amount. Pass as_index=False to groupby, or call reset_index() on the result, to get a DataFrame with customer_id as a column.
In pandas, how do you compute several differently named summary columns per customer with `groupby().agg`, and why prefer named aggregation?
basics
~20 sPass keywords to agg, each naming an output column and pairing a source column with a function: agg(total=('amount', 'sum'), orders=('amount', 'size')). Lists and dicts of functions produce two-level column labels to flatten; named aggregation gives flat, chosen names.
In pandas, how do `groupby().size()`, `groupby().count()` and `groupby().nunique()` differ when the `amount` column has missing values?
basics
~20 ssize() counts every row in each group, missing values included, and returns one number per group. count() counts non-missing values per column, so a NaN amount lowers it. nunique() counts distinct non-missing values per column.
In pandas, why is a per-department z-score computed with groupby apply slow when there are many departments, and what do you replace it with?
basics
~20 sgroupby apply calls a Python function once per group and then concatenates the pieces and infers the result's shape, so its cost grows with the number of groups. (x - g.transform('mean')) / g.transform('std') computes the same z-score with grouped built-ins.
In pandas, when do you combine two DataFrames with merge, with DataFrame.join, or with pd.concat?
basics
~20 sUse merge to match rows on key columns (default how='inner'), DataFrame.join to match against the other frame's index (default how='left'), and pd.concat to stack frames along an axis while aligning the labels of the other axis (default join='outer').
In pandas, how do you find customers who never placed an order with merge, and what did pandas 3.0 add for it?
basics
~10 sLeft-merge customers to orders with indicator=True and keep rows whose _merge is 'left_only'. In pandas 3.0, merge(how='left_anti') returns those customers directly.
In pandas, how does merge's validate argument stop an orders-to-customers merge from silently multiplying rows?
basics
~10 svalidate declares the expected key relationship; with validate='many_to_one', merge checks that customer_id is unique in the customers frame and raises MergeError instead of returning extra rows when it is not.
In pandas, after pd.concat of twelve monthly DataFrames, why does df.loc[0] return twelve rows, and what do ignore_index and keys change?
basics
~20 spd.concat keeps each piece's index, so twelve frames that each start at 0 stack into twelve rows labelled 0. ignore_index=True renumbers 0..n-1; keys=[...] adds an outer index level naming each month. In pandas 3.0, passing both raises ValueError.
Should a data team make merge's validate argument and a row-count check a standing review rule on every pandas merge in its pipelines?
basics
~20 sUsually yes for pipeline code: require explicit how= and validate= on every merge, plus a row-count or match-rate assertion where the business has an expectation. Keep notebooks exempt, and accept the cost of failing runs and an extra uniqueness pass.
In pandas, how do you use `melt` to turn a wide survey export with one column per question back into one row per answer?
basics
~10 sCall df.melt(id_vars="respondent", value_vars=["q1", "q2", "q3"], var_name="question", value_name="score"). Each listed column becomes rows: the old header goes into the var_name column and its cell into the value_name column.
In pandas, why does `pivot` raise on a long station, date and reading weather table when `pivot_table` succeeds, and what does `pivot_table` add?
basics
~10 sDataFrame.pivot only reshapes, so two rows sharing an index and column pair raise ValueError: Index contains duplicate entries, cannot reshape. pivot_table aggregates those duplicates with aggfunc (default mean) and adds fill_value and margins.
In pandas, what does `pd.crosstab(obs["station"], obs["condition"], normalize="index")` return, and how do margins change it?
basics
~20 spd.crosstab counts rows per station and condition pair; normalize="index" divides each count by its row total so every station's shares sum to 1. With margins=True it also adds an All row holding the overall shares.
In pandas, what does `DataFrame.explode` do to the index, to empty lists and to the dtype when a list-valued sensors column is exploded?
basics
~20 sDataFrame.explode gives each list element its own row, copying the other columns and repeating the original index label. An empty list becomes one row with NaN, and the exploded column keeps object dtype for numbers.
In pandas, why can `pd.get_dummies(batch, columns=["plan"], drop_first=True)` on a scoring batch silently disagree with the columns built from the training frame?
basics
~20 sget_dummies builds one column per level present in the data it is given, and drop_first drops the first of those levels. A batch missing some plans gets different columns, so fix the levels with a CategoricalDtype before encoding.
In pandas, how do you turn a text date column into real datetimes and then pull out its year and weekday?
basics
~10 sConvert the column with pd.to_datetime and assign it back, then read its parts through the .dt accessor: .dt.year for the year and .dt.dayofweek (Monday is 0) or .dt.day_name() for the weekday.
In pandas, how do you roll minute-level web traffic up to hourly and daily totals with resample, and what does the result look like?
basics
~10 sCall resample with a rule on a frame that has a DatetimeIndex, then aggregate: traffic.resample('h').sum() and traffic.resample('D').sum(). The result has one row per hour or day, stamped with each bucket's start time.
With pd.to_datetime in pandas 3.0, how do format, errors='coerce' and unit='s' handle an event log of mixed date strings and epoch seconds?
basics
~10 spd.to_datetime applies one format to the whole column, guessed from the first non-missing value unless format= is given; errors='coerce' turns non-matching rows into NaT, and unit='s' reads integers as seconds since 1970.
In pandas, what is the difference between Series.dt.tz_localize and Series.dt.tz_convert, and when do you call each one?
basics
~10 stz_localize attaches a zone to naive datetimes and keeps their clock reading; tz_convert takes already-aware datetimes and shows the same instants in another zone. Localize once at ingest, convert whenever you need local display.
In pandas, how do `rolling(7)` and `rolling('7D')` differ for a 7-day moving average of daily traffic that has missing days?
basics
~10 srolling(7) averages the last seven rows, however many days they span; rolling('7D') averages every row in the seven days ending at each timestamp. With missing days, only the offset window really means a week.
In pandas, why does a DataFrame saved with df.to_csv('out.csv') come back from pd.read_csv with an extra 'Unnamed: 0' column?
basics
~10 sDataFrame.to_csv writes the row index as the first column by default (index=True), under an empty header when the index has no name. read_csv names that empty header 'Unnamed: 0' and builds a fresh RangeIndex.
In pandas 3.0, how do read_csv's dtype and parse_dates keep zip codes like 02134 intact and timestamps typed, and what dtypes result?
basics
~10 sPass dtype={'zip': 'str'} so the parser never turns zip codes into numbers, and parse_dates=['ts'] for timestamps. In pandas 3.0 the zip column gets the str dtype and ISO timestamps become datetime64[us].
When pandas DataFrame.to_sql loads a nightly result into an existing database table, what do if_exists and index default to, and what goes wrong?
basics
~10 sif_exists defaults to 'fail', so an existing table raises ValueError; index defaults to True, so the DataFrame index is written as an extra column. Use 'append', or in pandas 3.0 'delete_rows', with index=False.
In pandas read_csv, how do na_values and keep_default_na decide which strings become NaN, and when must you turn the defaults off?
basics
~10 sread_csv treats a built-in list of strings ('', 'NA', 'NULL', 'null', 'None', 'nan' and others) as missing, and na_values adds to that list. Set keep_default_na=False when real values such as 'NA' must survive.
In pandas, how do you read every sheet of an Excel workbook, what does read_excel return, and how do you write several sheets back?
basics
~10 spd.read_excel(path, sheet_name=None) returns a dict of DataFrames keyed by sheet name; the default sheet_name=0 reads only the first sheet. To write several sheets, open one pd.ExcelWriter and call to_excel(writer, sheet_name=...) per frame.
In pandas, why is a loop over df.iterrows() slow, and what does switching to df.itertuples() change?
basics
~20 siterrows() builds a full pandas Series for every row, which is slow and coerces the row's values to one dtype. itertuples() yields lightweight namedtuples of the values, is much faster and keeps each value's type, but is still a Python loop.
In pandas, why can df.info() report a DataFrame as far smaller than it really is, and how does memory_usage(deep=True) fix the measurement?
basics
~10 sBy default pandas counts only each column's array buffer, so an object column counts 8-byte pointers, not the Python strings behind them. memory_usage(deep=True) or info(memory_usage='deep') measures those objects too.
When does pandas' DataFrame.eval() with the numexpr engine make a column expression faster, and when does it make it slower?
basics
~20 sDataFrame.eval() helps when a compound arithmetic or boolean expression runs over large frames: numexpr evaluates it in one chunked, multi-core pass without full-size temporaries. On small frames, simple expressions or text, parsing overhead makes it slower than plain operators.
A 50-million-row pandas events DataFrame takes 12 GB in memory; which dtype and read-time changes shrink it, and how do you confirm each one paid off?
basics
~10 sRank columns with memory_usage(deep=True), then read only the needed columns with usecols and declared dtypes, downcast numbers with to_numeric(downcast=), and turn repeated labels into category; re-measure after each step and watch the load's peak.
When should a team stop tuning a pandas pipeline and move the workload to a different engine, and what does that move cost?
basics
~20 sMove when, after measured dtype, read, vectorisation and compiled-engine work, the job still exceeds one machine's memory or its time budget, especially when its core steps resist chunking. The move costs a rewrite, semantic differences, new skills and new operations.
In pandas, why does df.to_numpy() on a feature frame with an Int64 column return an object array, and how do dtype= and na_value= fix it?
basics
~20 sDataFrame.to_numpy() builds one array with one common dtype, and a nullable Int64 column mixed with other columns resolves to object, holding pd.NA. Passing dtype='float64' gives a numeric array with NaN; na_value= fills missing values with a number you choose.
In pandas, what does DataFrame.plot draw by default, what does it return, and how do you pick another kind such as bar or hist?
basics
~10 sBy default DataFrame.plot draws one line per numeric column against the index, through Matplotlib, and returns the Matplotlib Axes it drew on. kind='bar', 'hist', 'box' or 'scatter', or df.plot.bar() and friends, picks another chart.
In pandas, what does DataFrame.to_dict() return by default, and with orient='records', 'list' or 'index'?
basics
~10 sBy default to_dict() returns {column: {index label: value}}. orient='records' gives a list with one {column: value} dict per row, 'list' gives {column: [values]}, and 'index' gives {index label: {column: value}}.
In pandas 3.0, how do Series.values, Series.array and Series.to_numpy() differ on a text, Int64 or category column, and which should library code use?
basics
~20 sSeries.array returns the backing ExtensionArray without converting; to_numpy() always returns a NumPy ndarray, converting or copying when needed. .values is legacy: an ndarray for NumPy dtypes, the ExtensionArray otherwise, so in pandas 3.0 text columns give an ArrowStringArray.
In pandas, why do df.plot.hist() and df.hist() draw the same price and quantity columns so differently, and what does each return?
basics
~20 sdf.plot.hist() overlays all numeric columns on one Axes with bin edges computed from every column pooled, so a narrow column collapses into a bin or two. df.hist() bins each column in its own subplot and returns an array of Axes.