In pandas 3.0, why can `wide.stack()` return more rows than the long Series you unstacked into `wide`, and how do you make the round trip exact?
answer
- unstack fills missing pairs
- holes become NaN, ints become float
- the new stack keeps every cell
- dropna and sort may not be passed
basics
~20 sunstack writes NaN into every station and date pair that had no row, and in pandas 3.0 stack keeps every cell, NaN included. Chain dropna() and cast back, or unstack with fill_value, to restore the original rows and dtype.
solid answer
~40 s`unstack` moves an index level into the columns, so every combination of the remaining index and the new columns gets a cell; pairs that had no row become NaN, and an `int64` Series becomes `float64`. In pandas 3.0, `DataFrame.stack` uses the implementation introduced in 2.1 by default: it never drops values and never sorts, so stacking a 2-by-2 frame with one hole returns 4 rows, the NaN included. Passing `dropna` or `sort` now raises `ValueError`, and `future_stack=False` brings back the old implementation only with a deprecation warning. To restore the original I chain `.dropna()` and `.astype("int64")`, or I unstack with `fill_value` when a real default exists, and I call `.sort_index()` if I need the sorted order the old stack sometimes gave.
code
python · 14 linesimport pandas as pd
wide = pd.DataFrame(
{"2026-03-01": [4.0, 7.0], "2026-03-02": [5.0, None]},
index=pd.Index(["ST01", "ST02"], name="station"),
)
print(len(wide.stack())) # 4: the NaN cell is kept
for kwargs in ({"dropna": True}, {"sort": True}):
try:
wide.stack(**kwargs)
except ValueError as err:
print(type(err).__name__, "-", str(err)[:40])go deeper
Recall that unstack moves an index level into the columns and stack moves a column level back into the index.
Explain why unstack creates NaN for missing pairs and turns int64 into float64, and how fill_value avoids it.
Know that pandas 3.0's stack keeps NaN and rejects dropna and sort, and migrate 2.x calls with explicit dropna, casts and sort_index.
Weigh relying on future_stack=False during a 3.0 upgrade against rewriting each stack call while the old path is deprecated.
## What `stack` and `unstack` move Both methods move labels between the **row index** and the **column labels** of a pandas object. - `unstack(level=-1)` takes a level of the row index (by default the innermost) and turns its values into column labels. A Series with a two-level index becomes a DataFrame. - `stack(level=-1)` does the reverse: it takes a level of the column labels and turns it into the innermost level of the row index. A frame with single-level columns becomes a Series. They are the low-level form of reshaping: `pivot` is `set_index` followed by `unstack`. ## Where the extra rows come from `unstack` builds a full grid: every row label times every new column label. A long Series of readings indexed by `(station, date)` has no entry for `ST02` on `2026-03-02`, so the grid gets a hole there. ```python import pandas as pd long = pd.DataFrame({ "station": ["ST01", "ST01", "ST02"], "date": ["2026-03-01", "2026-03-02", "2026-03-01"], "reading": [4, 5, 7], }).set_index(["station", "date"])["reading"] # 3 rows, int64 wide = long.unstack("date") # 2 x 2, one NaN, both columns float64 back = wide.stack() # pandas 3.0: 4 rows, NaN kept, float64 fixed = back.dropna().astype("int64") # 3 rows, int64 again ``` Two things happen in `unstack`: 1. The missing pair becomes `NaN`. 2. Because a NumPy `int64` column cannot hold `NaN`, the columns become `float64`. ## What changed in pandas 3.0 | | Old implementation (default in 2.x) | Pandas 3.0 default | |---|---|---| | NaN cells | Dropped by `dropna=True`, including NaN that was real data | Every cell kept, NaN included | | Introduced NaN on multi-level stacks | Created, then usually dropped, dtype still coerced to float | Not created, dtype not coerced | | Sorting | `sort=True`, applied inconsistently | Never sorts; input order kept | | `dropna=` / `sort=` | Accepted | `ValueError` if passed | | `future_stack=False` | The default | Still works, with a deprecation warning | So the 2.x idiom `wide.stack()` quietly removed the hole and returned 3 rows. In pandas 3.0 the same call returns 4, and code that compared row counts, or merged the result back on the assumption of one row per reading, breaks. The new behaviour is more honest: a stack no longer deletes readings that were genuinely missing in the input. ## Making the round trip exact - **Drop the holes:** `wide.stack().dropna()`. This also drops readings that were `NaN` in the original long data; if those matter, mark them first or keep the original long frame instead of re-deriving it. - **Cast back:** `.astype("int64")` after the `dropna`, because the holes already forced `float64`. - **Avoid the hole:** `long.unstack("date", fill_value=0)` keeps `int64`, and stacking it returns 4 rows with an honest 0, which is right only when 0 really means "no reading". pandas 3.0 deprecates a `fill_value` that the column's dtype cannot hold. - **Use a nullable dtype:** with `long.astype("Int64")`, the hole becomes `<NA>` and the columns stay integer. - **Restore an order:** chain `.sort_index()`; the new stack keeps the input row order and the column order, and never sorts for you. ## Reading a stacked result - When the columns have a single level, `stack` returns a **Series** whose index gains the column labels as its innermost level; with a `MultiIndex` on the columns and one level stacked, it returns a DataFrame. - The new index level takes the **name of the column axis**. After `long.unstack("date")` that name is `date`, so the round trip restores `(station, date)`; a frame whose columns have no name produces an unnamed level that `reset_index` calls `level_1`. - `back.reset_index(name="reading")` turns the stacked Series into an ordinary long DataFrame with `station`, `date` and `reading` columns. ## Errors on both sides 1. `unstack` raises `ValueError: Index contains duplicate entries, cannot reshape` when two rows share the same `(station, date)` label, the same error `pivot` gives. 2. The new `stack` raises `ValueError: Columns with duplicate values are not supported in stack` when two columns share a label. 3. `stack(dropna=True)` raises because the new implementation does not introduce rows of NA values; `stack(sort=False)` raises and tells you to use `sort_index` instead. ## Migrating 2.x code Search for `.stack(` calls. Any call that passes `dropna` or `sort` fails on pandas 3.0 and must be rewritten; any call that relied on the default dropping of NaN needs an explicit `.dropna()`. `future_stack=False` is a stopgap that still warns and goes away in a later major release, so treat it as temporary.
- Which level does `unstack()` move when called with no argument?The innermost index level, because `level` defaults to `-1`. On a `(station, date)` Series that makes dates the columns; `unstack("station")` or `unstack(0)` makes stations the columns instead.
- Why does `stack(future_stack=False)` still work on pandas 3.0, and should you use it?pandas 3.0 keeps the old implementation reachable for migration, but specifying `False` emits a deprecation warning and the old path is slated for removal. Use it only to bridge a migration; rewrite the code with an explicit `.dropna()` or `.sort_index()` instead.
- Why does `wide.stack().dropna()` risk losing data?It drops every NaN, not only the holes `unstack` created. A reading that was recorded as missing in the original long data is removed as well, so the round trip silently shrinks the data. Keep the original long frame, or flag real missing readings before reshaping.
- In pandas 3.0, stack still drops NaN rows by default
- Passing dropna=False to stack is the fix in pandas 3.0
- unstack keeps an int64 column int64 even when it creates holes
- The new stack sorts the resulting index like sort_index
- unstack aggregates repeated index pairs instead of raising
Assumes pandas 3.0, where DataFrame.stack defaults to future_stack=True; pandas 2.x defaulted to the old implementation, which dropped NaN and sometimes sorted.