In pandas, what does Series.str.split with expand=True return for SKU codes with different numbers of parts, and how do n and rsplit help?
answer
- lists versus columns
- columns numbered from zero
- short rows padded on the right
- cap the number of cuts
- count cuts from the end
basics
~20 sWith expand=True, Series.str.split returns a DataFrame with one column per piece, numbered from 0, and pads shorter rows with missing values on the right. n caps the number of splits and rsplit counts them from the end.
solid answer
~40 sBy default `s.str.split("-")` returns a Series of lists; `expand=True` spreads the pieces into a DataFrame with columns `0, 1, 2, ...`, as many as the longest row needs. Rows with fewer pieces are padded with missing values on the right, so for `"MUG-BLK"` the colour lands in column 1, the column that holds the size for `"TSH-L-RED"`: columns are positions, not meanings. `n=1` makes at most one split, so at most two columns, and `rsplit("-", n=1)` cuts from the end so the last piece is always the colour. `n` and `expand` are keyword-only. A missing input gives a row of missing values. When parts are optional, `str.extract` with named groups is the safer tool, and `str.cat(others, sep="-")` joins pieces back element-wise.
code
python · 10 linesimport pandas as pd
skus = pd.Series(["TSH-L-RED", "HOD-XL-BLU", "MUG-BLK", None])
print(skus.str.split("-").tolist()[:2]) # lists by default
wide = skus.str.split("-", expand=True)
print(wide)
tail = skus.str.rsplit("-", n=1, expand=True)
print(tail)
print(wide[0].str.cat(tail[1], sep="-", na_rep="?").tolist())go deeper
Recall that expand=True turns the split pieces into numbered columns and that shorter rows get missing values.
Explain how the column count is set, how n and rsplit control the cuts, and why a multi-character separator is treated as a regex.
Point out the positional trap on ragged codes and show how rsplit, a fixed column set or extract keeps meanings in the right columns.
Discuss when a team should require a fixed code structure upstream rather than accept ragged identifiers that every consumer must parse defensively.
## Lists by default, columns with expand `Series.str.split(pat=None, *, n=-1, expand=False, regex=None)` cuts each string at a separator. Its default output is a **Series of Python lists**, one list per row, which keeps the original shape. Passing **`expand=True`** turns the pieces into a **DataFrame**: - columns are labelled `0, 1, 2, ...` by position; - the number of columns is set by the **longest** row in that call; - the index is the source index, so the result lines up with the original frame. `n` and `expand` are **keyword-only**: `s.str.split("-", 1)` raises `TypeError`, so write `n=1`. ## Ragged rows and padding When rows split into different numbers of pieces, shorter rows are **padded with missing values on the right**: | SKU | column 0 | column 1 | column 2 | |---|---|---|---| | `TSH-L-RED` | `TSH` | `L` | `RED` | | `HOD-XL-BLU` | `HOD` | `XL` | `BLU` | | `MUG-BLK` | `MUG` | `BLK` | missing | | missing | missing | missing | missing | The trap is semantic, not technical: column 1 holds a size for two rows and a colour for the third. Renaming the columns to `product, size, colour` would silently store `BLK` as a size. Split by position only when every row has the same structure. ## Limiting splits: n and rsplit - **`n`** caps the number of splits; `n=-1`, `0` and `None` all mean "split everywhere". `n=1` makes at most one cut, giving at most two pieces: `"TSH-L-RED"` becomes `TSH` and `L-RED`. - **`str.rsplit`** works from the end of the string. `rsplit("-", n=1)` gives `TSH-L` and `RED`, so the **last** piece (here the colour) is always in column 1, whatever came before it. - Combining `split(n=1)` for the head and `rsplit(n=1)` for the tail pins both ends of a code without depending on what sits in the middle. ## Missing values and empty pieces - A **missing** input value produces a row that is missing in every column; it is not split into anything. - Two separators in a row, as in `"TSH--RED"`, produce an **empty string** piece between them, which is text, not a missing value; filter or replace empty pieces explicitly if they mean "absent". - With no separator present, the whole value lands in column 0 and the other columns are missing for that row. ## How the separator is read `regex=None`, the default, decides per pattern: 1. A one-character `pat` such as `"-"` or `"."` is treated as a **literal**. 2. A longer `pat` is treated as a **regular expression**, so `"||"` or `"-+"` follows regex rules. 3. `regex=False` forces a literal match; `regex=True` forces a regex. 4. With no `pat`, the string is split on whitespace. ## Assigning the pieces back to the frame A common line is `catalog[["product", "size", "colour"]] = catalog["sku"].str.split("-", expand=True)`. It works only when the split produced exactly three columns. If no SKU in the batch has three parts, the right-hand side has two columns and pandas raises `ValueError: Columns must be same length as key`; if some code had four parts, it fails the same way. Either fix the width first with `.reindex(columns=range(3))`, or let `str.extract`, whose column count is fixed by its groups, do the parsing. ## Taking one piece without expanding On the list output, `s.str.split("-").str.get(0)` or `s.str.split("-").str[0]` picks one piece per row and returns missing where the list is too short. This avoids building a wide frame when only one part is needed. ## Putting pieces back together with cat `Series.str.cat(others=None, sep=None, na_rep=None, join="left")` is the inverse: - with `others` (a Series, a frame or a list-like), it joins **element-wise**, aligning Series on the index (`join="left"` by default); a row with a missing piece gives a missing result unless `na_rep` supplies a stand-in; - with no `others`, it concatenates the whole Series into **one Python string**, skipping missing values unless `na_rep` is set; - `sep` defaults to the empty string. ## split versus extract Use `split(expand=True)` when the separator is reliable and every row has the same number of parts. When parts are optional or ambiguous, `str.extract` with named groups places each part in the column its meaning belongs to and leaves a missing value where a part is absent.
- Why can split(expand=True) give a different number of columns on another batch?The column count comes from the longest row in that call. A batch in which no SKU has three parts yields only columns 0 and 1, so code selecting column 2 raises `KeyError`. Pin the shape with `.reindex(columns=range(3))`, or use `str.extract`, whose columns are fixed by the pattern's groups.
- Is the separator in str.split a regular expression?It depends on `regex`, which defaults to `None`: a single-character pattern is literal, a longer one is compiled as a regular expression. Pass `regex=False` to split on a multi-character literal such as `"||"` without escaping, or `regex=True` to force regex treatment of a one-character pattern.
- str.split returns a DataFrame even without expand=True.
- Rows with fewer pieces make split(expand=True) raise an error.
- n=1 keeps only the first piece of each string.
- s.str.split('-', 1) is the same as passing n=1.
- Padding goes on the left so that the last pieces line up.