In pandas, how do Series.value_counts(normalize=True) shares of churn reasons change when some reasons are missing and you pass dropna=False?
answer
- what the denominator includes
- missing values dropped by default
- a NaN row appears
- count versus proportion name
- descending unless ascending=True
basics
~20 sBy default value_counts drops missing values from both the counts and the denominator, so the shares of known reasons sum to 1. dropna=False adds a NaN row and divides by every row, so each known share shrinks.
solid answer
~40 s`s.value_counts()` returns a Series indexed by the distinct values, named `count`, sorted from most to least frequent, with missing values excluded because `dropna=True` is the default. `normalize=True` divides each count by the total of the counts in the result, and the result is named `proportion`. So with `dropna=True` the denominator is only the non-missing rows: 3 of 6 known reasons is 0.5, even if 2 more rows have no reason. With `dropna=False` a `NaN` entry joins the result, the denominator becomes all 8 rows, and the same reason is 0.375. Neither is wrong; they answer "share of stated reasons" versus "share of all churned customers", and a report should say which one it shows.
code
python · 16 linesimport pandas as pd
reason = pd.Series(
["price", "support", None, "price", "moved", "support", None, "price"],
name="reason",
)
print(reason.value_counts(normalize=True).round(3).to_dict())
# {'price': 0.5, 'support': 0.333, 'moved': 0.167}
shares = reason.value_counts(normalize=True, dropna=False)
print(shares.name) # proportion
print(shares.round(3).tolist()) # [0.375, 0.25, 0.25, 0.125]
print(shares.sum()) # 1.0 - the NaN row is part of the total
print(reason.isna().mean()) # 0.25 - share with no stated reasongo deeper
Recall that value_counts drops missing values by default, sorts by count descending, and names the result count or proportion.
Explain that normalize divides by the sum of the counts kept, so dropna changes the denominator and every share.
Show that you state which denominator a reported share uses and report the missing share separately so totals reconcile.
Push for one team convention on whether shares include missing answers, documented on every dashboard that shows percentages.
## What value_counts returns `Series.value_counts(normalize=False, sort=True, ascending=False, bins=None, dropna=True)` counts how often each distinct value occurs. The result is itself a **Series**: - its **index** holds the distinct values, and the index is named after the original Series (for a column `reason`, the index is named `reason`); - its **values** are counts, and the Series is named `count`; - it is **sorted by count, highest first**; `ascending=True` flips that, and `sort=False` keeps the order in which values first appear; - **missing values are excluded** unless you pass `dropna=False`. For a whole DataFrame, `DataFrame.value_counts` counts combinations of values across columns; that variant has its own ordering rules and its own lesson. ## normalize=True and its denominator With `normalize=True` the counts become shares and the Series is named `proportion`. In the source the division is by the **sum of the counts that remain in the result** (`result / result.sum()`). That detail is the whole answer to this question: - with `dropna=True` (default), missing values were never counted, so the denominator is the number of **non-missing** rows and the shares of the stated reasons sum to 1; - with `dropna=False`, missing values are counted as one more category, the denominator is **every row**, and every stated reason's share gets smaller. | Call | Denominator | `price` share (3 of 6 stated, 8 rows) | Missing shown | |---|---|---|---| | `value_counts(normalize=True)` | 6 stated reasons | 0.5 | no | | `value_counts(normalize=True, dropna=False)` | 8 rows | 0.375 | yes, as `NaN` | The `bins=` option for numeric data, which groups values into intervals the way `cut` does, follows a different denominator rule; for categorical labels such as churn reasons it does not apply. ## Worked example ```python import pandas as pd reason = pd.Series( ["price", "support", None, "price", "moved", "support", None, "price"], name="reason", ) reason.value_counts() # price 3, support 2, moved 1 name: count reason.value_counts(normalize=True) # price 0.500, support 0.333, moved 0.167 name: proportion reason.value_counts(normalize=True, dropna=False) # price 0.375, support 0.250, NaN 0.250, moved 0.125 ``` Notice where the `NaN` row lands: it is sorted by its count like any other value, not pinned to the end. ## Choosing between the two Both numbers are legitimate; they answer different questions: 1. **"Among customers who gave a reason, what share cited price?"** Use the default. The missing rows are out of scope. 2. **"Among all churned customers, what share cited price?"** Use `dropna=False`. A large `NaN` share is itself a finding: the exit survey is not being filled in. 3. **Both, side by side.** Report the stated-reason shares and, separately, `reason.isna().mean()`, the share with no reason. What goes wrong in practice is mixing them: a dashboard computes shares with the default, someone adds up the counts and compares against the number of churned customers, and the two do not reconcile. ## Reading values out of the result Because the result is a Series indexed by the reasons, you read it by label: - `counts.loc['price']` returns the count for one reason and raises `KeyError` for a reason that never occurred, since absent values get no row at all; - `counts.get('refund', 0)` returns a default instead of raising, which suits code that asks about a fixed list of reasons; - `counts.index` holds the reasons in the result's order, so `counts.index[0]` is the most frequent one under the default sort; - `counts.head(3)` is the top three, and with `normalize=True` and the default `dropna` their shares are shares of the stated reasons only. A reason with zero occurrences is simply missing from the result; categorical columns are the exception, because their declared categories are reported even when unused. ## Related arguments worth knowing - `ascending=True` lists the rarest values first, useful for spotting typos such as `'pricing'` next to `'price'`. - `sort=False` keeps first-appearance order, which is handy when a later step needs the original order. - The result is an ordinary Series, so `.head(3)`, `.loc['price']` and `.to_frame()` work on it; `.to_frame()` gives a one-column DataFrame named `count` or `proportion`. - `reason.value_counts().size` equals `reason.nunique()`, because both exclude missing values by default. ## Pitfalls - Reading `normalize=True` output as a share of all rows when missing values exist. - Expecting the missing count to appear without `dropna=False`. - Forgetting that two spellings of one reason are counted separately; normalise text before counting. - Counting per group, such as churn reasons per plan, is grouping work and is covered with grouping and aggregation.
- How do you show counts and shares in one table?Compute both and join them: `pd.concat([s.value_counts(dropna=False), s.value_counts(normalize=True, dropna=False)], axis=1)`. The columns come out named `count` and `proportion`, aligned on the reason values, so they cannot drift apart.
- Why might value_counts list 'price' and 'Price ' as separate reasons?It counts exact values. Differences in case or trailing spaces make distinct values, so normalise the column first, for example with `str.strip()` and `str.lower()`, then count. `ascending=True` helps find such variants, since typos tend to be rare.
- normalize=True always divides by the total number of rows, missing ones included
- value_counts includes a NaN count by default
- The NaN row from dropna=False is always placed last
- The result of value_counts is a DataFrame with value and count columns
- Shares from normalize=True and dropna=False can be compared directly