skip to content

In pandas, how do Series.value_counts(normalize=True) shares of churn reasons change when some reasons are missing and you pass dropna=False?

level: middleimportance: must knowfreq 64%

answer

  1. what the denominator includes
  2. missing values dropped by default
  3. a NaN row appears
  4. count versus proportion name
  5. descending unless ascending=True

basics

~20 s

By default value_counts drops missing values from both the counts and the denominator, so the shares of known reasons sum to 1. dropna=False adds a NaN row and divides by every row, so each known share shrinks.

solid answer

~40 s

`s.value_counts()` returns a Series indexed by the distinct values, named `count`, sorted from most to least frequent, with missing values excluded because `dropna=True` is the default. `normalize=True` divides each count by the total of the counts in the result, and the result is named `proportion`. So with `dropna=True` the denominator is only the non-missing rows: 3 of 6 known reasons is 0.5, even if 2 more rows have no reason. With `dropna=False` a `NaN` entry joins the result, the denominator becomes all 8 rows, and the same reason is 0.375. Neither is wrong; they answer "share of stated reasons" versus "share of all churned customers", and a report should say which one it shows.

code

python · 16 lines
python
import pandas as pd

reason = pd.Series(
    ["price", "support", None, "price", "moved", "support", None, "price"],
    name="reason",
)

print(reason.value_counts(normalize=True).round(3).to_dict())
# {'price': 0.5, 'support': 0.333, 'moved': 0.167}

shares = reason.value_counts(normalize=True, dropna=False)
print(shares.name)                 # proportion
print(shares.round(3).tolist())    # [0.375, 0.25, 0.25, 0.125]
print(shares.sum())                # 1.0 - the NaN row is part of the total

print(reason.isna().mean())        # 0.25 - share with no stated reason

go deeper

for a junior

Recall that value_counts drops missing values by default, sorts by count descending, and names the result count or proportion.

for a middle

Explain that normalize divides by the sum of the counts kept, so dropna changes the denominator and every share.

for a senior

Show that you state which denominator a reported share uses and report the missing share separately so totals reconcile.

for a principal

Push for one team convention on whether shares include missing answers, documented on every dashboard that shows percentages.

## What value_counts returns `Series.value_counts(normalize=False, sort=True, ascending=False, bins=None, dropna=True)` counts how often each distinct value occurs. The result is itself a **Series**: - its **index** holds the distinct values, and the index is named after the original Series (for a column `reason`, the index is named `reason`); - its **values** are counts, and the Series is named `count`; - it is **sorted by count, highest first**; `ascending=True` flips that, and `sort=False` keeps the order in which values first appear; - **missing values are excluded** unless you pass `dropna=False`. For a whole DataFrame, `DataFrame.value_counts` counts combinations of values across columns; that variant has its own ordering rules and its own lesson. ## normalize=True and its denominator With `normalize=True` the counts become shares and the Series is named `proportion`. In the source the division is by the **sum of the counts that remain in the result** (`result / result.sum()`). That detail is the whole answer to this question: - with `dropna=True` (default), missing values were never counted, so the denominator is the number of **non-missing** rows and the shares of the stated reasons sum to 1; - with `dropna=False`, missing values are counted as one more category, the denominator is **every row**, and every stated reason's share gets smaller. | Call | Denominator | `price` share (3 of 6 stated, 8 rows) | Missing shown | |---|---|---|---| | `value_counts(normalize=True)` | 6 stated reasons | 0.5 | no | | `value_counts(normalize=True, dropna=False)` | 8 rows | 0.375 | yes, as `NaN` | The `bins=` option for numeric data, which groups values into intervals the way `cut` does, follows a different denominator rule; for categorical labels such as churn reasons it does not apply. ## Worked example ```python import pandas as pd reason = pd.Series( ["price", "support", None, "price", "moved", "support", None, "price"], name="reason", ) reason.value_counts() # price 3, support 2, moved 1 name: count reason.value_counts(normalize=True) # price 0.500, support 0.333, moved 0.167 name: proportion reason.value_counts(normalize=True, dropna=False) # price 0.375, support 0.250, NaN 0.250, moved 0.125 ``` Notice where the `NaN` row lands: it is sorted by its count like any other value, not pinned to the end. ## Choosing between the two Both numbers are legitimate; they answer different questions: 1. **"Among customers who gave a reason, what share cited price?"** Use the default. The missing rows are out of scope. 2. **"Among all churned customers, what share cited price?"** Use `dropna=False`. A large `NaN` share is itself a finding: the exit survey is not being filled in. 3. **Both, side by side.** Report the stated-reason shares and, separately, `reason.isna().mean()`, the share with no reason. What goes wrong in practice is mixing them: a dashboard computes shares with the default, someone adds up the counts and compares against the number of churned customers, and the two do not reconcile. ## Reading values out of the result Because the result is a Series indexed by the reasons, you read it by label: - `counts.loc['price']` returns the count for one reason and raises `KeyError` for a reason that never occurred, since absent values get no row at all; - `counts.get('refund', 0)` returns a default instead of raising, which suits code that asks about a fixed list of reasons; - `counts.index` holds the reasons in the result's order, so `counts.index[0]` is the most frequent one under the default sort; - `counts.head(3)` is the top three, and with `normalize=True` and the default `dropna` their shares are shares of the stated reasons only. A reason with zero occurrences is simply missing from the result; categorical columns are the exception, because their declared categories are reported even when unused. ## Related arguments worth knowing - `ascending=True` lists the rarest values first, useful for spotting typos such as `'pricing'` next to `'price'`. - `sort=False` keeps first-appearance order, which is handy when a later step needs the original order. - The result is an ordinary Series, so `.head(3)`, `.loc['price']` and `.to_frame()` work on it; `.to_frame()` gives a one-column DataFrame named `count` or `proportion`. - `reason.value_counts().size` equals `reason.nunique()`, because both exclude missing values by default. ## Pitfalls - Reading `normalize=True` output as a share of all rows when missing values exist. - Expecting the missing count to appear without `dropna=False`. - Forgetting that two spellings of one reason are counted separately; normalise text before counting. - Counting per group, such as churn reasons per plan, is grouping work and is covered with grouping and aggregation.

  • How do you show counts and shares in one table?
    Compute both and join them: `pd.concat([s.value_counts(dropna=False), s.value_counts(normalize=True, dropna=False)], axis=1)`. The columns come out named `count` and `proportion`, aligned on the reason values, so they cannot drift apart.
  • Why might value_counts list 'price' and 'Price ' as separate reasons?
    It counts exact values. Differences in case or trailing spaces make distinct values, so normalise the column first, for example with `str.strip()` and `str.lower()`, then count. `ascending=True` helps find such variants, since typos tend to be rare.

  • normalize=True always divides by the total number of rows, missing ones included
  • value_counts includes a NaN count by default
  • The NaN row from dropna=False is always placed last
  • The result of value_counts is a DataFrame with value and count columns
  • Shares from normalize=True and dropna=False can be compared directly