Why are app-store star ratings a biased estimate of average user satisfaction?

## Volunteer samples A **volunteer** or **self-selected** sample is one where membership is decided by the unit, not by the researcher. Public product ratings, open web polls, banner-recruited surveys and call-in votes are all volunteer samples. They are the cheapest data in existence and, for estimating a population mean, close to worthless — because the decision to participate is made by the same person whose opinion you are trying to measure, using that opinion as an input. Formally, let `V = 1` mark a user who leaves a rating and `Y` be the rating they would give. The observed average estimates the conditional mean `E[Y | V = 1]`. This equals the population mean `E[Y]` only if participation is independent of the opinion. For reviews it plainly is not: annoyance and delight both raise the probability of posting, and mild contentment lowers it. ## The J-shaped distribution Public ratings across many platforms show a characteristic shape: a large mass at the top of the scale, a secondary bump at the bottom, and very little in between. Two forces produce it: 1. **Effort filtering by intensity.** Writing a review costs a few minutes. Users pay that cost when they feel strongly. A user whose experience was "fine" has no motivation to spend it. 2. **Purchase self-selection upstream.** People who chose to install an app already expected to like it, which lifts the top of the distribution relative to the population of all potential users. The practical consequence is that the *mean* of a J-shaped, self-selected distribution is unstable and hard to interpret; a handful of one-star posts moves it substantially, and the movement reflects who was angry enough to post rather than a shift in satisfaction. ## The solicitation mechanism is part of the frame The sharper, more interview-relevant issue is that ratings are usually *prompted*. If the in-app prompt fires after a completed purchase, a finished workout or a successful sync, then the sampling frame is not "users" but "users who just had a good moment". That is a designed selection on the outcome. Three consequences follow: - The average is inflated by construction, and the size of the inflation depends on the trigger, not on the product. - **Version-to-version comparisons break** if the prompt logic changed between releases. A rise from 4.3 to 4.6 after a release that also moved the prompt tells you about the prompt. - Users who churned before ever reaching the trigger cannot appear at all — the least satisfied segment has zero probability of being sampled, which is undercoverage in its purest form. ## What to do instead - **Draw the sample yourself.** Prompt a random subset of users, chosen by the analytics system rather than by their own enthusiasm, and prompt them at a time that is not conditioned on a success event. - **Measure and report the response rate**, not just the number of responses. A 3% response rate on a random draw is still a volunteer sample of a random invitation, and should be reported that way. - **Compare respondents to everyone on observables.** You usually know tenure, plan, platform and engagement level for every user, respondent or not. If respondents differ sharply on those, say so; if the differences run through variables you observe, they can be weighted back toward the user base. - **Triangulate with behaviour.** Retention, repeat usage and support-contact rates are measured on the whole population and do not require anyone to volunteer. They answer a different question than satisfaction, but they are not selection-filtered in the same way. - **Use ratings for what they are good for.** The *text* of reviews is a rich source of specific defects and feature requests. Treating reviews as a bug-report channel is sound; treating their arithmetic mean as a satisfaction metric is not. ## Interview framing The strongest answers do three things: state precisely which conditional mean the published number estimates, identify the *two* selection steps (who installs and reviews at all, and who the prompt reaches), and propose a concrete alternative design rather than just criticising the metric. A weak answer stops at "reviews are biased because unhappy people complain more" — half the story, and it misses the prompt mechanism entirely.