In an A/B test, what is the difference between a count, a rate, and a share metric?
answer
- sum, normalise, or compose
- what sits under the line
- denominator fixed by assignment or by behaviour
- shares within a partition sum to 100%
- a rate needs a per-what
basics
~20 sA count metric sums events, such as total orders. A rate metric divides a count by the units that were exposed, such as orders per visitor. A share metric divides one count by a related total, such as the percentage of orders placed on mobile.
solid answer
~50 sA count is a raw sum of events in an arm: total orders, total add-to-carts. It scales with how many units landed in that arm, so it is never directly comparable between arms unless the exposure is identical. A rate normalises that count by the exposed population: `orders / exposed visitors`. The denominator is fixed by the assignment, so the two arms are comparable by construction. A share is compositional: `mobile orders / all orders`. Its denominator is itself an outcome, and the shares across categories are forced to sum to 100%, so a share can move because a different category moved. The practical rule is that the denominator is part of the metric name. "Conversion rate = 12%" is not a metric until you say 12% of what — visitors, sessions, or eligible users are three different numbers with three different interpretations.
go deeper
Be ready to name the three shapes and give one concrete example of each, and to say out loud that a rate is meaningless until you state its denominator. Practise turning a vague brief like measure conversion into a full definition.
Explain why a rate normalised by the randomised unit is comparable across arms while a count and a share are not, and describe how a de-duplication rule changes what a count metric actually measures.
Show that you write metric definitions others can implement: numerator event, eligible denominator, window, and counting rule, all fixed before launch so two analysts produce the same number from the same logs.
Own the question of which shapes are allowed to drive decisions at all. Argue when a share belongs on a review deck as context rather than as a result, and what the cost is when teams publish shares without their underlying counts.
## The three shapes Almost every experiment metric is one of three shapes, and confusing them is one of the most common analysis errors. **Count metrics** are raw sums over an arm: total orders, total searches, total support tickets. They are the easiest to instrument and the least useful to compare. A count answers "how much happened here", and how much happened depends on how many units were sent here. If the split is 50/50 and both arms logged the same number of exposures, a count difference is interpretable; if the split is 90/10, if one arm had a logging outage, or if the ramp changed mid-experiment, the count difference is mostly an artefact of exposure. Counts are still worth reporting as a sanity check and as a business-size figure, but the decision should not hang on them. **Rate metrics** (also called per-unit or normalised metrics) divide a count by the number of units that were exposed: `orders / exposed visitors`, `searches / exposed user`, `revenue / exposed user`. The essential property is that the denominator is fixed by the randomisation: every assigned unit contributes exactly one to the denominator, whatever it then does. That makes the two arms comparable by construction and makes the metric an average over a well-defined population. Two sub-shapes live here. A **proportion** has a binary numerator per unit (did this visitor convert at all: 0 or 1) and is bounded in [0, 1]. A **mean** has an unbounded numerator per unit (how many orders, how much revenue) and can be skewed. Both share the same virtue: one row per randomised unit. **Share metrics** (composition or mix metrics) divide one count by a related total rather than by the exposed population: percentage of orders placed on mobile, percentage of sessions that used search, percentage of revenue from subscriptions. The denominator here is another behavioural outcome, not a fixed population. Two consequences follow. First, the metric moves when the numerator moves, when the denominator moves, or when both move — a rise in mobile share is fully consistent with mobile orders being flat and desktop orders falling. Second, shares within a partition are constrained to sum to 100%, so they cannot all improve; one category's gain is arithmetically another's loss. Shares are excellent descriptive and diagnostic metrics and poor decision metrics on their own. ## Why the denominator is part of the definition The word "rate" carries no information about the denominator. "Conversion rate" can mean conversions per visitor, per session, per eligible user, per search, or per product-page view, and these are genuinely different quantities that can move in different directions in the same experiment. A metric definition is only complete when it states four things: 1. **The numerator event** — which logged event counts, and under what conditions. 2. **The denominator population** — which units are in scope, and as of when. 3. **The observation window** — over what period both are accumulated. 4. **The counting rule** — whether repeats count, and at what granularity they are de-duplicated. The fourth point is where a lot of silent damage happens. If a user clicks "add to cart" three times because the button did not respond, a raw event count records three; a per-user-per-session de-duplicated count records one. A treatment that only changes button responsiveness will move the first metric a long way and the second not at all. Neither counting rule is wrong; what is wrong is not deciding, or deciding differently between arms or between reads. De-duplication is a definitional choice with a rationale — count occurrences when intensity is the thing you care about, count distinct units when adoption is the thing you care about — and it belongs written down next to the metric, not buried in a query. ## Choosing between them Start from the question. If the question is "does this change help a user do the thing", you want a rate whose denominator is the randomised unit, because randomisation guarantees the denominators are comparable. If the question is "how big is this in absolute business terms", report a count alongside, and expect to scale it by exposure. If the question is genuinely about composition — which channel, which device, which plan — a share is the right shape, but report the underlying counts next to it so a reader can see whether the share moved because the part grew or the whole shrank. A useful discipline: for every headline share you publish, publish the numerator and the denominator as per-unit rates too. It costs one extra column and removes an entire class of misreadings.
- Why is a raw count of orders a poor metric to compare arms when the traffic split is uneven?A count scales with the number of units assigned, so a 90/10 split produces roughly a nine-fold count difference with no behavioural change at all. Logging gaps, ramp changes and bot filtering shift exposure too. Normalising by exposed units removes that dependency, which is why the decision metric is a rate and the count is reported only as a business-size figure.
- When you count add-to-cart events, should repeats inside one session be de-duplicated?It depends on the question, and the choice must be written into the definition. De-duplicate to one per user per session when you are measuring adoption — did they do it at all. Keep raw occurrences when intensity matters, for example basket-building behaviour. The hazard of raw counts is that retries and double-taps inflate them, so a treatment that only changes button responsiveness moves the metric without changing real behaviour.
- Is revenue per exposed user a rate or a share?A rate — specifically a mean, because the denominator is the exposed population fixed by assignment and the numerator is unbounded per user. It is not a share, because nothing constrains it to a total. It differs from a proportion in that it is heavily skewed by a small number of large spenders, so its uncertainty behaves differently from a conversion proportion.
A count is the total distance a car travelled, a rate is its fuel economy per litre, and a share is the fraction of that distance driven on motorways. Only the middle one is comparable between two cars driven for different lengths of time.
saying these in an interview costs you the question
- Says conversion rate without naming the denominator
- Compares raw event counts between arms of different sizes
- Assumes a rising share means the numerator grew
- Treats de-duplication as a query detail, not part of the definition
- Calls any x-per-y number a rate, including shares