Why is percentage of orders placed on mobile a weak metric to judge an experiment on?

## Where the ambiguity comes from Write the metric out: `mobile share = mobile orders / all orders`. Both parts are things the treatment can change. There are three distinct ways to make the share rise, and the metric does not distinguish between them: 1. Mobile orders increase while desktop orders hold — the outcome you probably wanted. 2. Mobile orders hold while desktop orders fall — total orders down, share up, business worse. 3. Both fall, with desktop falling faster — total orders down badly, share up. The same three patterns exist in mirror image for a falling share, including the case where a share falls because the *other* category grew, which is usually a success. A metric whose improvement is compatible with the business getting smaller cannot carry a decision by itself. ## The closure constraint Shares within a partition are compositional: they are non-negative and sum to one. That has a consequence people often miss when setting goals. If mobile, desktop and tablet shares of orders sum to 100%, then no set of targets can raise all three. Handing three teams a share target guarantees that at least one fails arithmetically, regardless of how much each grows in absolute terms. Absolute per-unit metrics have no such constraint: mobile orders per visitor and desktop orders per visitor can both rise. The constraint also distorts comparisons over time or across segments. A market where one channel dominates has less room for that channel's share to grow, so equal-effort improvements show up as unequal share movements. Comparing share deltas across segments therefore compares things that had different ceilings. ## Reading a flat share Stability is just as ambiguous as movement. If the mobile share is identical in both arms, the possibilities include: nothing changed anywhere; both categories rose by the same proportion; both fell by the same proportion. A flat share is not evidence of a null effect and must never be reported as one. The only way to tell those cases apart is to look at levels, which is the argument for always publishing the counts. ## When a share is the right metric Shares are not a defect in themselves. They are the correct shape when the composition genuinely is the decision: - Capacity and cost planning, where the mix determines what you must provision. - Portfolio or channel questions, where the total is fixed by a budget and the only question is allocation. - Diagnostic reads, where a shift in composition explains a movement in a metric that already has a fixed denominator. In each case the total is either fixed by design or reported alongside, and the ambiguity disappears. ## What to report instead For an experiment readout, replace the share with two rates whose denominators the randomisation controls: - mobile orders per exposed visitor - non-mobile orders per exposed visitor Both denominators are the assigned population, so both are clean contrasts, and any composition story is recoverable from the pair. Publish the share underneath as descriptive context if it aids comprehension. This is a one-line change to a report that removes an entire family of misreadings, and it also removes the temptation to set a target on a number that a team can move by making a different part of the business worse. ## The interview version of the answer A strong answer names the mechanism (a denominator that is itself an outcome), gives the concrete failure case (share up because the total fell), notes the closure constraint (shares cannot all rise), and lands on the practical remedy (report the parts as per-exposed-unit rates and keep the share as context). Weaker answers stop at "percentages can be misleading" without saying what specifically goes wrong or what to do instead.