Why is queries-per-user a dangerous OEC for a search product?

## The requirement being violated An Overall Evaluation Criterion is not just a metric; it is a metric *plus a direction* plus a decision rule. If you cannot write down before launch which way the number has to move for the change to count as an improvement, you do not have a criterion — you have a number that will be interpreted after the fact to suit whatever happened. Queries per user is the classic metric that fails at the direction step. ## The two stories **Engagement story.** The change made search more useful. Users trust it with more of their questions, so they issue more queries per user. Up is good. **Failure story.** The change made results worse. A user who would have found the answer with one query now searches, scans, fails, rephrases, and searches again. Each failed attempt is another query. Up is bad. Both stories move the same metric in the same direction, and the metric contains no information that separates them. This is not a subtle statistical point — the estimate can be perfectly precise and the interpretation is still undetermined. The reverse ambiguity exists too. A *fall* in queries per user can mean users abandoned the product, or that each query now works so well that fewer are needed. Neither direction is safe to declare in advance, which is precisely the problem. ## What makes a metric directionally ambiguous The general pattern: the metric counts an *activity* that users perform as a means to an end, rather than the *end* itself. Activity counts go up both when the means is more attractive and when the means is less effective and has to be repeated. Any metric of the form *number of attempts* inherits this. Recognising the pattern matters more than memorising the example. When a candidate proposes a criterion, a good question to ask is: can I tell a plausible story where this number rises and users are worse off? If yes, either the metric is not the criterion, or the criterion needs to be redefined so the bad story can no longer produce a rise. ## Fixing it Three standard repairs, in increasing order of how much they cost you: **Move up a level, from action to task.** Count queries that were *answered* rather than queries issued — for example, a query where the user acted on a result and did not immediately reformulate. Now a reformulation loop, the signature of the failure story, no longer inflates the number; it deflates it. **Redefine the unit as the task, not the action.** Make the criterion successful tasks per user, where a task is a coherent run of activity toward one need. The failure story now collapses several queries into one task with no success, so the two stories separate cleanly and the direction is unambiguous: more successful tasks is better. **Move to an outcome only a success can produce.** Use something downstream that a frustrated user cannot generate by trying harder. This is the least ambiguous and usually the least sensitive, since such events are rarer. All three share a principle: make the metric count *ends*, not *attempts*. That is what restores a declarable direction. ## Where the ambiguous metric still belongs Queries per user is not useless. As a driver or diagnostic metric it is genuinely informative — read alongside a criterion that already tells you whether the change helped, a rise in queries means something quite different depending on the criterion's verdict, and the pair together locate the mechanism. What it cannot do is carry the decision by itself. This distinction is the point of the question. Interviewers use it to check that a candidate understands that the metric hierarchy is about *roles*, and that a metric can be excellent in the explaining role and disqualified from the deciding role. ## The wrong answers - *Just look at whether it went up and use judgment.* This is post-hoc interpretation, exactly what declaring a criterion is meant to prevent. Whichever story fits the team's hopes will be the one told. - *Pair it with a second metric and read them together.* Better, but if there is no rule for what to do when they conflict, the decision is still unstructured. If two metrics jointly define success, they need to be combined into one criterion with an explicit rule. - *It is fine because in practice more search means more engagement.* This asserts one of the two stories as a fact. If it were reliably true, the metric would have a direction — and the whole reason the question is asked is that it is not. ## Compact summary A criterion must have a direction you can state before you see the data. Metrics that count attempts rather than accomplishments usually cannot, because effort rises both when the product is more attractive and when it is less effective. Redefine toward success-shaped counts, keep the ambiguous metric as a diagnostic, and never let a number with two opposite readings decide a launch.