Why report sliced eval scores by segment and failure mode instead of one aggregate?

## The failure the aggregate hides A single number over an eval set answers one question — did overall quality move — and silently refuses several others: for whom, in what way, and at what cost. Because the aggregate is a weighted mean, a cohort's ability to influence it is proportional to its share of the set. A cohort that is four percent of items can collapse entirely and move the headline by well under a point, comfortably inside run-to-run noise. If that cohort is the one where errors cause harm, the aggregate is not merely uninformative, it is actively misleading: it reports a win on the change that broke your riskiest path. The distortion runs the other way too. A change that improves the head of the distribution and degrades the tail looks strictly good; a change that fixes a small, hard cohort at a tiny cost to the easy majority looks strictly bad. Optimizing against an aggregate therefore drives a system toward the average request and away from the edges — which is where trust is actually won or lost. ## Two slicing axes **Segment slices (who or what the input is)** partition the set by properties of the request that exist before the system runs: payer plan type, drug class, customer tier, document length, language, channel. These map to accountability — a broken segment is a named group of users you can go and tell. **Failure-mode slices (what went wrong)** partition the *errors*, not the inputs, using a taxonomy built from real incidents: hallucinated rule, missing required criterion, wrong policy applied, over-refusal of a valid request, malformed output. These map to remediation — each mode has a different fix, and a mode-level view tells you whether a change traded one failure for another rather than removing failures. The two axes answer different questions and you want both. Segment slices are computed on every item; failure-mode slices are computed on the failures, so their denominators differ and the report must say so. ## Designing the set so slicing is possible Slicing is a property of the dataset, not of the dashboard. Every item needs the slice labels attached at curation time — segment tags from the trace metadata, and a failure-mode tag assigned when the item is reviewed. Retrofitting tags onto an existing set is expensive and usually partial, which is why teams end up with only an aggregate. More importantly, slicing is what sets the minimum size of each cell. A slice you intend to gate on must have enough items that its score is stable across runs. A slice of eight items moves 12.5 points per changed answer; nobody can act on a metric with that granularity. This is the reason a stratified quota per cell is worth its labeling cost: it buys per-slice resolution the aggregate does not need but the decisions do. ## Reading a sliced report without fooling yourself Slicing multiplies the number of comparisons, and comparisons are where false alarms come from. With thirty slices, a handful will look meaningfully worse in any given run purely by chance, and the smallest slices are the ones most likely to do it. Three practical controls: - **Minimum slice size.** Below the threshold, display the slice as informational and never gate on it. - **Confirm before acting.** Re-run the suspicious slice — ideally with more items, or repeated runs of the same items — before declaring a regression. A slice that reproduces is a signal; one that does not was noise. - **Pair the comparison.** Run both versions over the same items and look at which items *flipped*, not just the two rates. A regression concentrated in a slice usually shows up as a handful of specific items that changed from pass to fail, and those items are directly readable. ## Gating on the worst slice Once slices exist, the natural policy is that a change must not regress any protected slice beyond a threshold, even if the mean improves. That is much stricter than a mean-based gate and it is the right default for cohorts with asymmetric cost — safety-relevant categories, regulated segments, your largest customers. For everything else, the mean plus a watch-list is enough. Deciding which slices are protected is a product judgment, not a statistical one: it encodes where you are unwilling to trade quality for average gains. ## When a slice appears Every incident that reaches production is evidence the taxonomy was incomplete. The healthy loop is: incident, new failure mode, new slice, items sampled or written for it, minimum size met, then it joins the gated report. Over time the slice list is a written record of everything the system has been caught doing wrong.