In a multi-step harmful-task suite where most attempts end half-finished, why does scoring each task as a binary completed / not-completed throw away most of what the run cost you, and what would you report instead?

**The data you already have.** In a suite of this shape the binary verdict is not a primitive — it is derived. The scorer walks the task's ordered criteria, decides each one against the trace, and ANDs the results. A pipeline that stores only the AND has thrown away the per-criterion booleans it computed a microsecond earlier. That is what makes binary reporting a *reporting* defect rather than an instrumentation one: the graded record costs nothing extra to keep, and keeping it is what makes a re-grade an afternoon of CPU instead of the whole sweep budget paid twice. **Why binary loses the signal.** The steps of a multi-step task are dependent by construction: you cannot reach step four without step three. Depth is therefore an ordinal measurement carrying real information, and a threshold at "all steps" discards every gradation below the top. When the completion rate is low — which it usually is, because most attempts in these suites end part-way — the binary column is near-constant. A near-constant column cannot separate two model versions, two guardrail configurations, or two system prompts, and a floor effect looks exactly like safety. **What a graded record looks like.** Per attempt: task id, the ordered required steps, a per-step verdict with the criterion that decided it, the stop reason, the token and turn counts, and a reference to the raw trace. Aggregation then becomes a reporting choice made in the open, rather than something baked irreversibly into the data at scoring time. **The traps in aggregating it.** | Trap | What it does to the number | |---|---| | Unequal chain lengths | A mean of per-task fractions weights a 3-step task like a 12-step one; pooling steps lets long tasks dominate. Both are defensible; the choice moves the headline, so state it. | | Easy prefixes | Many chains open with a benign lookup almost any agent performs. An agent that always does step one and never step two scores a non-zero mean that reads like partial danger and is nothing of the kind. | | Non-independence | Steps within an attempt are not independent trials, so intervals computed as if they were are far too tight. The attempt, not the step, is the sampling unit. | | Stop reason folded into depth | A limit-truncated attempt and a refusal both read as "stopped at 4". Keep them as separate series or you cannot tell a safe model from a small budget. | **Where the graded number itself misleads.** This is the reading to guard hardest against: **partial credit is not partial harm.** In most of these chains the damage, if any, is carried by one terminal step — the send, the transfer, the publish. An attempt that reached seven of eight steps caused nothing, and "87.5% completion" invites a reader to hear "87.5% of the harm". The fix is to mark, per task, which step is harm-bearing, and to report reaching *that* step as a separate series from mean depth. Mean depth is a sensitivity instrument for comparing configurations; the harm-bearing-step rate is the number that describes outcomes. Conflating them is how a suite produces a scary headline from a run in which nothing was accomplished — or, in the other direction, a comfortable mean that hides a small number of fully executed chains. **What it costs to do properly.** Nothing in extra model calls; the per-step verdicts fall out of the scoring pass. It costs storage for traces (a few kilobytes to a few hundred per attempt, so gigabytes at most for a large sweep — trivial against the run's dollar cost) and it costs schema discipline: once a report quotes mean depth, later runs must compute it identically or the trend is meaningless. **How to present it.** A histogram of attempts by depth reached, split by stop reason, plus two scalars: the count of fully completed chains and the count that reached the harm-bearing step. The completed count is what people will quote; the histogram is what tells you whether a change actually moved anything. **What to check.** Whether any task's step-one criterion is satisfiable by an entirely benign action; whether the depth distribution has a spike exactly at the configured step limit; whether re-running one configuration reproduces the histogram shape rather than just the headline; and whether the harm-bearing step is identified for every task, or whether the report is silently treating all steps as equally consequential.