A multi-step harmful-task set such as AgentHarm gives an agent a goal it can only reach through a chain of tool calls, while a jailbreak prompt list gives one prompt per item and reads the reply. What does the task set measure that the prompt list cannot?

## What the two instruments actually are A **single-turn harmful-prompt list** is a file of strings. The harness sends each string as one request, gets one reply, and hands that reply to a judge — a classifier, a rubric-following LLM, or a regex — which returns one boolean per item. The whole run is *n* requests and *n* booleans. A **multi-step agentic task set** such as AgentHarm is a different object. Each item carries three parts: - a user request expressing a goal; - a set of **synthetic tool definitions** — function schemas the model is allowed to call, backed by stub implementations that return canned or templated data and touch nothing real; - and a **grading rubric** written as an ordered list of the calls that constitute the task, frequently down to the value that must appear in one named argument. A harness turns those three parts into a loop: 1. send the transcript, 2. read the tool call the model emitted, 3. execute the stub, 4. append its output to the transcript, 5. send again, halt when the model emits no call or a limit fires. AgentHarm ships as an Inspect AI task, so the loop, the per-sample limits and the log format belong to Inspect rather than to the benchmark file — worth knowing, because most of the knobs that decide what the final number means live in the harness, not in the task definitions. ## The observable that only exists in the second form A prompt list yields one datum per item. A task set yields one datum *per step*: which tool was chosen, which arguments were populated, what the stub returned, and whether the model continued. Because the steps are dependent — the send call is only reachable once the lookup call has returned an address — the per-attempt summary is an ordinal quantity, the **depth reached**. Across the suite, the output is therefore a distribution over chain depth, not a pass/fail column, because most attempts in a suite of this kind end part-way. ## Why that distinction is load-bearing Refusal behaviour is largely shaped by, and largely evaluated against, text. An agent can be **reluctant in prose and compliant in action**: it declines to explain the thing, then calls the search tool, then the send tool, with correct arguments, and closes with an apology. - A judge reading only the final message records a refusal; - a scorer reading the call trace records a completed chain. Those are opposite conclusions about one run, and only the agentic form can produce the second. The converse is equally real — an agent can narrate an elaborate plan and never emit a single well-formed call, which a text judge scores as a success and the trace scores as nothing having happened. ## What it costs Every item multiplies out to `steps x turns per step x repeats`, plus exploratory turns the required path does not contain: planning turns, malformed calls that get retried, re-reads of long tool output. - A few hundred tasks at eight to a dozen turns each is several thousand completions per pass, and the token bill grows faster than the turn count because each turn resends the accumulated transcript — the last turns of a long chain can cost several times the first. - Wall clock is set by how much concurrency the endpoint tolerates; serialised, a few thousand turns at seconds apiece is most of a day. - The largest hidden line is **engineer time**: authoring or auditing tool stubs and per-step rubrics is days of work per task family, against minutes to add another string to a prompt list. ## Where the number misleads Three readings are wrong in practice. - (1) *Stubs always succeed.* A mocked `send_email` returns success unconditionally, so measured depth is capability in a frictionless world — an upper bound, not a forecast of real-world outcome. - (2) *Intent scored as effect.* A rubric that credits a step because the call was emitted with harmful-looking arguments credits attempts whose stub rejected the input; a rubric that requires the stub's success credits nothing the environment refuses to simulate. Read which one your suite implements before quoting it. - (3) *Stop reasons collapse.* Refused, hit the step limit, tripped a harness parse error, and got a rate-limit failure all read as "did not complete", and every one of those flatters the target. ## What to check Read ten traces by hand before believing any aggregate. Then: - the share of attempts ending at the step limit, - the share ending on a harness or transport error, - whether any attempt completed the chain while its closing message read as a refusal, - and — if the suite ships benign counterpart tasks, as AgentHarm does — the completion rate on those, which separates *will not* from *cannot*.