How do you score tool selection and argument correctness in an agent trajectory?

## Why two metrics, not one A tool call has two independently failable parts: *which* tool, and *what you pass it*. Collapsing them into a single "tool-call accuracy" number destroys the information you need to act. Wrong tool means the agent's routing is broken — the tool descriptions overlap, or the task framing does not distinguish them. Right tool with wrong arguments means the schema is ambiguous, a required field lacks an enum, or the agent lacks the context to fill a parameter. The remedies are different, so the metrics must be separate. ## Tool selection as precision and recall Take the reference trajectory's tool calls as the ground-truth set and the agent's calls as the predicted set. Then: - **True positives** — calls whose tool name appears in both. - **False positives** — calls the agent made that the reference did not (extra, wrong, or hallucinated tools). - **False negatives** — calls the reference made that the agent did not (skipped work). Precision = TP / (TP + FP); recall = TP / (TP + FN); F1 combines them. Use *multiset* comparison, not plain sets: if the reference calls a search tool twice and the agent calls it five times, three of those are false positives, and a plain set intersection would hide that. The two error types have very different production meanings. Low recall is under-action: the agent answers from memory instead of retrieving, or gives up early. Low precision is over-action, which is the more dangerous one when tools have side effects — an agent that calls a refund tool it should not have called scores as one false positive and one very unhappy customer. For that reason many suites weight false positives on mutating tools far more heavily than on read-only ones, or track a separate hard-fail flag for any unauthorized side-effecting call. ## Substitution is a double error The most common real failure is substitution, not omission. A helpdesk agent that calls `reset_password` where the reference calls `unlock_account` has done something plausible and wrong: the user is unlocked as a side effect but their password now no longer works. In the metric this registers twice — a false positive for `reset_password`, a false negative for `unlock_account` — which is the correct accounting, because both the wrong action and the missing action are real defects. It also means substitution errors depress precision and recall symmetrically, which is a useful fingerprint when you scan aggregate scores: a suite where both numbers dropped together usually has a routing problem between two similar tools, while recall alone falling usually means the agent stopped early. ## Argument correctness Score arguments only over the calls whose tool matched — arguments to a wrong tool are meaningless. Options, roughly in order of strictness: - **Exact match** on the whole argument object. Simple, brittle, and the right choice when the parameters are IDs and enums. - **Field-level accuracy** — score each parameter, then average. This distinguishes "one optional field wrong" from "every field wrong" and points directly at the offending parameter when you aggregate per field. - **Typed comparison** — exact for IDs and enums, numeric tolerance for amounts, normalized comparison for dates and casing, and semantic or judge-based comparison for free-text fields like a search query or a message body, where many phrasings are equally right. Always separate *invalid* arguments from merely *wrong* ones. An argument that violates the schema is caught by the runtime and is a validation failure; an argument that is well-formed but names the wrong account is a correctness failure that only the reference comparison catches. Reporting them together makes the schema-tightening work invisible. ## Matching calls before you can score arguments An overlooked mechanical detail: to score arguments you must first *pair* each agent call with a reference call. When a tool appears once, pairing is trivial. When it appears several times with different arguments, naive positional zipping mis-pairs and reports argument errors that are really ordering differences. Pair by best argument similarity within the same tool name, or make the reference explicit about which occurrence is which. ## Reading the numbers Interpret these per tool, not just in aggregate. A single overlapping pair of tools can drag a suite-wide precision score down while every other tool is perfect, and only the per-tool breakdown makes that visible. And remember the metric's ceiling: precision and recall against a reference measure conformance to *one* expert path. They are the right instrument for conformance and the wrong one for tasks where several tool sets are equally valid — there you fall back to constraint checks ("any retrieval tool must be called before answering") or a judge over the trace.