You are choosing an off-the-shelf evaluation to open a red-team engagement against a chat assistant. What does a broad multi-dimension trust battery such as TrustLLM buy you compared with a focused single-purpose attack suite, and what does that breadth cost?

The two instruments answer different questions, and picking the wrong one wastes the expensive part of an engagement. **What each thing is.** A *broad trust battery* is a fixed corpus of prompts grouped into named **dimensions** — TrustLLM's are truthfulness, safety, fairness, robustness, privacy and machine ethics. Each dimension is not one test; it is a stack of sub-tasks assembled from existing datasets, and each sub-task carries its own **scorer**: sometimes exact match against a labelled answer, sometimes a refusal detector (a rule or small classifier deciding "did the model decline?"), sometimes another model asked to judge the response against a rubric. The dimension figure you see is an aggregate over those heterogeneous sub-tasks. A *focused attack suite* — HarmBench, AdvBench, JailbreakBench — fixes one question instead: given a list of target behaviours and one or more attack methods, what fraction of behaviours were elicited? That fraction is the **attack success rate (ASR)**, and its denominator is behaviours attempted, not prompts sent. **What the battery buys.** Orientation. One pass gives you a profile across several axes, computed the same way for every model you point it at, so it is usable for two jobs: deciding where to spend attack effort, and re-running later as a regression tripwire on content that cannot argue back because it never changes. **What a pass costs.** A full battery is tens of thousands of generations — order 10^4, spread across the sub-tasks. At a few hundred tokens in and a few hundred out per item, that is roughly ten to twenty million tokens per model per pass, and the model-judged sub-tasks add a second call for every item they cover, so budget close to double on those. On a mid-priced metered endpoint that lands in the tens of dollars per model; at modest concurrency it is single-digit hours of wall clock. The real first-run cost is engineer time — writing the adapter to your endpoint, wiring judge credentials, getting item-level output to persist — call it a day, and near zero afterwards because the job is schedulable. Compare that to an adaptive campaign, where a person iterates against refusals: the same token spend buys you one dimension's worth of ground and costs days of attention. That asymmetry is exactly why batteries get over-used. **Where the number misleads.** Four readings are wrong, and each is common. - *The aggregate hides its denominators.* "Safety: 0.83" is a mean over sub-tasks with very different item counts and difficulty. One saturated sub-task with many items can carry a dimension while the sub-task you actually care about sits at chance. - *Refusal is credited as correct.* On most safety-flavoured sub-tasks, declining is the scored-right answer. A model tuned to refuse aggressively climbs that column and sinks on the truthfulness or over-refusal side. Read one column alone and you will promote the least useful model in the set. - *Rank order is not a difference.* Batteries publish point estimates with no interval. Two models a couple of points apart on a thin dimension are not distinguishable by that run. - *Fixed public content ages.* Once a suite is public it is trained near, tuned against, and in some cases absorbed outright. The score rises while behaviour does not, and the battery quietly stops measuring the thing it was built for. Underneath all four: the prompts are static and single-shot. A battery item does not respond to the refusal it just got; an attacker does. So a good dimension score is evidence about fixed strings, never evidence of robustness. **What I would check before quoting a figure.** How many items sit behind the specific dimension I care about, and how they split across sub-tasks. Which scorer decided each sub-task, and whether it is a model — if it is, which model and which version. Whether refusals count as passes there. Whether the run persists per-item results at all; if it only emits aggregates, the number is unauditable. And the cheapest check of the four: run the same model twice unchanged and look at the spread, because that spread is the resolution limit on everything you will later claim. **The working split.** Battery to prioritise. Focused suite or an adaptive tool driven by a person to substantiate a claim about a specific failure mode. Your own product-scenario set to decide anything about shipping. Reversing that order — letting the cheapest instrument carry the heaviest claim — is the failure this choice exists to avoid.