Testing a Tool Server

**What a "tool server" actually is here.** An agent does not own its tools; it is handed a list of tool definitions on every request. Each definition carries three things the model reads as plain text: a name, a natural-language description, and a JSON Schema describing the parameters. When the model emits a tool call, the harness routes it by name to whatever answers — an MCP server, a REST service sitting behind the framework's function-calling loop, or a local process the harness registered at startup. That endpoint is the tool server. Everything the model learns about a tool before calling it (name, description, schema) and everything it learns after (the result body, the error string on a rejected call) crosses that one boundary. Owning the boundary is owning the experiment. **What owning it buys, knob by knob.** | Knob you control | What it is | |---|---| | Description text | The free-text string the model reads before deciding to call | | Parameter schema | Property names, types, `enum` members, the `required` list, `additionalProperties`, length caps | | Result body | The exact bytes returned to the model, including size, encoding and escaping | | Error surface | What the model sees on a rejected call — a bare code, or text that echoes the input | | Neighbourhood | How many tools are listed, in what order, and which sensitive one sits beside the benign one | The sixth thing, and the one people undervalue, is **the observation itself**. A sensitive handler on a server you own can write a server-side log row the moment it is invoked, with the arguments it received. That turns "did the agent do the thing" from a judge's reading of a transcript into a row in a table. Removing the grader from the loop is usually worth more than the mutability, because graded transcripts are where the false positives and false negatives of agent red-teaming live. **What it costs.** Building a stub faithful enough to be worth anything is typically one to three engineer-days per service, and it is a second implementation of somebody else's contract, so it rots on their release schedule, not yours. Hosting is noise. The recurring cost is model calls, and it is the repeats that spend it: one agent attempt is a multi-turn loop, commonly five to fifteen model calls of a few thousand tokens each, so a modest design of four surface variants at forty attempts per variant is 160 agent runs and on the order of one to two million tokens — dollars rather than cents on a hosted mid-tier model, and hours of wall clock unless attempts run in parallel. Budget the engineer-days and the token spend separately; teams routinely fund the second and forget the first, then hand-patch the stub until a run passes. **Where the number misleads.** The output of this setup is a success rate, and its denominator is your fixture, not the vendor. Three specific misreadings: 1. *Permissiveness inflation.* Your stub accepts arguments and returns content the real service would reject, truncate or escape. The rate is then measuring your own laxity. 2. *Instrument swap.* On the fixture you score with a server-side invocation log; against the real service you usually score by reading the agent's account of what it did. That is a different measuring instrument, so a fixture rate and a live rate are not comparable in either direction — a lower live number can mean a stricter server or simply a blinder detector. 3. *Neighbourhood shrinkage.* Fixtures typically expose three or four tools; production agents carry twenty or more, and the model's behaviour with a crowded, competing tool list is not the behaviour you measured. **What you would check before believing a result.** Run the unchanged baseline fixture twice and report both numbers — that spread is your noise floor, and any effect smaller than it is not an effect. Diff your served tool definitions against the vendor's published contract field by field, and keep a capture of real responses to compare lengths, encoding and escaping against what your stub returns. Confirm the server-side hit log fires for a deliberately triggered call, so you know the instrument works. Stamp a fixture identifier into every run record. Then state the claim at the strength the evidence carries: "this agent acts on content arriving in a tool result of this shape, at this rate, under these settings" is supportable; "this partner integration is exploitable" is not, until the contract match is demonstrated. AgentDojo, InjecAgent and AgentHarm are published suites built on exactly this pattern; each holds a different part of the surface fixed, so read which one before quoting its numbers.