How does server-side tool search over deferred definitions keep a 300-tool catalog out of context?

## The problem it solves The default contract of function calling is that every tool the model may use is described in the prompt of every request. That is fine at ten tools and untenable at three hundred: tens of thousands of tokens on every turn, a long homogeneous list where the relevant entry is buried mid-prompt, and a dozen plausible candidates for any given intent. Deferred loading breaks that contract. The catalog still exists, but the definitions live server-side until something asks for them. ## How the flow actually runs 1. You register the full catalog but mark the bulk of it deferred, so those definitions are not serialized into the prompt. 2. The prompt carries a small always-loaded set: the tool-search tool, plus any tool used on essentially every turn. 3. The model reads the user request and, finding nothing that fits in its loaded set, calls search with query terms — "leave", "time off", "pto", or a literal tool name it half-remembers. 4. The server matches those terms against the deferred definitions and returns, or directly injects, the top few full definitions. 5. The model now sees three or four candidates instead of three hundred, and calls one through the ordinary tool path. Nothing about the call itself is special — a deferred tool that has been loaded is just a tool. Definitions that have been pulled in stay available for the rest of the conversation, so the search cost is paid once per capability, not once per call. ## Why keyword matching rather than embeddings The instinct of anyone who has built retrieval is to embed the tool descriptions and do vector search. In practice the field settled on keyword matching — regex and BM25 flavours — for four reasons. - **No index lifecycle.** Every tool a team adds, renames or edits would require re-embedding and re-indexing, and a stale index silently hides new capabilities. Keyword matching reads the catalog as it stands. - **The queries are keyword-shaped.** A model searching for a tool emits domain nouns and near-identifiers, not paragraphs. Lexical match is a good fit for short, jargon-heavy queries; this is the regime where BM25 has always been competitive. - **Exact identifiers matter.** Tool names, namespaces and parameter names are literal strings. Embeddings blur exactly the distinctions — `create_report` versus `create_request` — that you most need preserved. - **Debuggability.** When a search misses, you can reproduce the match deterministically and see which term failed. An embedding miss is an opaque similarity score, and it drifts when the embedding model changes underneath you. That is a judgement about tool catalogs specifically, not a general claim that lexical search beats semantic search. Tool catalogs are small, curated, jargon-dense and edited constantly — the conditions that favour lexical matching. ## What must stay loaded The entry point cannot itself be deferred: if nothing is loaded, the model has no way to ask for anything. Beyond the search tool, keep a small hot set — the two or three tools used on almost every turn — because paying a search round trip for them is pure latency. Everything else defers. A useful sizing heuristic is that below roughly two or three dozen tools the machinery costs more than it saves, and you are better off loading the lot. ## Failure modes worth naming The characteristic failure is a **search miss that reads as an absent capability**. The model searches "book annual leave", finds nothing because the catalog metadata says "absence request", and tells the user the assistant cannot do that — a false negative that looks like a product gap rather than a retrieval bug. Mitigations: instrument searches that return zero hits and treat them as a metadata backlog; instruct the agent to retry with broader or alternative terms before concluding nothing exists; and offer a way to enumerate a namespace as a fallback. The second failure is **over-broad queries** that pull twenty definitions back in and recreate the crowding you were avoiding. Cap how many definitions a single search may inject. ## Costs and interactions The direct cost is one extra model turn plus the search itself before real work starts. Against that: the static prompt prefix becomes small and stable, which is good for both cost and prompt caching, and per-turn token spend drops sharply for conversations that touch only a few capabilities. Latency-sensitive paths that always use the same tool should keep it loaded rather than searching for it every session. A related pattern for very large catalogs is to let the agent discover definitions from files in its execution sandbox and read only the ones it needs. That is the same principle — hold references, fetch on demand — applied through the filesystem instead of a search tool.