Where do the trust boundaries sit in a retrieval- and tool-using LLM assistant?

## The boundary people draw, and why it is wrong Asked to mark trust boundaries on an LLM application, most teams draw one line at the HTTP edge: outside is the internet and the user, inside is the app, the retrieval index, the tool layer and the model. It mirrors how they diagram every other service, and for a stateless CRUD API it is roughly right, because in that system nothing the user sends is ever interpreted as a command by a downstream component. An LLM application breaks that assumption. The context window is fed from many sources, and everything in it is available to the model as instruction. Authenticating the caller at the edge tells you who initiated the request; it tells you nothing about who wrote the 4,000 tokens of retrieved manual that arrived three hops later. ## Trust follows the writer The usable rule is that a piece of data inherits the trust level of **whoever can write it**, not of the store it sits in or the network zone it travels through. Apply that to a plant-maintenance copilot that reads PLC sensor streams, supplier equipment manuals and the work-order database, and can schedule a line shutdown: - The vendor manual corpus is **untrusted**. Suppliers upload PDFs; nobody in your org authors them and nobody reads all of them. - The work-order database is **semi-trusted**. Writers are authenticated employees and contractors, but the fields are free text and the population is large, so an entry is attributable but not vetted. - The PLC telemetry stream is **machine-authored but attacker-influenceable** — values and labels can be steered by anyone who can affect the plant floor or the historian. - The SCADA write path is the **crown jewel**: the one place where the system's output stops being text and becomes physical consequence. Once that is written down, the map is no longer "user versus system". It is a set of sources of differing provenance flowing into a shared context, and one high-consequence sink. ## Where the boundaries actually are **Between each writer and each ingested source.** Every path that puts bytes into a corpus, memory store or tool result is a boundary crossing with a named owner on the far side. This is the boundary teams most often omit, because the ingestion job looks like plumbing rather than an attack surface. **Between the context window and every privileged sink.** The context is one zone; the sinks are the boundary. Concretely: any tool that writes, spends, sends, deletes or schedules; any string handed to a shell, a SQL engine, a template renderer or a browser; any outbound request that could carry data off the premises. Enforcement at that line must be deterministic code with its own view of the caller's identity and scope, because it is the only part of the system whose behaviour does not depend on how persuasive the surrounding text was. **Between the model's output and any interpreter.** Model output is untrusted input to whatever consumes it. If a downstream component parses it as markup, code or a command, the boundary is that parser, and the control is validating and escaping there — the same discipline you would apply to a form field. **Between tenants, sessions and agents.** Anywhere context or memory can cross from one principal to another, there is a boundary. A memory record written under one user's session and recalled under another's is a crossing even though nothing left the process. ## The model is not a boundary The single most consequential line in this exercise: **the model is not an enforcement point.** It cannot reliably separate instruction from data, and defenses that ask it to — system-prompt admonitions, delimiters, spotlighting, instruction-hierarchy training — are probabilistic. Adaptive attackers have broken published defenses of this kind at very high success rates. Frontier models may be under a percent on a single attempt and several percent across a hundred adaptive attempts, which is fine as risk reduction and useless as a boundary. Whatever must not happen has to be impossible in code that runs outside the model. A corollary that surprises people: giving the model a *stronger* system prompt does not move any boundary. Adding a deterministic check in front of the shutdown-scheduling call does. ## Turning the map into findings For each boundary, record three things: what crosses it, who owns the far side, and what enforcement runs at the crossing. That is what makes the map actionable. "Retrieved manual text crosses into the context window; owner is the supplier-onboarding team; enforcement is none today" is a finding with an owner. "We use RAG" is not. Then check reachability: for every privileged sink, list the request paths that can trigger it and mark which of them have consumed untrusted content. A sink reachable only from paths that touched nothing external is a genuinely different risk from one reachable after reading a supplier PDF. That reachability question — not the model's benchmark refusal rate — is what determines whether the design is defensible. ## Common failure signals Teams that have not done this exercise say things like "the corpus is internal so it is trusted", "the tool is ours so its output is safe", "we authenticate users so the input is fine", and "the system prompt forbids that". Each of those sentences names a boundary that does not exist.