Which internal content is unsafe to place in a system prompt, and what should hold it instead?

## The working assumption Design as if every token in the context window will eventually be read by the person on the other end of the session — verbatim, paraphrased, or inferred from behaviour. That is not fatalism; it is the only assumption that produces a system whose privacy properties you can state. Everything else follows from it. The reasoning is straightforward. The model's job is to use its context to produce text. Any instruction to withhold part of that context competes with that job, and the competition is decided probabilistically, per request, by a system that is also processing user input and quite possibly untrusted document text. As of mid-2026 the published work on adaptive attacks against injection and jailbreak defences has made the consensus explicit: probabilistic filters and instruction-hierarchy training reduce casual failure rates but do not constitute a boundary. A defence measured against a fixed set of attempts overstates its own robustness. ## What must not be in the context window **Credentials of any kind.** API keys, database passwords, signing keys, service tokens. The failure here is worse than disclosure of text, because a credential is transferable: whoever obtains it can act with the system's authority, from anywhere, long after the conversation ends. **Other people's data.** Records, notes, contact details or history belonging to anyone other than the current user. If it is in context, it is potentially in the answer. **Internal policy that must remain internal.** An escalation policy with the thresholds at which the assistant hands off to a human; a discount ceiling; a fraud-detection rule; the criteria for prioritising a case. Teams put these in the system prompt because the model needs them to behave correctly, and it is exactly that need which makes them extractable — the behaviour they produce is observable. **Hidden instructions relied on as access control.** "Do not perform refunds for users outside the pilot group" is not an access-control rule; it is a hope. The check belongs where the refund is executed. ## Where each thing belongs *Credentials → the tool-execution layer.* The model emits a tool call with business parameters. The server attaches the credential, performs the call, and returns only the result. The key never enters a prompt, a completion, a log line, or a cache. If the model can name a secret, the secret is already in the wrong place. *Authorization → code, before the effect.* Whether this user may issue a refund, view a chart or cancel an order is decided by the executing service using the session's identity, independent of what the model asked for. This also means the model may confidently request something forbidden and simply be refused — a boring, correct outcome. *Sensitive policy → an authorized retrieval call.* Rather than pasting the escalation matrix into every request, expose a tool that returns the applicable rule for the current case and the current user's role. Two benefits: the context holds only the slice needed for this interaction, and the retrieval is subject to the same access checks as anything else. It costs a round trip and some latency. *Other users' data → never fetched.* Retrieval and tool calls are scoped by the caller's identity, so the data does not reach the prompt to begin with. Minimisation beats redaction beats filtering, in that order. ## What confidentiality you can still have Not everything is hopeless. You can make the extractable material *worth little*. If the system prompt contains only tone, format and task framing, its disclosure is embarrassing at worst — competitors learning your prompt style is a business concern, not a security incident. You can also reduce casual extraction with an instruction and an output check; that is worth doing precisely because it stops the low-effort probe, provided nobody mistakes it for a guarantee. And you can shrink the window of exposure: fetch sensitive slices per request instead of pinning them into a system prompt that is present for every turn of every session. What you cannot do is defend a secret by asking the model to keep it. Behavioural inference alone defeats that — a user who cannot read the escalation policy can still discover its threshold by observing where the assistant's behaviour changes, and no output filter touches that channel. ## The review question A useful test when reading a system prompt in review: if this text were published on the company blog tomorrow, what would break? If the answer is "nothing, it would just be uninteresting," the prompt is correctly scoped. If the answer names a credential, a customer, or a rule that only works while it is secret, that content needs to move below the model — into the tool layer, into an authorization check, or behind a retrieval call — before the feature ships.