How do you keep API keys and credentials out of LLM prompt logs and traces?

## Why prompt logs are a distinct problem Every serious LLM application logs prompts and traces, because you cannot debug or evaluate a system whose behaviour you cannot replay. The consequence is a store that contains, for every request, the entire assembled context: system prompt, user message, retrieved documents, tool definitions, tool results and the model's output. It is the most complete copy of your application's working data that exists anywhere, and it typically lives in a tracing or observability tool where read access is granted to the whole engineering organisation rather than to the narrow group entitled to the underlying records. Credentials end up in that store more often than people expect. A user pastes an API key into a support chat while describing their problem. A tool result echoes back a request that included an authorization header. A configuration file retrieved as context contains a connection string. A stack trace captured as a tool error includes a token in a URL. None of these require an attacker; they are ordinary application behaviour, and each one writes a live secret into a searchable log. ## Prevention comes first The better control is upstream. A model does not need a credential to use a tool, because the model does not make the outbound request; the tool runtime does. Keep keys in the runtime's configuration or secret store, attach them when the request is built, and never place them in a tool definition, a system prompt or a tool argument. If a design requires the model to pass a token around, that design has an easy fix: give the tool a handle or session reference and let the runtime resolve it to the real credential. Similarly, tool results should be shaped before they enter the context. Returning a raw HTTP exchange, including headers, is convenient and is how authorization headers reach prompts. Return the fields the model needs. ## Redaction at the write path Containment still matters, because prevention is never complete and users will paste things. The rule is that redaction runs before the record is written, not on the way out to a dashboard and not in a nightly cleanup job. Once a live key is in the store it has been exposed to everyone with read access and to any downstream export, and a later scrub does not undo that. What to match: the credential formats you issue yourself, which is why issuing keys with a recognisable, high-entropy prefix pays for itself here; common third-party key shapes; authorization headers and bearer tokens; private key blocks; connection strings with embedded passwords. Pattern matching is imperfect in both directions, so pair it with a shape-based check such as high-entropy strings in fields that should not contain them. What to write instead: a fixed placeholder is simplest. A keyed hash of the matched value is often better, because it lets you correlate that the same secret appeared in fifty traces, or match against a known leaked value, without storing anything usable. Storing a truncated prefix alongside the hash helps humans identify which key it was without recovering it. ## Treat a redaction hit as a signal The most valuable output of the redactor is not the redacted log; it is the count. Every hit means a real credential reached a real prompt, which means the prevention layer failed somewhere: a tool returning raw headers, a document ingested with a config file in it, a user flow that invites pasting secrets. Alert on hits, attribute them to a code path, and fix upstream. A redactor whose hit rate is quietly climbing is a system leaking secrets into context faster than anyone is noticing. ## Access and blast radius Beyond redaction, the trace store deserves the access control its contents warrant: authenticated access, role-scoped rather than org-wide reads, and audit of who queried what. If traces are exported to an evaluation dataset or replayed against a third-party judge, that export is a second copy with its own boundary and needs the same redaction applied, not merely inherited. ## Rotation is the backstop If a live credential did reach the store before redaction existed, treat it as disclosed and rotate it. This is the point where a candidate either understands secrets or does not: a secret that has been written to a broadly readable system is compromised regardless of whether anyone is known to have read it, and the only remediation that restores the property you cared about is issuing a new one.