LLM Safety & Security

A schema constrains shape, not meaning. It guarantees the reply parses and carries the required fields and types. It says nothing about whether a field holds an invented price, a competitor's name, an unsafe instruction, or text that is dangerous where you paste it.

An agent running on a human's session inherits every permission that human has, and its actions are logged as theirs. Giving the agent its own principal lets you scope it narrowly, revoke it alone, and attribute what it actually did.

Input screening rejects harmful requests before you pay for a generation and before they enter the context. Output screening catches harm the model produces anyway, including harm that arrived through retrieved documents or tool results the input check never saw.

Shipped recognizers cover identifiers standardised nationally or industry-wide; a facility-assigned medical record number has no fixed format, so nothing matches it. Misses leak silently, while blanket masking strips the dosages, dates and lab values the answer depended on.

Three placements, each catching a different failure. Input rails screen the incoming turn before or alongside generation. Output rails screen the finished response before the user sees it. Tool rails wrap each function call, checking arguments before the side effect and treating the returned result as fresh untrusted input.

The lethal trifecta is one agent session holding three things at once: access to private data, exposure to untrusted content, and an outbound channel. Any two are survivable; all three let injected text steal data. Remove one capability.

A fixed suite only measures attacks someone already wrote down. A real attacker sees your defense and adapts, so passing the suite proves the known attacks are patched — not that the system resists new ones.

Direct injection arrives in the user's own turn, so the user is the adversary. Indirect injection hides in content the application feeds the model later — retrieved documents, tool results, memory — so a third party attacks through your data pipeline.

Replace generic tools with narrow, capability-shaped ones, bind the dangerous parameters server-side rather than letting the model choose them, split read from write, and enforce every limit at the API that executes the call — never in the prompt.

At query time in the retrieval layer, as a pre-filter derived from the caller's verified identity — or as physically separate indexes. Anything above that, including instructions to the model, is advice rather than a boundary, because a retrieved chunk is already disclosed.

Detection is probabilistic and only has to be wrong once, while an attacker can retry and tune against it. Adaptive attacks against a dozen published defenses succeeded over 90% of the time. Filters reduce noise; a deterministic architectural limit is the boundary.

Attack-success rate is meaningless without its denominator and the attacker's budget. Report success per attempt, not per prompt, state how many attempts and whether the attacker adapted, and publish the curve of success against attempts rather than one number.

Not at the network edge. Everything that lands in the context window is one untrusted zone, whatever its source. The boundaries that matter sit between that context and any privileged action, and between each writer and each data source the system ingests.

Isolation strong enough to assume the code is hostile: a hardened runtime or microVM rather than a shared process, no host mounts, no ambient cloud credentials or metadata access, default-deny network egress with a narrow allowlist, resource caps, and a fresh instance destroyed after each session.

Score the image, not just the words around it. Text-only classifiers pass a clean caption over an unsafe picture, so the pipeline needs image-capable moderation — either a joint text-plus-image call or a dedicated image classifier — plus sampled frames for video rather than one thumbnail.

Treat anything in the context window as disclosable to that session's user: no credentials, no other users' data, no internal policy you would not publish. Secrets belong in the tool-execution layer, access decisions in code, sensitive policy behind an authorized retrieval call.

Plan-Then-Execute fixes the sequence of tool calls before any untrusted content is read, so injected text cannot add or reorder steps. It stops control-flow hijacking. It does not stop untrusted data from poisoning the arguments and content of the steps already planned.

Model-level probes such as garak or PyRIT attack a bare model endpoint and measure what it says. Agent-level suites such as AgentDojo or AgentHarm attack the assembled system and measure what it does — which tools fired and whether a harmful task was completed end to end.

The LLM Top 10 models a prompt-in, text-out application. The Agentic Top 10 covers what appears only when a system plans, keeps memory across sessions, calls tools and talks to other agents — goal hijack, tool misuse, memory poisoning, rogue agents.

Treat tool metadata as untrusted code that lands in the model's context: pin servers to a reviewed version or digest, require a diff review before any description change takes effect, fail closed on unexpected changes, and keep capability enforcement at the resource server so no description can grant access.

When your policy is specific to your product and changes often. A policy-conditioned classifier reads your written policy at inference time, so a rule change is a text edit rather than a labelling-and-retraining cycle. Fixed taxonomies stay cheaper and faster for stable, universal harm categories.

A framework pays when policy churns, when non-engineers must read or edit it, and when you need many off-the-shelf checks composed the same way. Hand-rolled validators win when you have three rules, tight latency budgets, and no appetite for a second runtime and its DSL.

Treat a blocked turn as a product state, not an error. Name the category of what you cannot do, offer the sanctioned path for it, and carry the user's context into that path. A refusal that ends the conversation converts a policy win into an abandoned session.

Dual-LLM keeps untrusted text away from the privileged model by passing symbolic references, but nothing checks what the privileged program then does with those values. CaMeL and FIDES attach provenance and permission labels to every value and enforce policy at the point of action.

Run the red-team job against every release candidate, block on any new success in high-severity categories rather than on an aggregate score, keep every past incident as a permanent case, and run each case several times because any single success counts as a failure.

Each hop gets its own token. A tool server must never replay a token minted for something else at a downstream API, and every service must reject tokens whose audience does not name it — otherwise the server becomes a confused deputy acting with authority it was never granted.

Treat human review as a component of the system, not a safety net bolted on. Every automated decision records the policy clause and model version behind it, affected users are told which rule they broke, appeals go to a reviewer with that evidence, and reversals feed back into the policy.

Give rails distinct verbs and a fixed precedence. Blocks outrank redactions, which outrank rewrites; a block short-circuits. Decide per rail whether it fails open or closed when it errors, run mutating rails in a defined order, and cap any repair attempt so rails cannot fight each other indefinitely.

The Rule of Two allows an agent session at most two of: untrusted input, sensitive access, and the ability to change state or communicate externally. A triage agent holding all three should be split into a public-only reader that emits a structured result and a privileged session that never reads issue text.