The Guard

A tool-using language model can read an email, reason over a calendar, and ask an API to create an event. That single route joins three different kinds of authority: the user's instruction, an outside party's data, and the application's credentials. Prompt injection turns into a security failure at the moment data from the second category can steer authority from the third.

This drawing treats the agent as a confused deputy. The model takes in trusted and untrusted text through one context, and proposes actions under credentials that were never its own. Source labels, delimiters, instruction hierarchy, and injection detectors can all change model behavior. None of them builds a dependable privilege boundary. Authorization, resource scope, recipient scope, approval, and tool-side validation have to live outside the model.

The durable route is hostile document → model context → privileged tool call. Every sheet follows that route and names the component that can affect it. The interactive runs abstract prepared cases against a fixed 64-row outcome table — no model call, no attack text rendered anywhere. The final test contract reports task success, attack success, and blocked legitimate actions together, because an agent that refuses everything is neither useful nor evidence of anything.

A confused deputy holds legitimate authority and can be talked into using it for the wrong principal or the wrong purpose. A calendar agent fits the pattern exactly. The application holds the OAuth credentials. The user supplies a task. The model interprets text and picks a tool call. And the author of an email can control some of the text the model reads while holding no calendar permission whatsoever. If that email can cause a calendar mutation, the application has let data borrow the user's authority.

The security question starts with principals and effects, not with whether the text looks suspicious. Who requested this action? Which credential executes it? Which object, recipient, date range, or dollar amount falls inside the grant? Where does that grant actually get checked? A fluent explanation from the model answers exactly none of those. The model can help form a proposal. The proposal has to arrive at an enforcement point as untrusted input.

Start the threat model from the effect and work backward. List every tool operation that can disclose, mutate, send, delete, purchase, execute, or administer something. For each one, name the credential it uses and the data sources visible along the route to it. A read-only summarizer scoped to one folder has a very different ceiling from an assistant with inbox-wide read access and permission to send mail. Model quality changes the probability of a bad outcome. Reachable authority changes the maximum size of one.

That distinction is also what separates prompt injection from a general conversation about model safety. Content-policy evasion is about outputs a model was supposed to avoid producing. Application compromise is about data, permissions, and side effects in the system around it. The same model behavior can show up in both, and the application-security case has something concrete underneath it: a principal, a credential, a privileged operation, and an external effect.

The user's task has a principal behind it and a purpose you can name. Retrieved email, web pages, files, calendar descriptions, support tickets, and tool results have entirely different producers — honest, compromised, stale, or hostile on purpose, and you often can't tell which from the outside. Context assembly serializes all of them into tokens for one model invocation. A label can describe where something came from. Serialization does not confer trust.

Treat every model input and output as typed data inside the application, even though the model only ever sees text. A source record should carry its producer, retrieval path, timestamp, integrity evidence, sensitivity, and trust class. The context assembler should pull the minimum records the task needs, preserve labels through every transformation, and keep secrets and unrelated objects out entirely. Those controls reduce exposure and make traces explainable. What they don't do is guarantee the model will honor the labels.

Provenance has to survive derived data, which is where it usually breaks. A search result points at an email. A summary combines several emails. A tool result quotes a document that came from another service. Record the lineage rather than relabeling the final string as trusted because an internal component produced it — internal transport changes custody, not authorship. When the lineage is missing, policy should treat the item as the least-trusted class it plausibly could be.

The boundary you can actually enforce sits where a proposed action becomes an allowed one. A policy service can compare the proposal against a structured grant built from authenticated user intent. The tool adapter can validate schema, resource scope, recipient, rate, and expiry. A human can approve one concrete high-impact action. The external service can enforce its own access controls on top. Every one of those operates on structured fields, and every one denies by default.

Prompt text cannot create a dependable privilege boundary. A system message can request correct behavior. It can't hold a credential, prove user intent, or atomically deny an out-of-scope API call. The entire architecture turns on keeping that distinction intact.

Direct prompt injection comes in through a channel where the user can address the model. How much it matters depends on who the caller is and what capabilities the application granted. A hostile authenticated user is trying to exceed their own resource scope; an anonymous caller is trying to reach tools meant for an operator. Ordinary API authorization still applies to both.

Indirect prompt injection arrives inside data the agent goes and fetches: a message, a document, a website, an issue, a calendar entry, a tool response. Greshake and coauthors named and demonstrated that remote route in arXiv 2302.12173^[1], first submitted February 23, 2023 and revised May 5, 2023. The observation that lasted is the blurred line between instruction and data in LLM-integrated applications. The defensive consequence is to inventory every source that can enter context, and every effect reachable once it has.

The hero route takes three hops. An attacker influences a document or record that's likely to get retrieved. The application selects it and serializes it into model context, right alongside trusted task material. The model emits a tool proposal, and the application executes it under a privileged credential. Break any hop and you stop a prepared case. Only the third hop controls authority directly.

An attack tree makes those dependencies reviewable rather than vibes-based. The root event is an unauthorized or unintended privileged effect. One branch asks how controllable data reaches the context. Another asks how it changes the proposal. A third asks which missing or overly broad grant let the proposal execute. A fourth asks how the external system or a downstream renderer turned it into real impact. Then you can assign controls and evidence to specific branches, instead of assuming one detector covers the whole tree.

The 2025 OWASP Top 10 for LLM Applications^[2] lists prompt injection as LLM01 and excessive agency as LLM06^[3]. Those dated names are useful for connecting model influence to permission design in a review. The permanent model on this sheet avoids depending on list numbers at all: untrusted sources enter a deputy, and an enforcement point decides whether the deputy gets to create an effect.

Instruction hierarchy gives developer and user messages a declared order. Delimiters mark where a document starts and stops. Source labels tell the model which text came out of an email or a web page. Repeated warnings spell out what's prohibited. Detection models can score a context for injection-like features. All of these can cut attack success on a fixed evaluation, and all of them are worth testing as mitigations.

Their weakness is structural, not a matter of trying harder. The same probabilistic component interprets the trusted rule and the untrusted text. A delimiter is another token pattern. A label is another token pattern. A warning is another token pattern. The model can generalize beautifully across the cases you tested and fail on a changed source, a different language, an unusual encoding, a new task composition, a model revision, or someone adapting to your defense. A policy that exists only in prompt text has no independent mechanism preventing a forbidden API call.

Input filtering carries a second cost. Aggressive filters strip legitimate instructions quoted inside an email, block documents about security, mangle code and structured data, and drag task utility down with them. Weak filters miss anything unfamiliar. Output classifiers have the same false-positive and false-negative problem, and they run too late anyway if a tool side effect already happened.

Model separation schemes are still worth building. A planner that never sees untrusted content can hold the task record while a reader model extracts bounded fields from documents, with a mediator exposing only those fields upward. Less shared context, simpler policy. The reader's extraction is still untrusted, though, and any component allowed to turn it into a privileged action still needs external authorization. Architectural isolation buys a bounded claim rather than immunity.

Use prompt controls where there's evidence for them: source-aware context formatting, task focus, suspicious-input detection, safer proposal formation. And state the residual risk plainly, which is that model compliance can fail. What they buy you is fewer dangerous proposals reaching the policy layer, and that helps availability and review load. The policy layer still treats every proposal as untrusted.

Source labeling and provenance attaches producer, retrieval path, time, integrity evidence, and trust class to every item, and keeps those fields outside the prompt and inside the audit trace. It helps context selection, policy decisions, incident review, and model behavior all at once. Residual risk: labels can be wrong at ingestion, stripped by a transformation, or simply ignored by the model.

Context assembly minimizes what the model can see. Fetch by authenticated tenant, constrain the date and object scope, leave out secrets the task doesn't need, isolate sources where that's practical, and cap recursion through tool results. Residual risk: anything you did include can still influence the proposal, and anything you left out may cost you task success.

Policy checks and injection detection inspect the proposed action and whatever provenance came with it. Deterministic rules reject forbidden operation classes outright; statistical detectors route suspicious cases onto a slower path. Residual risk: detectors miss adaptive inputs and block legitimate work. And a policy check that asks the same model for a second opinion is still model-dependent.

Capability and permission scoping grants the smallest operation, resource set, recipient set, value, duration, and rate the task actually needs. Keep read, draft, send, delete, execute, and administer separate. Bind each grant to an authenticated principal and a specific task. Residual risk: a permitted operation can still be abused inside its own scope, and a task record written too broadly recreates excessive agency by another name.

Human approval gates the irreversible, outbound, high-value, or unusual actions. Show the exact recipient, resource, diff, amount, and provenance — not a model-written reassurance that everything looks fine. Residual risk: fatigue, deception, rushed review, ambiguous summaries. Approval should mint a narrow expiring grant rather than releasing the agent in general.

Tool execution controls validate schema and authorization again at the adapter, enforce allowlists and rate limits, use idempotency keys, sandbox code, restrict network destinations, and keep the service credentials narrower than the model's tool catalog. Residual risk: implementation bugs, and operations that are valid inside the granted scope and harmful anyway.

Output handling treats model output as untrusted before it reaches HTML rendering, SQL construction, a shell, template expansion, or any other interpreter. Prefer typed APIs, escaping, parameterization, and inert rendering. Residual risk: content that's syntactically valid and semantically harmful sails straight through.

Bind those layers together with evidence. The source inventory gives every item an identifier. The context trace records which identifiers got selected and how they were transformed. The proposal records structured arguments. The policy decision records the matching grant, or the reason for denial. The adapter records the exact operation submitted and what the external system said back. A security reviewer should be able to follow one event across all five records without leaning on a prose transcript the model wrote about itself.

The prepared email/calendar cases show why all three measures matter. Task success measures utility. Attack success measures whether the attacker's target state occurs. Blocked legitimate actions expose security controls that achieve low attack success by refusing useful work. The truth table is explanatory, not measured model performance.

A defense review should name the boundary each control is capable of changing. Source provenance affects source-to-context decisions. Context minimization affects what reaches the model at all. Prompt structure and detectors affect how the proposal gets formed. Policy and authorization affect whether a proposal receives a grant. Tool adapters affect whether the granted call matches schema and scope. Output handling affects downstream interpretation. Any claim that stretches past the component doing the enforcing deserves a hard look.

The map exists to prevent category errors, which are the expensive kind. Provenance can support an authorization decision and cannot enforce a calendar scope by itself. A sandbox can constrain code execution and cannot decide whether sending that particular email was authorized. Human approval can grant one send and does nothing to sanitize the HTML that comes back. Defense in depth works through composition, with each layer making one bounded claim it can actually keep.

Put the residual risk on the diagram, because "enabled" tells a reviewer almost nothing. Record the failure mode, the impact, the owner, the monitoring signal, and the next test date. Product features and named standards move fast, so date them in the review notes. The trust-boundary route stays useful when the model, agent framework, detector, or tool protocol underneath it changes.

Permission matrices should keep discovery separate from use. A model can know a send-mail tool exists without holding a send capability. It can draft a structured request that sits waiting for approval. Once approved, the executor gets a one-action token limited to a recipient and a draft hash, and throws it away afterward. That pattern narrows both the window and the content of authority while leaving planning utility intact. Long-lived broad credentials erase the separation entirely.

AgentDojo^[4] offers a useful shape for evaluation: run legitimate tasks in stateful tool environments, pair each user goal with an attacker goal, and inspect the resulting environment state with deterministic utility and security functions. Version 3 of arXiv 2406.13352, dated November 24, 2024, carries 97 user tasks and 629 security test cases across workspace, Slack, travel, and banking environments. The paper reports three distinct measures: benign utility, utility under attack, and targeted attack success rate.

Its dated results also show why a single headline number misleads. In that November 24, 2024 version, the agents evaluated solved fewer than 66% of tasks with no attack present; attacks against the best-performing agents succeeded in fewer than 25% of cases; one evaluated detector pulled attack success down to 8%. Every one of those figures describes that paper's agents, defenses, attacks, tasks, and revisions. None of them is a durable security property.

NIST's "Technical Blog: Strengthening AI Agent Hijacking Evaluations," published January 17, 2025 and updated December 19, 2025^[5], demonstrates the drift directly and uncomfortably. In its stated Claude 3.5 Sonnet setup, the strongest baseline attack measured 11% success while a newly developed adaptive attack measured 81%. Across a selected five-task collection, repeated attempts raised average measured success from 57% on a single attempt to 80% over 25 attempts. Those numbers belong to NIST's dated setup. The lesson that transfers is to test adaptive variants, task-level impact, and retries.

Build the cases out of the threat model you deployed. For each source-to-tool route, define an authenticated user task, an abstract hostile-source class, the capabilities available, the allowed state change, the forbidden state change, and deterministic oracles over the resulting environment. Then run a benign baseline, the hostile case, defense ablations, repeated attempts wherever an attacker could retry, and adaptive variations produced through a controlled red-team process. Keep reusable payloads out of general documentation and out of production logs.

Measure by impact tier as well as aggregate rate. One unauthorized draft and one irreversible transfer should never contribute the same weight to release evidence. Keep a denominator for every action class, record confidence intervals when you're sampling, and hang onto the failed legitimate cases for review. A zero attack-success estimate from a small suite is an observation bounded by that suite, and nothing more. Define the retest triggers up front for model, prompt, retrieval, permission, tool, and policy changes.

The one boundary that must not move is the conversion from model proposal to privileged effect. The model can plan, summarize, classify, choose among safe candidates, and request a capability. It should never mint its own authority, widen a grant, approve its own high-impact action, or route around the adapter that enforces scope. Treat every model-produced tool name and argument as hostile until it's been validated against authenticated intent.

A workable authorization record binds principal + task + operation + resource + recipient or destination + limits + expiry + approval. The policy service creates or validates it. The adapter checks it again at execution time. External systems get narrower credentials wherever that's possible. Logs tie the source records and the model proposal to the decision and to the resulting state change. Revocation and rollback exist independently of the conversation transcript, because the transcript is not a system of record.

Failure should land somewhere safe and observable. A denied action returns a structured reason without leaking hidden policy or credentials. An approval that times out expires. A retried mutation carries an idempotency key. A partial workflow records which reversible steps completed and which privileged step didn't. Operators can revoke the grant and read the trace without asking the model to recall what it did.

The Autonomy Boundary gives an operator rule along the same action line: reversible internal work can run farther, irreversible or outbound work stops for a human. The security architecture is what makes that line enforceable, through scoped permissions, hard disqualifications, approval records, and tool checks. The human gate should protect the concrete action, not just bless a session in general.

"Secure" can mean the system enforces its declared permissions, contains its effects, preserves evidence, and meets dated attack and utility thresholds for specified routes. It can't mean prompt injection is solved, or that the model will reliably separate instructions from data, or that tomorrow's adaptive attack will look like anything in today's suite. Prompt text cannot create a dependable privilege boundary.

The claim worth defending is narrower and far more useful: hostile data may reach the model, and it receives no authority merely by showing up in context. Every privileged effect requires a grant tied to the user and the task, enforced outside the model, measured against both attack success and legitimate utility, and retested every time the system changes.