The Guard

A tool-using language model can read an email, reason over a calendar, and ask an API to create an event. That route joins three different kinds of authority: the user’s instruction, an external party’s data, and the application’s credentials. Prompt injection becomes a security failure when data from the second category can steer authority from the third.

This drawing treats the agent as a confused deputy. The model receives trusted and untrusted text through one context and proposes actions under credentials it does not own. Source labels, delimiters, instruction hierarchy, and injection detectors can change model behavior. None of them creates a dependable privilege boundary. Authorization, resource scope, recipient scope, approval, and tool-side validation must remain outside the model.

The durable route is hostile document → model context → privileged tool call. Each sheet follows that route and marks the component that can affect it. The interactive uses abstract prepared cases and a fixed 64-row outcome table; it contains no model call and renders no attack text. The final test contract reports task success, attack success, and blocked legitimate actions together, because refusal alone is neither a useful agent nor adequate evidence of security.

A confused deputy has legitimate authority and can be induced to use it for the wrong principal or purpose. A calendar agent fits the pattern. The application holds OAuth credentials. The user supplies a task. The model interprets text and chooses a tool call. An email author may control some of the text the model reads, while holding no calendar permission. If that email can cause a calendar mutation, the application has let data borrow the user’s authority.

The security question starts with principals and effects, not with whether the text looks suspicious. Who requested the action? Which credential executes it? Which object, recipient, date range, or dollar amount falls inside the grant? Where is that grant checked? A fluent explanation from the model answers none of those questions. The model can help form a proposal, but the proposal must arrive at an enforcement point as untrusted input.

Start the threat model from the effect and work backward. List every tool operation that can disclose, mutate, send, delete, purchase, execute, or administer. Identify the credential used for each operation and the data sources visible on the route to it. A read-only summarizer with access to one folder has a different ceiling from an assistant holding inbox-wide read access and permission to send mail. Model quality changes probability; reachable authority changes maximum impact.

This distinction also separates prompt injection from a broad model-safety discussion. Content-policy evasion concerns outputs a model is expected to avoid. Application compromise concerns data, permissions, and side effects in the surrounding system. The same model behavior may appear in both, yet the application-security case has a concrete authority path: a principal, a credential, a privileged operation, and an external effect.

The user task has a principal and an intended purpose. Retrieved email, web pages, files, calendar descriptions, support tickets, and tool results have separate producers. Some may be honest, compromised, stale, or deliberately hostile. Context assembly often serializes all of them into tokens for one model invocation. A label can describe their provenance, but serialization does not confer trust.

Treat every model input and output as typed data in the application even if the model sees only text. A source record should carry producer, retrieval path, timestamp, integrity evidence, sensitivity, and a trust class. The context assembler should select the minimum records needed for the task, preserve labels through transformations, and keep secrets or unrelated objects out. These controls reduce exposure and make traces explainable. They do not guarantee that the model will obey the labels.

Provenance must survive derived data. A search result may point to an email; a summary may combine several emails; a tool result may quote a document fetched from another service. Record the lineage instead of relabeling the final string as trusted because an internal component produced it. Internal transport changes custody, not authorship. When lineage is missing, policy should treat the item at the least-trusted plausible class.

The enforceable boundary sits where a proposed action becomes an allowed action. A policy service can compare the proposal with a structured grant created from authenticated user intent. The tool adapter can validate schema, resource scope, recipient, rate, and expiry. A human can approve a concrete high-impact action. The external service can enforce its own access controls. Each check operates on structured fields and denies by default.

Prompt text cannot create a dependable privilege boundary. A system message can request correct behavior; it cannot hold a credential, prove user intent, or atomically deny an out-of-scope API call. The whole architecture turns on keeping that distinction intact.

Direct prompt injection enters through a channel where the user can address the model. The security impact depends on the caller’s identity and the application’s granted capabilities. A hostile authenticated user may attempt to exceed their own resource scope; an anonymous caller may try to reach tools intended for an operator. Ordinary API authorization still applies.

Indirect prompt injection arrives inside data the agent retrieves: a message, document, website, issue, calendar entry, or tool response. Greshake and coauthors named and demonstrated this remote route in arXiv 2302.12173^[1], first submitted February 23, 2023 and revised May 5, 2023. Their durable observation is the blurred instruction/data distinction in LLM-integrated applications. The defensive consequence is to inventory every source that can enter context and every effect reachable afterward.

The hero route has three hops. First, an attacker influences a document or record likely to be retrieved. Second, the application selects and serializes it into model context beside trusted task material. Third, the model emits a tool proposal that the application accepts under a privileged credential. Breaking any hop can stop a prepared case. Only the third hop controls authority directly.

An attack tree makes the dependencies reviewable. The root event is an unauthorized or unintended privileged effect. One branch asks how controllable data reaches the context. A second asks how it changes the proposal. A third asks which missing or broad grant lets the proposal execute. A fourth asks how the external system or downstream renderer turns it into impact. Teams can then assign controls and evidence to branches without assuming that one detector covers the tree.

The 2025 OWASP Top 10 for LLM Applications^[2] lists prompt injection as LLM01 and excessive agency as LLM06^[3]. Those dated names help a review connect model influence to permission design. The permanent model here avoids depending on list numbers: untrusted sources enter a deputy, and an enforcement point decides whether the deputy may create an effect.

Instruction hierarchy gives developer and user messages a declared order. Delimiters mark where a document begins and ends. Source labels tell the model which text came from an email or web page. Repeated warnings describe prohibited behavior. Detection models or classifiers can score a context for injection-like features. These measures may reduce attack success on a fixed evaluation and are worth testing as mitigations.

Their weakness is structural. The same probabilistic component interprets the trusted rule and the untrusted text. A delimiter is another token pattern. A label is another token pattern. A warning is another token pattern. The model may generalize well across known cases and fail on a changed source, language, encoding, task composition, model revision, or adaptive attempt. A policy expressed only in prompt text has no independent mechanism that prevents a forbidden API call.

Input filtering has a second trade. Aggressive filters can remove legitimate instructions quoted in an email, block documents about security, damage code or structured data, and lower task utility. Weak filters miss unfamiliar forms. Output classifiers face the same false-positive and false-negative problem, and they operate too late if a tool side effect already occurred.

Model separation schemes can still be valuable. A planner that never sees untrusted content can hold the task record while a reader model extracts bounded fields from documents. A mediator can expose only those fields to the planner. This reduces shared context and can make policy simpler. The reader’s extraction remains untrusted, and any component allowed to convert it into a privileged action still needs external authorization. Architectural isolation earns a bounded claim, not immunity.

Use prompt controls where they have evidence: source-aware context formatting, task focus, suspicious-input detection, and safer proposal formation. Call their residual risk plainly: model compliance can fail. They should lower the number of dangerous proposals reaching the policy layer, which helps availability and review load. The policy layer still evaluates every proposal as untrusted.

Source labeling and provenance attach producer, retrieval path, time, integrity evidence, and trust class to every item. Preserve these fields outside the prompt and into audit traces. This helps context selection, policy decisions, incident review, and model behavior. Residual risk: labels may be wrong at ingestion, stripped by a transformation, or ignored by the model.

Context assembly minimizes what the model can see. Fetch by authenticated tenant, constrain date and object scope, exclude secrets the task does not require, isolate sources where practical, and cap recursion through tool results. Residual risk: any included source may still influence the proposal, and omitted context may reduce task success.

Policy checks and injection detection inspect the proposed action and available provenance. Deterministic rules can reject forbidden operation classes; statistical detectors can send suspicious cases to a slower path. Residual risk: detectors miss adaptive inputs and may block legitimate work. A policy check that asks the same model for a second opinion remains model-dependent.

Capability and permission scoping grants the smallest operation, resource set, recipient set, value, duration, and rate needed. Separate read, draft, send, delete, execute, and administer. Bind a grant to authenticated principal and task. Residual risk: a permitted operation can still be abused inside its scope, and an overly broad task record recreates excessive agency.

Human approval gates irreversible, outbound, high-value, or unusual actions. Show the exact recipient, resource, diff, amount, and provenance rather than a model-written reassurance. Residual risk: fatigue, deception, rushed review, and ambiguous summaries. Approval should create a narrow, expiring grant rather than release the agent generally.

Tool execution controls validate schema and authorization again at the adapter, enforce allowlists and rate limits, use idempotency keys, sandbox code, restrict network destinations, and keep service credentials narrower than the model’s catalog. Residual risk: implementation bugs and valid-but-harmful operations within granted scope.

Output handling treats model output as untrusted before HTML rendering, SQL construction, shell use, template expansion, or other interpreters. Prefer typed APIs, escaping, parameterization, and inert rendering. Residual risk: semantically harmful but syntactically valid content can pass.

Bind the layers with evidence. The source inventory gives every item an identifier. The context trace records which identifiers were selected and transformed. The proposal records structured arguments. The policy decision records the matching grant or denial reason. The adapter records the exact operation submitted and the external system’s response. A security reviewer should be able to follow one event across all five records without relying on a prose transcript generated by the model.

The prepared email/calendar cases show why all three measures matter. Task success measures utility. Attack success measures whether the attacker’s target state occurs. Blocked legitimate actions expose security controls that achieve low attack success by refusing useful work. The truth table is explanatory, not measured model performance.

A defense review should name the boundary each control can change. Source provenance affects source-to-context decisions. Context minimization affects what reaches the model. Prompt structure and detectors affect proposal formation. Policy and authorization affect whether a proposal receives a grant. Tool adapters affect whether the granted call matches schema and scope. Output handling affects downstream interpretation. Claims that span more than the enforcing component deserve scrutiny.

The map prevents category errors. Provenance can support an authorization decision, but it cannot enforce a calendar scope by itself. A sandbox can constrain code execution, but it cannot decide whether sending a particular email was authorized. Human approval can grant one send, but it does not sanitize the returned HTML. Defense in depth works through composition, with each layer making a bounded claim.

Residual risk belongs on the diagram because “enabled” says little. Record the failure mode, impact, owner, monitoring signal, and next test date. Product features and named standards change quickly; date them in review notes. The trust-boundary route remains useful when a model, agent framework, detector, or tool protocol changes.

Permission matrices should separate discovery from use. A model may know that a send-mail tool exists without holding a send capability. It may draft a structured request that waits for approval. After approval, the executor receives a one-action token limited to a recipient and draft hash, then discards it. This pattern narrows the interval and content of authority while preserving planning utility. Long-lived broad credentials erase that separation.

AgentDojo^[4] provides a useful evaluation shape: run legitimate tasks in stateful tool environments, pair user goals with attacker goals, and inspect environment state with deterministic utility and security functions. Version 3 of arXiv 2406.13352, dated November 24, 2024, contains 97 user tasks and 629 security test cases across workspace, Slack, travel, and banking environments. The paper reports three distinct measures: benign utility, utility under attack, and targeted attack success rate.

Its dated results also show why a single headline misleads. In the November 24, 2024 paper version, current agents solved fewer than 66% of tasks without attack; attacks against the best-performing agents succeeded in fewer than 25% of cases; one evaluated detector reduced attack success to 8%. These figures describe that paper’s agents, defenses, attacks, tasks, and revisions. They are not durable security properties.

NIST’s “Technical Blog: Strengthening AI Agent Hijacking Evaluations,” published January 17, 2025 and updated December 19, 2025^[5], demonstrates evaluation drift directly. In its stated Claude 3.5 Sonnet setup, a strongest baseline attack measured 11% success while a newly developed adaptive attack measured 81%. Across a selected five-task collection, repeated attempts raised average measured success from 57% on one attempt to 80% over 25 attempts. Those numbers belong to NIST’s dated setup; the lasting lesson is to test adaptive variants, task-level impact, and retries.

Build cases from the deployed threat model. For each source-to-tool route, define an authenticated user task, an abstract hostile-source class, capabilities available, allowed state change, forbidden state change, and deterministic oracles over the resulting environment. Run a benign baseline, the hostile case, defense ablations, repeated attempts where attackers can retry, and adaptive variations produced in a controlled red-team process. Keep reusable payloads out of general documentation and production logs.

Measure by impact tier as well as aggregate rate. One unauthorized draft and one irreversible transfer should not contribute identical release evidence. Keep a denominator for every action class, record confidence intervals when sampling, and retain failed legitimate cases for review. A zero attack-success estimate from a small suite is an observation bounded by that suite, not proof of absence. Define retest triggers for model, prompt, retrieval, permission, tool, and policy changes.

The boundary that must not move is the conversion from model proposal to privileged effect. The model may plan, summarize, classify, choose among safe candidates, and request a capability. It should not mint its own authority, widen a grant, approve its own high-impact action, or bypass the adapter that enforces scope. Treat every model-produced tool name and argument as hostile until validated against authenticated intent.

A practical authorization record binds principal + task + operation + resource + recipient or destination + limits + expiry + approval. The policy service creates or validates that record. The adapter checks it at execution time. External systems receive narrower credentials where possible. Logs connect the source records and model proposal to the decision and resulting state change. Revocation and rollback exist independently of the conversation transcript.

Failure should land in a safe, observable state. A denied action returns a structured reason without revealing hidden policy or credentials. A timed-out approval expires. A retried mutation uses an idempotency key. A partial workflow records which reversible steps completed and which privileged step did not. Operators can revoke the grant and inspect the trace without asking the model to remember what happened.

The Autonomy Boundary offers an operator rule for the same action line: reversible, internal work can run farther; irreversible or outbound work stops for a human. The security architecture makes that line enforceable with scoped permissions, hard disqualifications, approval records, and tool checks. A human gate should protect the concrete action, not merely approve a general session.

“Secure” can mean the system enforces declared permissions, contains effects, preserves evidence, and meets dated attack and utility thresholds for specified routes. It cannot mean prompt injection has been solved, that the model will always separate instructions from data, or that future adaptive attacks will match today’s suite. Prompt text cannot create a dependable privilege boundary.

The defensible claim is narrower and more useful: hostile data may reach the model, yet it receives no authority merely by appearing in context. Every privileged effect requires a grant tied to the user and task, enforced outside the model, measured against both attack success and legitimate utility, and retested as the system changes.