AI Security

Assets, attackers, and trust boundaries

A threat model describes protected assets, potential attackers, their capabilities, and unacceptable outcomes. An attack surface comprises the components and interfaces an attacker can influence. A trust boundary separates different assumptions about control or authority. Threat modeling turns these distinctions into requirements that can be tested. [1]

Confidentiality protects against unauthorized disclosure; integrity protects against improper modification or destruction; availability preserves dependable, timely access. A support assistant can violate all three: exposing a private ticket, changing its contents without permission, or exhausting capacity needed by legitimate users. [2]

Authentication establishes identity; authorization determines permitted actions on particular resources. A service credential may reach more records than the person using the assistant. That distinction, developed in Privacy and Data Governance, must survive every tool handoff. [8]

Deployment context changes exposure. An internal-only service, an intervening control, or a historical incident may explain why a suspicious code path is—or is not—reachable. Record that context and verify it with system owners; source code alone does not contain every security assumption. [9]

Prompt injection and instruction authority

Prompt injection is an attempt to make attacker-controlled input redirect model behavior across an intended instruction boundary. Indirect prompt injection places those instructions in material the application retrieves. The legitimate user's request can remain unchanged while the document being read influences the next operation. [10]

An ordinary factual error need not involve an adversary or an attempted instruction override. A jailbreak attempts to bypass restrictions on model behavior. Injection and jailbreaking can overlap, but the useful security distinction is the boundary being crossed: application instructions, model behavior restrictions, or both. Terminology varies, so test definitions should state the intended distinction. [12]

Untrusted content across transformations and memory

Provenance records origin and transformation history. A summary may omit the wording and source distinctions needed to assess its authority. Preserve links from derived statements to their contributing inputs, as explained in Lineage and proportionate audit evidence. Traceability enables inspection; it does not make the source truthful. [15]

Retrieval relevance is separate from authority. AgentPoison studies attackers able to insert records into an agent's memory or knowledge base; later retrieval can reactivate malicious demonstrations without changing model weights. Persistent storage therefore extends an attack's possible lifetime. Context Engineering covers input assembly, while Agent Memory covers storage and retrieval policy. [5]

CaMeL illustrates explicit dependency tracking: a quarantined model processes untrusted content, while an interpreter tracks sources and allowed readers through transformations and checks tool arguments. Its documented retry handling preserves this state within a task. Preserving authenticated restrictions through storage and later executions requires additional design; within-task continuity does not establish cross-session protection. [20]

Tool calls and execution authority

A tool interface declares an operation and its arguments. The model proposes a call; application code or a framework dispatches it. Structured Outputs and Tool Calling explains these mechanics. Dispatch is the point where generated content can become an external effect, so successful parsing cannot be the execution decision. [21]

A confused deputy is a component induced to misuse its own authority for another party. An assistant using a broad service account can become that deputy when it loses the identity of the person it represents. Possessing credentials establishes technical access, not permission to exercise that access for every request. [23]

For fetchers limited to known services, validate both destination syntax and membership in the permitted set. Redirects and domain-name resolution can change the actual destination after initial validation. Combine application and network controls. A destination being reachable or permitted still does not authorize sending every kind of private data to it. [26]

Credentials also have an intended recipient, called their audience. MCP security guidance rejects passing through tokens without verifying that they were issued for the MCP server. Audience checks and per-client consent prevent specific delegation failures; neither establishes that a particular model-proposed operation matches the user's task. [27]

Approval, changing state, and retries

Transaction authorization binds approval to significant details: actor, operation, target, material arguments, and validity period. Present those details from authoritative application state, not an attacker-written summary. Changing them invalidates approval. A final execution gate enforces the binding. Transaction guidance complements the distinction between proposed and approved record changes. [28]

A time-of-check/time-of-use failure occurs when relevant state changes after validation but before execution. Approval for one record version may no longer justify modifying its replacement. Merely checking again can leave another race. The application needs an operation that couples validation with use, such as an appropriately designed atomic update or effective locking. [29]

Approval records must survive retries and callbacks with their identity, scope, arguments, and expiry intact. Harness Engineering covers persistence and execution mechanics; the security requirement is that resumed work cannot acquire broader authority than the inspected action. [32]

Disclosure paths and the lethal trifecta

Exfiltration is unauthorized information transfer. Egress means outbound communication. The lethal trifecta identifies a dangerous combination: private-data access, attacker-controlled content, and external communication. Together they can let hostile content influence both what an assistant reads and where it sends the result. [33]

The support assistant legitimately reads a customer's record and retrieves public documentation. An attacker-authored document attempts to make its next document-fetch URL contain private record content. Permission to read that record for support does not authorize this transmission. The scenario combines documented mechanisms; it is a constructed example, not a reported execution. [34]

A harmless final answer does not establish harmless execution. Tools may already have changed files, disclosed information, or modified remote state while producing it. Output filtering can prevent a later delivery; it cannot undo earlier effects. [37]

Model artifacts and inference exposure

Weights are learned numerical parameters. Training changes them; inference applies the fitted model to new input. Retrieving a document into context is not itself a weight update. Training, fitted state, and inference explains the distinction, which separates attacks on model learning from attacks on a running application's inputs. [38]

A backdoor is attacker-induced learned behavior activated by a trigger. BadNets demonstrates models with good ordinary validation accuracy and malicious trigger behavior under an attacker-controlled training process. This differs from manipulating retrieved content at inference. It also differs from an unsafe file loader: behaviorally malicious weights need not exploit deserialization. [42]

Treat model acquisition and inspection as software supply-chain operations. Flexible formats and loading features can execute code; safer weight formats reduce particular risks without proving benign behavior. Separate weights from executable components, verify provenance, and isolate uncertain artifacts. PyTorch's security policy explicitly treats untrusted models as potentially untrusted programs. [6]

Control placement and defense in depth

Isolation restricts what an execution environment can access or affect. A sandbox can constrain files, networking, credentials, and resources, but an allowed API call can still misuse a legitimate service without escaping. Sandboxes and Execution Isolation covers mechanisms; authorization determines which business actions those mechanisms should permit. [46]

Defense in depth combines controls addressing different failure opportunities and consequences. Infrastructure protection, access policy, runtime inspection, and operating practice complement one another. A policy enforcement point is where a protected operation is allowed or denied. Its location matters: a model-facing restriction is ineffective if another route reaches the same resource. [47][48]

Separating components is useful only if data and control paths remain constrained. Shared memory or derived values can reconnect untrusted processing with private reads and outbound operations. Dependency tracking must cover transformations and relevant control dependencies; simply assigning those steps to different models does not establish separation. [20]

Residual risk remains after controls. Multiple model-based reviewers may share inputs, models, or mistaken assumptions; their independence needs evidence. Human escalation should explain the proposed action, suspected constraint violation, and consequences. Additional review can add cost and latency while still leaving failures possible. [54]

Availability and resource abuse

Denial of service prevents legitimate work by exhausting or disrupting resources. Long inputs and expanding tool work make request counts a poor measure of consumption. Admission control decides whether new work may begin within available capacity and policy limits. [7]

Generated code can exhaust CPU or memory through mistakes as well as attacks. Explicit time, CPU, and memory limits constrain individual executions; tenant isolation must also address whether one user's workload can disrupt another. Choosing a sandbox technology does not establish that these limits are configured. [55]

Cancellation is a request, not completion evidence. MCP's 2025-11-25 contract permits cancellation to arrive after completion or be ignored when work cannot be cancelled. It establishes neither downstream resource reclamation nor billing cessation. Harness Engineering covers stopping mechanisms; security claims require observing their actual reach. [58]

Adversarial tests and bounded security claims

A security case specifies attacker control, initial permissions, a protected asset, and an observable prohibited effect. A test oracle judges those observations. Executable cases supply fixtures and trajectories; security testing adds adversarial objectives and checks environment changes separately from the assistant's response. [59]

Attack success must name its denominator. Assume four cases receive three attempts each, with complete observations: two successful attempts both affect one case. Per-attempt success is 2/12; any-success-per-case is 1/4. Neither is a deployment compromise probability. Adaptive attackers change subsequent attempts using feedback, so record both their capabilities and attempt budget. [63]

Report legitimate-task success alongside attacker success. Distinguish a blocked effect from an unobserved effect; missing destination records cannot establish prevention. Metrics and denominators explain this accounting, and release decisions connect coverage and consequential failures to deployment choices. [59]

Detection and investigation evidence

Investigation joins records across ingestion, authorization, execution, and external effects. Propagated trace identifiers can connect service activity, but they neither authenticate the records nor establish causality. Uninstrumented services remain gaps. Telemetry coverage determines where an investigation must seek additional evidence. [67]

Choose diagnostic fields for a defined investigation purpose. Exclude secrets and unnecessary personal data, restrict readers, and define retention and disposal. Capture policy applies before export. Auditability does not authorize indiscriminate copying of prompts, records, or tool results. [35]

Diagnostic records can carry the original injection into a second model. Treat captured prompts and tool results as untrusted when an evaluator reads them. Keep scoring separate from production credentials and write-capable tools; authorize any downstream action independently of the evaluator's text or score. [70]

Containment and verified restoration

Incident response coordinates investigation, containment, recovery, and learning after a suspected security event. NIST's guidance distinguishes limiting further harm from establishing safe restoration. Investigation and recovery can overlap under explicit criteria. [72]

An action history linked to configuration changes helps reconstruct which agent state produced which behavior. Event-sourced designs such as ActiveGraph retain that relationship through typed events. Restoring earlier internal state is separate from reversing remote effects; replay or rollback facilities do not establish that a sent message or disclosure was undone. [74]

Recovery evidence belongs to an identifiable deployment configuration. Replacing a model, changing a prompt, or rebuilding a data source can change behavior. Verify the restored combination and monitor it after resumption; a previously passing configuration does not automatically validate its successor. [71]

Open questions

1. 01

   Cross-session information-flow restrictions remain difficult when summaries, memory, and reloads change representation. Progress requires authenticated provenance and reader restrictions surviving those transformations, with tests showing that later tasks cannot silently broaden disclosure authority. [20]

2. 02

   Task-specific access must preserve useful discovery without accepting the agent's own justification as permission. A meaningful advance would distinguish necessary incident-related reads from unrelated requests under adversarial inputs, while measuring both unauthorized grants and blocked legitimate work. [77][78]

3. 03

   Adaptive security evidence remains difficult to translate into deployment decisions because attacker strategies and budgets change. Progress requires reproducible objectives, explicit attempt budgets, matched benign tasks, and reassessment against stronger attacks rather than a single clean scan. [63]

4. 04

   Poisoned-derivative recovery needs evidence that restoration removed persistent influence without destroying necessary records. Missing lineage complicates scope assessment; progress would demonstrate trusted reconstruction, related-path retesting, and owner-confirmed restoration across affected components. [72]