Your KV Cache Can Spot Attacks on Your AI Agents - Ari Azarafrooz, Intrinsec AI

An inference engine keeps the keys and values it computes for each token so it can reuse them as it generates a response. For a team running its own inference stack, that key-value (KV) cache offers a second use. Before the agent generates a response that might become a tool call, the engine has already recorded part of the computation that led to it.

A judge model can check prompts and responses against a policy. A team can also train a classifier for each failure it knows to look for, from prompt injection to sensitive data exposure. The cache gives those guardrails another source of evidence. An input that slips through a filter still affects the model’s computation. A detector can examine that computation in place, even when the team cannot read raw inputs or outputs.

The unit of comparison is a page, a block of cached keys and values covering a span of tokens. The detector first learns what pages look like in ordinary traffic for a particular workload. It then flags pages that are rare against that baseline. Rarity tells an engineer where to look. A classifier or explainer can take the further step of describing what happened there.

Caching makes inference stateful in a practical sense. A page can outlive the computation that produced it, and its location matters for the next request. LMCache can store pages across tiers extending from GPU memory to remote storage. Cache-aware routing systems such as llm-d and NVIDIA Dynamo can direct a new request toward a worker that already holds a relevant page.

Where should a detector read those pages? The choice depends on when someone needs the result.

• Inside the inference engine. A detector integrated with an engine such as vLLM or SGLang could raise an alert while the request is running. The alert could accompany a subsequent tool call, giving the guardrail that evaluates the call evidence about the computation that preceded it.
• At the cache manager. A detector integrated with LMCache could scan stored pages for ones that stand out from ordinary traffic. This gives an investigator a way to discover suspicious activity across retained requests, including ones they had no prior reason to inspect.

The voice-agent experiment started with audio and no input text to filter. The team built its baseline from 72 ordinary clips spoken in four voices. It divided the cache into 16-token pages. Choosing short pages gave the detector more examples of ordinary computation from the same 72 recordings.

Could the detector tell an engineer where to look within a recording? The test used different voices and compared a clean clip with a voicemail containing a hidden spoken instruction. In the voicemail, simple anomaly detection flagged an unusual KV page. Because that page covered a particular span of the request, the team could narrow its inspection to the part of the recording associated with the anomaly, without first transcribing the instruction.

The team then trained a second model to use KV pages as features and assign descriptive tags. Although this tagger had received no audio training, Azarafrooz reports that it correctly identified categories including personal information, home address, location, travel information and destination in the example. That gave an investigator two clues from the same cache. The unusual page located the event, and the tags described the kind of information present.

For text agents, the team examined traffic from a live 14-billion-parameter model. A comparison of attack traffic with ordinary traffic illustrated how the two tended to separate. A benchmark then posed a question an engineer could use to budget review time. If you inspect only the ten highest-ranked alerts, how often do you find something worth investigating?

The benchmark supplied its own baseline of benign agent traces and eight kinds of agent failure, including injected tool descriptions and misinformation. Of the detector’s top ten alerts, about seven or eight pointed to activity the team judged meaningfully anomalous. That measures the yield at the top of the review queue. Estimating what fraction of all failures the detector caught would also require counting failures outside those ten alerts.

Each model writes keys and values in its own coordinates. A detector familiar with one model’s ordinary traffic therefore needs a way to interpret another model’s pages. Fine-tunes and new releases create this problem over time. Agents running different models, or a router choosing a specialist for each request, can create it within a single deployment. Relearning the baseline, classifier and explainer for every model can become computationally prohibitive. Azarafrooz proposes mapping their pages into a shared representation so the same analysis can apply across models.

Real-time analysis has a second cost. The engine has already computed the keys and values, but transferring a chunk of KV data to a detector takes time. Compression could reduce how much data has to move, and streaming could let analysis begin as the data arrives. For an alert meant to inform a tool-call decision, the deadline is the decision itself. The detector has to receive the relevant data, analyze it and deliver the alert while the guardrail can still act.