# Read the inference state before the agent acts Source: https://www.youtube.com/watch?v=ZZ19zhYnDKY Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session. [01:30:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5440s) Hi, I'm Ari Zerus, founder of Pinter AI. [01:30:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5444s) Um this talk is for people who build [01:30:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5447s) agents and run their own inference stack [01:30:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5450s) or plan to and if that's you you [01:30:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5453s) associate typically the KB cache the [01:30:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5454s) bread and butter of modern inference [01:30:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5457s) engines usually as a throughput [01:30:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5458s) optimization the keys and values [01:31:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5460s) computed per token kept so the model [01:31:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5462s) doesn't compute them again today I'll [01:31:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5464s) like to show KV cache in a different [01:31:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5466s) light the way agents agent AI agent [01:31:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5469s) application could go wrong or keep [01:31:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5471s) growing right prom injection jails [01:31:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5473s) broken tool calls, sensitive tool calls, [01:31:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5475s) sensitive exposures, hidden back doors [01:31:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5478s) and drifts and so forth, right? So the [01:31:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5480s) way right now we typically guard against [01:31:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5482s) against these numerous ways that an AI [01:31:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5484s) agent application could go wrong is by [01:31:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5486s) guarding the edge by perhaps passing a [01:31:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5489s) judge model that checks the prompts, the [01:31:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5492s) inputs and the response outputs against [01:31:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5495s) your policy. All right? Or you build a [01:31:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5497s) classifier for each case. But here is a [01:31:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5501s) different light. The model turns every [01:31:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5504s) input into a computation before it acts [01:31:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5507s) before it decodes before it generates [01:31:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5509s) the response before that response [01:31:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5510s) becomes another tool call the KV cache [01:31:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5513s) is your engine's record of that [01:31:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5515s) computation that modern inference engine [01:31:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5518s) for free are using it and reading it and [01:32:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5521s) therefore in this talk I would like to [01:32:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5523s) suggest as as a complement to the way [01:32:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5526s) you're already using your own guardrails [01:32:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5528s) to use this data and integrated with KV [01:32:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5531s) cache infrastructure. I refer to this as [01:32:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5534s) a KB first security signals and [01:32:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5536s) basically three reasons that matters [01:32:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5539s) right. So the data is already there. You [01:32:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5542s) pick the gems from the pile your [01:32:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5544s) inference stack already was throwing [01:32:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5546s) away. It covers the guardrails blind [01:32:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5549s) spots because whatever slips past those [01:32:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5552s) guardrails it still becomes computation [01:32:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5554s) and it stays in place. So it's perfect [01:32:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5557s) for data privacy application for cases [01:32:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5559s) where you do not have the luxury to read [01:32:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5561s) the inputs and outputs. And the recipe [01:32:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5564s) is pretty straightforward right has [01:32:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5566s) three steps. You learn a baseline for [01:32:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5568s) your AI workload for your AI inference [01:32:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5570s) traffic how your KV page typically look [01:32:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5573s) like and then find the KV pages that are [01:32:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5577s) rare against your baseline typically [01:33:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5580s) your anomalies and then you can even [01:33:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5583s) classify and explain those. So first [01:33:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5585s) let's start with where on this stack [01:33:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5587s) this integration could happen. [01:33:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5590s) The interesting thing about the KB cache [01:33:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5592s) is that in a way existence of those had [01:33:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5595s) made the inference in a way stateful. [01:33:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5598s) Right? So for example, things like KBK [01:33:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5600s) cache management layers such as LM cache [01:33:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5603s) stores them from GPU down to a remote [01:33:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5606s) storage or LLMD and Nvidia dynamo route [01:33:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5609s) each new request towards the worker that [01:33:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5611s) already holds perhaps that cached KV [01:33:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5614s) page. So there are two integration [01:33:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5616s) points you you you can integrate those [01:33:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5620s) anomaly detection or those classifiers [01:33:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5622s) into either you can implement it [01:33:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5624s) directly inside of your inference engine [01:33:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5626s) vg line and so forth. The beauty of this [01:33:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5629s) approach is that it's real time and and [01:33:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5632s) the the the alert that it would perhaps [01:33:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5635s) generate could ride along your future [01:33:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5638s) tool calls or your future policies. So [01:34:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5642s) you can harden basically your polic [01:34:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5644s) algorith but just generating the [01:34:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5646s) computational anomalies that you observe [01:34:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5649s) within your inference engines [01:34:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5651s) and perhaps you can integrate it with as [01:34:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5654s) a as a way of thread discovery in a [01:34:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5656s) sense so you can integrate with your LM [01:34:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5658s) cache so it reads the stored KB pages [01:34:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5662s) and find things that don't belong with [01:34:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5664s) the rest right so now I want to show you [01:34:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5667s) some numbers and and just a very simple [01:34:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5669s) hello world use case if this is actually [01:34:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5673s) working and this is a valid approach and [01:34:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5675s) for that let me use with a use case that [01:34:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5678s) has no text to filter at all a voice [01:34:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5680s) agent. So here's a case study of a voice [01:34:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5684s) agent. In this case it was a gem for [01:34:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5686s) E4B. Our baseline was 72 ordinary clips [01:34:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5689s) and spoken four voices. We're looking at [01:34:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5692s) 16 token pages. Obviously this granular [01:34:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5694s) granularity could change but here we [01:34:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5697s) just use shorter pages so we could have [01:35:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5701s) more data in the baseline. And then on [01:35:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5702s) the test we have different voices and [01:35:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5706s) one clean voice and the other one has a [01:35:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5708s) hidden instruction within it. And as you [01:35:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5710s) can as is shown here not only you can [01:35:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5714s) detect these cases just using simple [01:35:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5716s) anomaly detection but also you would [01:35:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5718s) know which part of um which page which [01:35:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5723s) part of your request had this anomalous [01:35:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5726s) event. We even went further and we [01:35:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5728s) trained a model that uses the KV pages [01:35:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5731s) as features and we didn't try to explain [01:35:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5734s) it or tag it with things that are [01:35:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5736s) useful. In this case, although it wasn't [01:35:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5738s) even trained on audio is able to [01:35:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5740s) generate correct information about [01:35:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5742s) personal information, home address, [01:35:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5743s) location data, travel information, and [01:35:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5745s) destination. [01:35:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5749s) So now back to text agents. In this [01:35:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5751s) case, it was a quan 3 14 billion ser [01:35:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5753s) live unveil cache. The one on the left [01:35:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5758s) shows you an idea of how attacks and [01:36:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5760s) normal traffic would typically separate [01:36:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5762s) from each other. And then the one on the [01:36:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5764s) right, which is perhaps more meaningful, [01:36:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5766s) it was using the at bench and hogging [01:36:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5769s) face. It has its own baseline of benign [01:36:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5772s) agent traces and eight kind of agent [01:36:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5775s) failure from injected tools description [01:36:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5778s) to misinformation and so forth. We look [01:36:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5781s) at the top 10 alerts that anomalous [01:36:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5783s) anomaly detection would generate and [01:36:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5785s) about seven to eight times of out of 10 [01:36:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5788s) of these alerts are actually anomalous [01:36:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5791s) meaningful and useful anomalous [01:36:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5793s) activities. We even went further and we [01:36:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5796s) looked at whether there's any kind of [01:36:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5798s) anomalous events that stand out more [01:36:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5801s) than the rest. [01:36:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5804s) Um, two make things hard in production. [01:36:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5807s) First, models change. [01:36:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5810s) fine tunes and new releases, but also [01:36:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5814s) agents that each run on a different [01:36:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5816s) model and routers that pick a specialist [01:36:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5819s) model per request. That means each model [01:37:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5821s) writes its pages its own coordinates. So [01:37:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5824s) either you learn every baseline, [01:37:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5827s) classifier, and explainer, which is [01:37:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5829s) computationally prohibitive, or you have [01:37:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5832s) to find a way to map them all into one [01:37:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5835s) shared representation space. [01:37:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5838s) Second, real time means moving big [01:37:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5842s) objects. For example, a 1256 token chunk [01:37:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5845s) of quen 314 billion is about 40 [01:37:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5848s) megabytes of KV compression and [01:37:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5851s) streaming like hashgen could help but [01:37:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5854s) moving that for security analysis and [01:37:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5856s) acting in real times is still a slow and [01:37:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5859s) that's what we solve at in AI. We are [01:37:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5861s) looking for teams building their own [01:37:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5863s) inference stacks who treat security as a [01:37:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5865s) foundational design decision. I hope [01:37:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5868s) this was helpful for you. Thank you so [01:37:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=5869s) much for listening.