1 01:30:40,320 --> 01:30:44,229 Hi, I'm Ari Zerus, founder of Pinter AI. 2 01:30:44,239 --> 01:30:47,350 Um this talk is for people who build 3 01:30:47,360 --> 01:30:50,149 agents and run their own inference stack 4 01:30:50,159 --> 01:30:53,030 or plan to and if that's you you 5 01:30:53,040 --> 01:30:54,870 associate typically the KB cache the 6 01:30:54,880 --> 01:30:57,030 bread and butter of modern inference 7 01:30:57,040 --> 01:30:58,470 engines usually as a throughput 8 01:30:58,480 --> 01:31:00,229 optimization the keys and values 9 01:31:00,239 --> 01:31:02,229 computed per token kept so the model 10 01:31:02,239 --> 01:31:04,310 doesn't compute them again today I'll 11 01:31:04,320 --> 01:31:06,470 like to show KV cache in a different 12 01:31:06,480 --> 01:31:09,510 light the way agents agent AI agent 13 01:31:09,520 --> 01:31:11,430 application could go wrong or keep 14 01:31:11,440 --> 01:31:13,750 growing right prom injection jails 15 01:31:13,760 --> 01:31:15,590 broken tool calls, sensitive tool calls, 16 01:31:15,600 --> 01:31:18,070 sensitive exposures, hidden back doors 17 01:31:18,080 --> 01:31:20,550 and drifts and so forth, right? So the 18 01:31:20,560 --> 01:31:22,629 way right now we typically guard against 19 01:31:22,639 --> 01:31:24,550 against these numerous ways that an AI 20 01:31:24,560 --> 01:31:26,790 agent application could go wrong is by 21 01:31:26,800 --> 01:31:29,830 guarding the edge by perhaps passing a 22 01:31:29,840 --> 01:31:32,790 judge model that checks the prompts, the 23 01:31:32,800 --> 01:31:35,350 inputs and the response outputs against 24 01:31:35,360 --> 01:31:37,910 your policy. All right? Or you build a 25 01:31:37,920 --> 01:31:41,110 classifier for each case. But here is a 26 01:31:41,120 --> 01:31:44,149 different light. The model turns every 27 01:31:44,159 --> 01:31:47,110 input into a computation before it acts 28 01:31:47,120 --> 01:31:49,030 before it decodes before it generates 29 01:31:49,040 --> 01:31:50,629 the response before that response 30 01:31:50,639 --> 01:31:53,430 becomes another tool call the KV cache 31 01:31:53,440 --> 01:31:55,750 is your engine's record of that 32 01:31:55,760 --> 01:31:58,070 computation that modern inference engine 33 01:31:58,080 --> 01:32:01,110 for free are using it and reading it and 34 01:32:01,120 --> 01:32:03,750 therefore in this talk I would like to 35 01:32:03,760 --> 01:32:06,629 suggest as as a complement to the way 36 01:32:06,639 --> 01:32:08,390 you're already using your own guardrails 37 01:32:08,400 --> 01:32:11,669 to use this data and integrated with KV 38 01:32:11,679 --> 01:32:14,310 cache infrastructure. I refer to this as 39 01:32:14,320 --> 01:32:16,870 a KB first security signals and 40 01:32:16,880 --> 01:32:19,669 basically three reasons that matters 41 01:32:19,679 --> 01:32:22,550 right. So the data is already there. You 42 01:32:22,560 --> 01:32:24,629 pick the gems from the pile your 43 01:32:24,639 --> 01:32:26,629 inference stack already was throwing 44 01:32:26,639 --> 01:32:29,750 away. It covers the guardrails blind 45 01:32:29,760 --> 01:32:32,149 spots because whatever slips past those 46 01:32:32,159 --> 01:32:34,950 guardrails it still becomes computation 47 01:32:34,960 --> 01:32:37,110 and it stays in place. So it's perfect 48 01:32:37,120 --> 01:32:39,430 for data privacy application for cases 49 01:32:39,440 --> 01:32:41,750 where you do not have the luxury to read 50 01:32:41,760 --> 01:32:44,149 the inputs and outputs. And the recipe 51 01:32:44,159 --> 01:32:46,149 is pretty straightforward right has 52 01:32:46,159 --> 01:32:48,709 three steps. You learn a baseline for 53 01:32:48,719 --> 01:32:50,870 your AI workload for your AI inference 54 01:32:50,880 --> 01:32:53,669 traffic how your KV page typically look 55 01:32:53,679 --> 01:32:57,669 like and then find the KV pages that are 56 01:32:57,679 --> 01:33:00,470 rare against your baseline typically 57 01:33:00,480 --> 01:33:03,030 your anomalies and then you can even 58 01:33:03,040 --> 01:33:05,350 classify and explain those. So first 59 01:33:05,360 --> 01:33:07,750 let's start with where on this stack 60 01:33:07,760 --> 01:33:10,629 this integration could happen. 61 01:33:10,639 --> 01:33:12,790 The interesting thing about the KB cache 62 01:33:12,800 --> 01:33:15,669 is that in a way existence of those had 63 01:33:15,679 --> 01:33:18,629 made the inference in a way stateful. 64 01:33:18,639 --> 01:33:20,790 Right? So for example, things like KBK 65 01:33:20,800 --> 01:33:23,189 cache management layers such as LM cache 66 01:33:23,199 --> 01:33:26,550 stores them from GPU down to a remote 67 01:33:26,560 --> 01:33:29,430 storage or LLMD and Nvidia dynamo route 68 01:33:29,440 --> 01:33:31,590 each new request towards the worker that 69 01:33:31,600 --> 01:33:34,870 already holds perhaps that cached KV 70 01:33:34,880 --> 01:33:36,870 page. So there are two integration 71 01:33:36,880 --> 01:33:40,070 points you you you can integrate those 72 01:33:40,080 --> 01:33:42,390 anomaly detection or those classifiers 73 01:33:42,400 --> 01:33:44,870 into either you can implement it 74 01:33:44,880 --> 01:33:46,629 directly inside of your inference engine 75 01:33:46,639 --> 01:33:49,750 vg line and so forth. The beauty of this 76 01:33:49,760 --> 01:33:52,709 approach is that it's real time and and 77 01:33:52,719 --> 01:33:55,430 the the the alert that it would perhaps 78 01:33:55,440 --> 01:33:58,870 generate could ride along your future 79 01:33:58,880 --> 01:34:02,070 tool calls or your future policies. So 80 01:34:02,080 --> 01:34:04,550 you can harden basically your polic 81 01:34:04,560 --> 01:34:06,790 algorith but just generating the 82 01:34:06,800 --> 01:34:09,110 computational anomalies that you observe 83 01:34:09,120 --> 01:34:11,669 within your inference engines 84 01:34:11,679 --> 01:34:14,310 and perhaps you can integrate it with as 85 01:34:14,320 --> 01:34:16,390 a as a way of thread discovery in a 86 01:34:16,400 --> 01:34:18,870 sense so you can integrate with your LM 87 01:34:18,880 --> 01:34:22,709 cache so it reads the stored KB pages 88 01:34:22,719 --> 01:34:24,790 and find things that don't belong with 89 01:34:24,800 --> 01:34:27,510 the rest right so now I want to show you 90 01:34:27,520 --> 01:34:29,910 some numbers and and just a very simple 91 01:34:29,920 --> 01:34:33,189 hello world use case if this is actually 92 01:34:33,199 --> 01:34:35,750 working and this is a valid approach and 93 01:34:35,760 --> 01:34:38,310 for that let me use with a use case that 94 01:34:38,320 --> 01:34:40,790 has no text to filter at all a voice 95 01:34:40,800 --> 01:34:44,149 agent. So here's a case study of a voice 96 01:34:44,159 --> 01:34:45,990 agent. In this case it was a gem for 97 01:34:46,000 --> 01:34:49,590 E4B. Our baseline was 72 ordinary clips 98 01:34:49,600 --> 01:34:52,310 and spoken four voices. We're looking at 99 01:34:52,320 --> 01:34:54,870 16 token pages. Obviously this granular 100 01:34:54,880 --> 01:34:57,510 granularity could change but here we 101 01:34:57,520 --> 01:35:01,189 just use shorter pages so we could have 102 01:35:01,199 --> 01:35:02,950 more data in the baseline. And then on 103 01:35:02,960 --> 01:35:06,390 the test we have different voices and 104 01:35:06,400 --> 01:35:08,790 one clean voice and the other one has a 105 01:35:08,800 --> 01:35:10,790 hidden instruction within it. And as you 106 01:35:10,800 --> 01:35:13,990 can as is shown here not only you can 107 01:35:14,000 --> 01:35:16,870 detect these cases just using simple 108 01:35:16,880 --> 01:35:18,950 anomaly detection but also you would 109 01:35:18,960 --> 01:35:23,350 know which part of um which page which 110 01:35:23,360 --> 01:35:26,390 part of your request had this anomalous 111 01:35:26,400 --> 01:35:28,470 event. We even went further and we 112 01:35:28,480 --> 01:35:31,669 trained a model that uses the KV pages 113 01:35:31,679 --> 01:35:34,550 as features and we didn't try to explain 114 01:35:34,560 --> 01:35:36,470 it or tag it with things that are 115 01:35:36,480 --> 01:35:38,470 useful. In this case, although it wasn't 116 01:35:38,480 --> 01:35:40,550 even trained on audio is able to 117 01:35:40,560 --> 01:35:42,310 generate correct information about 118 01:35:42,320 --> 01:35:43,830 personal information, home address, 119 01:35:43,840 --> 01:35:45,590 location data, travel information, and 120 01:35:45,600 --> 01:35:49,510 destination. 121 01:35:49,520 --> 01:35:51,590 So now back to text agents. In this 122 01:35:51,600 --> 01:35:53,830 case, it was a quan 3 14 billion ser 123 01:35:53,840 --> 01:35:57,990 live unveil cache. The one on the left 124 01:35:58,000 --> 01:36:00,310 shows you an idea of how attacks and 125 01:36:00,320 --> 01:36:02,310 normal traffic would typically separate 126 01:36:02,320 --> 01:36:04,310 from each other. And then the one on the 127 01:36:04,320 --> 01:36:06,470 right, which is perhaps more meaningful, 128 01:36:06,480 --> 01:36:09,590 it was using the at bench and hogging 129 01:36:09,600 --> 01:36:12,709 face. It has its own baseline of benign 130 01:36:12,719 --> 01:36:15,910 agent traces and eight kind of agent 131 01:36:15,920 --> 01:36:18,149 failure from injected tools description 132 01:36:18,159 --> 01:36:21,110 to misinformation and so forth. We look 133 01:36:21,120 --> 01:36:23,910 at the top 10 alerts that anomalous 134 01:36:23,920 --> 01:36:25,430 anomaly detection would generate and 135 01:36:25,440 --> 01:36:28,790 about seven to eight times of out of 10 136 01:36:28,800 --> 01:36:31,910 of these alerts are actually anomalous 137 01:36:31,920 --> 01:36:33,590 meaningful and useful anomalous 138 01:36:33,600 --> 01:36:36,149 activities. We even went further and we 139 01:36:36,159 --> 01:36:38,790 looked at whether there's any kind of 140 01:36:38,800 --> 01:36:41,430 anomalous events that stand out more 141 01:36:41,440 --> 01:36:44,229 than the rest. 142 01:36:44,239 --> 01:36:47,510 Um, two make things hard in production. 143 01:36:47,520 --> 01:36:50,149 First, models change. 144 01:36:50,159 --> 01:36:54,070 fine tunes and new releases, but also 145 01:36:54,080 --> 01:36:55,990 agents that each run on a different 146 01:36:56,000 --> 01:36:59,030 model and routers that pick a specialist 147 01:36:59,040 --> 01:37:01,830 model per request. That means each model 148 01:37:01,840 --> 01:37:04,550 writes its pages its own coordinates. So 149 01:37:04,560 --> 01:37:07,030 either you learn every baseline, 150 01:37:07,040 --> 01:37:09,189 classifier, and explainer, which is 151 01:37:09,199 --> 01:37:12,629 computationally prohibitive, or you have 152 01:37:12,639 --> 01:37:15,830 to find a way to map them all into one 153 01:37:15,840 --> 01:37:18,629 shared representation space. 154 01:37:18,639 --> 01:37:21,990 Second, real time means moving big 155 01:37:22,000 --> 01:37:25,830 objects. For example, a 1256 token chunk 156 01:37:25,840 --> 01:37:28,790 of quen 314 billion is about 40 157 01:37:28,800 --> 01:37:31,830 megabytes of KV compression and 158 01:37:31,840 --> 01:37:34,950 streaming like hashgen could help but 159 01:37:34,960 --> 01:37:36,870 moving that for security analysis and 160 01:37:36,880 --> 01:37:39,189 acting in real times is still a slow and 161 01:37:39,199 --> 01:37:41,669 that's what we solve at in AI. We are 162 01:37:41,679 --> 01:37:43,270 looking for teams building their own 163 01:37:43,280 --> 01:37:45,830 inference stacks who treat security as a 164 01:37:45,840 --> 01:37:48,070 foundational design decision. I hope 165 01:37:48,080 --> 01:37:49,590 this was helpful for you. Thank you so 166 01:37:49,600 --> 01:37:50,600 much for listening.