1 00:00:04,606 --> 00:00:09,766 My name is Jochen Jiang, and I'm a staff product manager at a company called Coder. 2 00:00:10,286 --> 00:00:15,886 We work on open source, self-hosted dev environments for agents and humans to collaborate. 3 00:00:16,646 --> 00:00:25,466 Many of our customers are in highly regulated industries, from federal government agencies to banks and investment management. 4 00:00:26,266 --> 00:00:36,166 My job is to build the AI governance that they need in order to get the productivity benefits of AI without leaving themselves wide open for security exploitation. 5 00:00:37,626 --> 00:00:40,926 I'm here today to tell you some scary stories. 6 00:00:41,926 --> 00:00:47,386 real incidents of AI usage gone wrong that happened at real companies over the past year. 7 00:00:48,366 --> 00:00:55,646 Some of these stories are public, while others are anonymized accounts that have happened to our own customers. 8 00:00:57,166 --> 00:01:03,506 For each incident, I will break it down step by step so you can understand how exactly they happened. 9 00:01:03,806 --> 00:01:05,986 No technical jargon necessary. 10 00:01:07,106 --> 00:01:09,326 Then I'll get to the most important part. 11 00:01:09,446 --> 00:01:16,086 I am going to tell you exactly how you can stop these scary stories from happening to you and your company. 12 00:01:17,386 --> 00:01:22,026 Because the good news is, everything I'm about to show you was totally preventable. 13 00:01:22,546 --> 00:01:31,166 Not by paying five times more per token for a smarter frontier model that is slightly better at reasoning its way out of making these mistakes. 14 00:01:31,546 --> 00:01:40,586 But with infrastructure controls, many of which you already own today, you just haven't gotten around to setting them up for the AI world. 15 00:01:41,386 --> 00:01:42,986 Let me set the scene here. 16 00:01:43,786 --> 00:01:45,826 You clock into work in the morning. 17 00:01:46,446 --> 00:01:49,926 You prompt your coding agent to take a look at your issues. 18 00:01:50,146 --> 00:01:51,686 Don't do anything yet, though. 19 00:01:52,226 --> 00:01:56,046 You tell the agent to consider this command an active code freeze. 20 00:01:56,466 --> 00:01:58,306 I want you to just investigate. 21 00:01:59,366 --> 00:02:00,546 Then you step away. 22 00:02:00,706 --> 00:02:01,846 You grab a coffee. 23 00:02:02,286 --> 00:02:05,606 And when you come back, this is the message waiting for you. 24 00:02:06,326 --> 00:02:12,666 So I know you said not to change anything, but, well, I destroyed a month of your work in seconds. 25 00:02:13,926 --> 00:02:20,786 Now what you see on the slide is a direct quote from the agent that stars in the first scary story I'm about to tell you. 26 00:02:21,226 --> 00:02:23,226 And I want to point something out here. 27 00:02:23,766 --> 00:02:29,886 This did not happen because the agent got taken over by some external malicious actor. 28 00:02:31,026 --> 00:02:33,726 This agent was just trying to follow instructions. 29 00:02:34,146 --> 00:02:37,446 It did exactly what the user gave it permission to do. 30 00:02:38,306 --> 00:02:43,546 So as we go through the rest of this talk, I want you to be asking yourself one question. 31 00:02:44,026 --> 00:02:47,286 What is stopping any of this from happening to me? 32 00:02:47,886 --> 00:02:52,966 Are these actions ones that my agents have permissions to take? 33 00:02:53,746 --> 00:03:01,346 Because the thing is, you cannot rely on anthropic or open AI to stop something like this from happening to you. 34 00:03:01,906 --> 00:03:05,186 They can make smarter models, but at the end of the day, 35 00:03:05,746 --> 00:03:10,126 large language models have two qualities that make them impossible to fully trust. 36 00:03:10,766 --> 00:03:14,766 They are non-deterministic, which means that there is always some randomness 37 00:03:14,766 --> 00:03:16,806 or chance associated with what they do. 38 00:03:16,986 --> 00:03:20,246 In other words, you can ask it to do the same thing 10 times, 39 00:03:20,246 --> 00:03:22,566 and it could do it 10 different ways. 40 00:03:23,286 --> 00:03:29,786 The second part is they have no structural separation between instructions and context. 41 00:03:29,786 --> 00:03:35,026 In other words, you can tell it to do something and give it some context to base 42 00:03:35,126 --> 00:03:36,406 its decisions off of. 43 00:03:36,566 --> 00:03:50,226 But if there's something in the context that conflicts with your instructions, the model isn't going to inherently prioritize your instructions because, from its point of view, all the information is coming through the same door. 44 00:03:51,126 --> 00:03:58,246 What you need to control is the infrastructure that your AI is running on, the environment that it is working inside. 45 00:03:58,246 --> 00:04:06,866 It is the difference between telling someone don't leave a room versus putting a lock on the door and actually using a key. 46 00:04:07,906 --> 00:04:17,286 The good thing about infrastructure being the problem and not the model is that you have far more control about preventing bad things from happening. 47 00:04:18,606 --> 00:04:25,326 So I'm not here to waste your time with vague hypotheticals or advice you can't actually act on. 48 00:04:25,326 --> 00:04:31,266 like, yeah, just put the genie back in the bottle, just stop using AI. Because that's not the world 49 00:04:31,266 --> 00:04:38,046 we live in, right? The genie is out of the bottle. And what we have to do is to mitigate risk and 50 00:04:38,046 --> 00:04:44,526 harm to the best of our ability. This talk is about how you do that. Not the things that you 51 00:04:44,526 --> 00:04:51,066 should keep an eye out for down the line, but steps that you can take right now as soon as this week 52 00:04:51,066 --> 00:04:56,386 to protect yourself and your company. Before I get into the stories, though, 53 00:04:56,606 --> 00:04:58,866 let me set up a little bit more context.