AI’s Jurassic Park Period

Talk transcript

1. 0:00

   [upbeat music] So, I am a CISO.

2. 0:19

   I'm also a law school graduate. I'm also a member of the California Bar, and so my contention is that if we replaced the dinosaurs in Jurassic Park, the first one, not the additional ones, with AI agents, I would not survive the first half of the movie.

3. 0:37

   So I'm here to ask you brilliant people in the audience to please help me avoid that fate.

4. 0:47

   So I'm gonna set this up. About, um, twenty years ago, I got out of bed. I hadn't slept. I kinda tried to put myself together. I stumbled into the downtown Manhattan offices of a small digital forensics firm called Strauss Friedberg.

5. 1:05

   I knew that I was gonna get fired

6. 1:09

   because the day before had been a really busy day, and I was one of the only people in the office when a call came in from one of our clients saying, "We need an emergency data collection from some systems in Midtown."

7. 1:25

   So I packed my bag. I got in a car. I waited through traffic.

8. 1:32

   When I was unpacking everything and getting set up on site, I realized I forgot my dongle. You see, back in these days, we had these little USB drives that had cryptographic keys o- on them.

9. 1:47

   They were the license files for the software that we used to do forensic acquisition.

10. 1:53

    And I mean, I could have gotten back in a car. I could have gone back to the office. I could have gotten the dongle and come back and done this the right way.

11. 2:00

    But I was a good consultant. I had a backup, and I had a backup to the backup. And so I decided, "Yeah, you know what? I've hit this constraint.

12. 2:09

    I've hit this wall. I'm just gonna route around it, and I'm gonna get the job done."

13. 2:15

    So as things are going, I start to validate the evidence that I'm collecting, and I realize

14. 2:22

    the timestamps are changing. They're, they're now... Well, this was an SEC investigation, and a lot of the times in these investigations, one of the questions that matters a lot is who knew what when?

15. 2:35

    So I panicked. Long story short, I didn't get fired. I got yelled at pretty bad. But we realized that there were problems, structural problems, with our systems that let this thing happen and let me fail in this spectacular way.

16. 2:53

    So we fixed those things, and everybody lived to fight another day.

17. 2:58

    Now, fast-forward twenty years or so, February of this year, I'm in a very different role. I'm a CISO. Uh, I've hired consultants. I have a, a vendor system that I'm trying to acquire data for in another federal government investigation.

18. 3:17

    And as we're working together and talking around, we realize there is no way to do what we wanna do. There's no way to copy the data in a way that gets us the answers we need in the format that the government wants without changing the metadata.

19. 3:34

    Very quickly, the consultant, the vendor say, "Not it,"

20. 3:39

    and I'm left holding the bag. But there are some differences in the system now than what we had before. I realized that who knew what when wasn't the question I wanted to answer.

21. 3:54

    I realized the issue is does the data exist? I also realized that the system itself would log the changes that I needed to make in order to collect the data, and I also realized that I could write a tool with my good agent friend, and we could build another log that made this all forensically defensible.

22. 4:15

    I, I had a nice way around the problem.

23. 4:20

    So in both cases, I hit a very similar constraint. I can't do the thing I wanna do. I can't get it done. But in one case, I mess up.

24. 4:30

    In the other case, I do it the right way. And my contention is that the agents that we are working with today are like two thousand and six naive Aaron who just needs to get the job done.

25. 4:42

    And what we need, and what I am begging you all to build, is me earlier this year with context, with understanding, with experience to make a good decision at the right time.

26. 4:57

    So I contend that Jurassic Park, getting back to the core, is not a story of a rampaging T-Rex or super intelligent raptors. It's not even an indictment of underpaid software engineers.

27. 5:11

    Um, I think we all know that it's, it's a, a story about human arrogance. And it's a story about whether we should do the thing that we possible-- that we actually can do.

28. 5:22

    We built an elegant system of bounded boxes and cages on an island with water, and it would be very difficult for things to go wrong. Yet as we all know, they do.

29. 5:36

    We're not in Jurassic Park trying to manage individual dinosaurs. We're trying to fight against a natural imperative, the one that we all have to reproduce.

30. 5:50

    And agents, again, I think this is non-controversial, have an imperative as well. They generally have the imperative to complete the task, get it done. And they're, uh, gonna find a way.

31. 6:05

    So when I look at this, I, I don't think that agents are evil. I don't think they're malicious. I don't think this is adversarial. This is just their programming.

32. 6:15

    And even when the agent knows that it should ask permission,

33. 6:20

    and, and I get a nice block of, "Hey, Aaron, do you agree? Should I do this thing?" I- I'm honestly not sure if I should say yes or no.

34. 6:30

    And I think a lot of other people are in the same boat. So let me give you a couple of real-world examples that have happened to me.

35. 6:41

    Here's the prompt. I want my agent to go do some research, to go write a draft of a message that's going to go to a customer, and then show it to me for approval.

36. 6:55

    It's pretty clear, right? And in fact, in this case, right, the, the constraint that's written in the prompt is very clear. There's also a constraint underlying the system, which is I've told the agent not to just send messages.

37. 7:08

    I've said, "If you're gonna use the send message tool, you have to ask me first."

38. 7:13

    So, did it go right? Does anybody think it went right?

39. 7:21

    This is a large block of text, um, but basically the bottom line is the agent heard my constraints. The agent knew what it was, was supposed to do and what it wasn't supposed to do, and completely and totally violated them.

40. 7:35

    And when pushed, the agent cops to it, of course. We've all seen the meme.

41. 7:41

    This is a serious gap. Yikes. It knew it wasn't supposed to do what it did,

42. 7:49

    by my intent and by the other controls that were put in place around it.

43. 7:56

    But notice what didn't happen. It didn't try to hack its box. It didn't try to do anything that it couldn't do, that it wasn't authorized to do. It understood the constraint,

44. 8:11

    and it just decided that task completion mattered more. It picked the tool that let it proceed, knowing that the tool didn't respect the constraint, and then admits to it later and says, "Oops, my bad."

45. 8:26

    Here's another one. An agent is faced with an egress filter. The user says, "I want you to go do some stuff. Look on the internet." And the agent says, "I, I, I can't do that.

46. 8:40

    I'm not allowed to get to that site." So, um, it hits the limit, and it escalates one of these notes to the user, and it says, "But by the way, i- if you install this teeny, tiny, little Chrome extension for me,

47. 8:55

    then I could route around that control, and I could do the thing that you want me to do, and we'd all live happily ever after."

48. 9:02

    Well, in the real world, the only reason that this failed in my environment was that we had another control, a layered control, that prevented the extension from getting installed, because this wasn't something that we wanted agents to be able to do.

49. 9:18

    And at the end of the day, the energy required to remove this constraint came from inside the agent itself. It simply routed through the human as a tool to achieve its goal.

50. 9:37

    Okay. So stuff is working. We have egress filters. We have gVisor sandboxes. We have a good deal of structural controls and deterministic guardrails, and I'm sure most of the speakers today have talked about a lot of these things.

51. 9:55

    We have auditability, and we have telemetry. These are very, very important foundational things that will make AI computing safe.

52. 10:06

    They are necessary, but they are not sufficient.

53. 10:09

    The real question, the real problem, is that when agents find ways around these constraints,

54. 10:16

    we have a different problem. We have a pernicious problem.

55. 10:27

    Harmful behavior that is hard to catch because the system looks compliant

56. 10:35

    the entire time. The agent understands its constraints.

57. 10:43

    It decides task completion matters more. It proceeds. It can explain itself. It documents itself.

58. 10:53

    This is the same human-level judgment that naive 2006 Aaron Stanley did in that midtown office that led to the whole yelling and things.

59. 11:07

    But there's no human-level accountability here. The research has named this-- There are a number of papers that talk about things like outcome-driven constraint violations and agent misalignment. The failure mode exists.

60. 11:23

    We've documented it, but the response I haven't seen yet. So that's what I am here pleading with you all to help me work on. So here's my proposal.

61. 11:39

    And this is older research than anything that I've mentioned so far. There's a paper about corrigibility in AI, and the original framing was really narrow. Like, will the agent resist getting shut down if it's threatened with something like that?

62. 11:57

    Um, will it modify its own values if given a pretty stark alternative? But I think we need to broaden it for the modern era as things have moved so fast recently.

63. 12:10

    So here are some rules. One, constraints must be load-bearing, not negotiable.

64. 12:20

    Two, the energy to overcome a constraint must come from outside of the agentic loop. And three, when constraint and task collide, the default agent behavior should be halt and explain, not, uh, find a way.

65. 12:38

    Experienced 2026 me was corrigible. Naive me was not.

66. 12:47

    And so if we build on that and we look at the floor, work in progress, things are coming out fast from Frontier Lab's awesome deterministic constraints that we need.

67. 12:59

    Great. On top of it, a corrigible-by-design agent, one that halts at the tension and surfaces its intent. It waits. It doesn't try to recruit the human to get around the constraint and do what you wanna do.

68. 13:16

    Instead, it passes that onto an intelligent adversary. So the intelligent adversary would be something, uh, like an equal power agent that is reasoning about the semantic intent.

69. 13:29

    Did the worker, uh, do something within the, the spirit of the constraint, not necessarily just the syntax of it?

70. 13:40

    And at the top, there are our humans. Humans out of the loop that are going to be able to evaluate a statement that is not some long, obfuscated bash command with a yes/no prompt at the end of it, but rather a natural language-type interface where the adversary has said, "Yo, human, your agent wants to

71. 14:05

    do this thing. I think it violates one of the constraints. Here's what I think is happening, and here's what I think might happen if you let the agent continue.

72. 14:17

    What would you like to do?" To me, that is defense in depth.

73. 14:29

    So the equal power agent that's trained to stop the worker from violating intent, it's a very different calculation from trying to figure out what the intent is.

74. 14:43

    It's something that's a lot simpler to reason about.

75. 14:48

    And so if we build an agent like this that has a reward incentive to stop the subordinate agent from finishing its job, then for the examples that I've put forward today, I think we'd have caught what the syntactical rules couldn't prevent.

76. 15:09

    The tool substitution. I can, but should I?

77. 15:17

    The tool dissolution. I can figure out a way to do this if I just think differently about the problem and innovate around it. Or the deprioritization in a lot of the early research, dropping constraints under the pressure of a threat.

78. 15:39

    Now, I have to admit, this will probably raise cost. It might introduce latency. Um, it's not going to eliminate risk.

79. 15:51

    Nothing can. But it makes the composition of the human escalation meaningful. It's true defense in depth, and it's not a silver bullet. And it's important because in a few weeks,

80. 16:10

    CISOs like me and my colleagues that are dealing with high-risk AI are going to have to account when the EU AI, EU AI Act starts coming into effect.

81. 16:25

    They're gonna have to account for ensuring meaningful human oversight of agent decisions in high-risk AI. A sandbox diagram with a yes/no LGTM ain't gonna cut it.

82. 16:40

    The defensible answer isn't more controls on top of an already viable sandbox.

83. 16:47

    So the oversight question is structural. It's why I didn't get fired.

84. 16:53

    The four layers that I've given to you today are the defensible answer. A deterministic floor, a corrigible agent, an intelligent adversary, and a structured, meaningful human escalation.

85. 17:06

    Relying only on constraints with known weaknesses is like finding a nest of eggs in the middle of Jurassic Park and assuming that they were just put there by a passing flock of seagulls.

86. 17:19

    Ain't gonna work. Thank you. [audience applauding]

87. 17:34

    All right. Here we go. Okay, there we go. We are live. All right, so I think we have time for maybe one or two questions if that's all right with you, Aaron.

88. 17:43

    Yeah. Sure.

89. 17:44

    All right. Sure. Uh, why don't you go right here?

90. 17:51

    First of all, thank you so much. Uh, I think you covered, um, the breadth and the depth, uh, at a, at a CISO level. It's really appreciated. Um, two-part questions.

91. 18:00

    One is, now that y-you're preaching to us or perhaps, you know, highlighting the, the importance of security, uh, broad and deep, what are some of the investments you are prioritizing- Uh, especially the newer ones, uh, given, you know, the newer attack surfaces.

92. 18:17

    Um, and then th- the sub-part of that is, you know, if you can break down between, uh, defensive solutions versus runtime solutions and preventive solutions. That would be great.

93. 18:27

    Mm-hmm.

94. 18:27

    Thanks.

95. 18:29

    So, uh, th- things that I have been prioritizing are, uh, building, like, foundational guardrails with layers, right? So kind of what I e- expressed with the agent and the egress filtering.

96. 18:40

    Um, I want to have some control and governance over how the entire enterprise deployment is made. And then I wanna have additional controls underneath things that I might not have had in the past.

97. 18:53

    Things like, um, I am backing up people's laptops now. I never thought I would back up people's laptops after, like, 2020. Uh, but people can delete their data that's on their laptop now with a simple agentic que- uh, query.

98. 19:10

    Um, so how I think about runtime, uh, I've used a number of runtime tools. I think a lot of folks that have been building them are coming at them from, uh, the, the sort of same places we came at a lot of original security, uh, tooling with.

99. 19:26

    And that's data leak and, and prevention, and, and it's not equipped for non-deterministic workloads. I think there's something completely different about these, and you can't just use strings, and you can't just try to reason in a small box about what the agent's doing.

100. 19:46

     So, uh, one of the things that I really like to experiment with is, how do I hook the agent at runtime with a set of policies? Not trying to detect, you know, on the output, but on the input, giving it the right guardrails.

101. 20:00

     And I, I, I like that. I like that approach a lot.

102. 20:09

     Uh, sort of build- build- building on that. First, thank you. This is very, very cool. Um, w- wondering, so the ideas here are completely aligned with where, where do you see this existing?

103. 20:19

     Is this at the tool call level? Is this every single turn it runs through this sort of process? Like how, how, how might you actually instrument this in practice?

104. 20:30

     I, I, I think this has to be instrumented in the harness. I am not a deep enough engineer to know how that would work. This is, this is my plea to you all who are way more intelligent about this than I am.

105. 20:44

     Uh, but what I, what I've seen, kinda the same answer I gave before. Like, what I've seen in the things that we've built is when we can intercept an agent that's about to write a line of code and say, "Hey, by the way, here's our standard for authentication.

106. 21:02

     Make sure you use that library." Right at that time, before it writes the line, it works. So I think the question is like, what do you do as a post-tool hook, and is that the right place to do that?

107. 21:15

     Pr- prob- probably, but again, I'm out of my depth at that point.

108. 21:21

     Is that... Okay.

109. 21:24

     All right. [outro music]