How we hacked YC Spring 2025 batch’s AI agents

Talk transcript

1. 0:00

   [upbeat music] So yeah, who's ready to hack some agents?

2. 0:17

   Yeah? Oh, wow. All right. So [chuckles] let me first introduce myself a little bit. I'm Rene. I'm the CEO of Casco. We're a YC company, and we specialize in red teaming AI agents and apps.

3. 0:28

   And so we spent, uh, I, I spent my previous time at AWS working on AI agents, but I've always really loved working on AI. In fact, there's a video of me ten years ago building voice-to-code, and I won Europe's largest hackathon by doing that.

4. 0:44

   And so I would talk to it, say, "Build me a blog post," and it would generate the sites. And it was actually-- It was kinda fun. Like, it did, uh, things like, um, yeah, uploading pictures from San Francisco, and you can see how horribly slow the APIs were back then.

5. 0:58

   And I'm gonna-- about to give you a nightmare by showing you the architecture diagram of that thing. Um, but yeah, it kinda did the job, and this was, like, ten years ago.

6. 1:06

   Obviously, back then was no generative AI, and these things were extremely difficult to do. Um, but it is-- it really gave me a glimpse of what the future could look like even back then as technology gets better, right?

7. 1:18

   So obviously, many things have changed. Two months ago, I quit AWS and worked out of, uh, the garage with my co-founder, and, uh, we got into Y Combinator. So yay!

8. 1:27

   That's awesome. And so from there, we also looked into how else have things evolved. Well, this was my, um, architecture diagram from back then. You could see there was three different cloud providers, including IBM Watson, which was, like, forefront at the time.

9. 1:41

   No, it was-- It's true. And, uh, w- before it was like, uh, Microsoft LUIS, which was like some natural language understanding things. And you can see it was just a lot of, like, piecing things together, and that was already kind of difficult to do.

10. 1:54

    But nowadays, we see the stacks normalize significantly more, right? I think this is probably what the average agent stack looks like these days. Got some sort of front end.

11. 2:05

    You talk to an API server that talks to an LLM, connects up with tools, and then you have a bunch of data sources associated to it. So this kind of normalization of agent stacks are actually really good.

12. 2:15

    That, like, makes many things easier. Definitely better than my hackathon project ten years ago. Um, but we need to think about the security posture around these systems. And my general impression over the last, uh, last few years is like primary discussions around LLM security, really like, hey, is it, um, can you do prompt injection?

13. 2:34

    Can we get it to do harmful content? Um, which is all really important. But the reality with security is you need to look at all the different arrows in your system.

14. 2:45

    And that is typically where real damage happens, right? And so this is really agent security, and that is what I wanna talk about today. Now, one thing is, like, why did we even hack a bunch of agents?

15. 2:59

    That's kind of a weird thing to do. Um, the answer is, quite frankly, you know, we wanted to launch internally at Y Combinator and wanted a splashy headline. [laughs] And so we're like, "Uh-oh, what do we do?"

16. 3:10

    And fun fact, we have the second highest upvoted launch post inside Y Combinator of all time. So higher than Rippling. Yes. Okay. [chuckles] Um, so, uh, we, we b- we did basically this approach.

17. 3:22

    At the time, we were looking at, oh, which agents are already live? And then let's just set a timer for thirty minutes. We don't wanna waste too much time on this.

18. 3:29

    And then, you know, let's, let's figure out what their system prompts are and just kinda understand how they're working. And I, I have a feeling when I was creating this meme that this could be true, but it turns out it is true.

19. 3:40

    And then we looked at, oh, what kind of tool definitions do they have, right? Like, you know, what is it supposed to do? Is it supposed to access data?

20. 3:48

    Supposed to run code, right? And then we just, uh, try to exploit them and see what's, what's going on. Uh, and it was really fun because we hacked, uh, out of sixteen agents that were launched, within thirty minutes each, we were hacked, uh, we hacked, we hacked seven of them.

21. 4:02

    And there are three common issues we see across all of these ones. So I hope that we all learn today what the most common issues are so you don't make the same mistakes.

22. 4:11

    And also, this is gonna be the best investment if you're a VC this batch because they're all secure now. So first issue, cross-user data access. I mean, you guys were just here at the OAuth talk.

23. 4:23

    You know where this is gonna head into, right? Um, [clears throat] so we first leaked this company's, uh, system prompt, and we saw, huh, it has a bunch of interesting tools attached to it, including looking up user info by ID, suspicious, uh, document by ID, and a bunch of other things.

24. 4:41

    And then, you know, like, when you see this, you just wanna like, oh yeah, there's this thing called IDOR, like Insecure Direct Object Reference. It's basically when you make a request and you validate that, hey, the token's valid, and then you just let the request through, right?

25. 4:55

    And you're kinda betting on the fact that the ID cannot be guessed. Well, that's obviously not good. Um, so yeah, we looked up a product demo video that they recorded, and we found the user ID in the URL bar and just, like, tried to plug it in.

26. 5:10

    Uh, this is a different ID, by the way. Don't worry, guys. This is my co-founder's ID now. And, uh, yeah, we were able to find their personal information, including their email, nickname, whatever.

27. 5:19

    Um, but it gets better because these things are also interconnected. So you had not only the user ID, but you also had, like, oh, the chat ID. Uh-oh. [chuckles] And their document ID.

28. 5:31

    And then these things ultimately linked up together and allows you to traverse the entire system, right?

29. 5:38

    It's not good. So [chuckles] what's the fix for that? There was a really comprehensive talk literally right before this. Sorry for the folks that missed it, but this is the basic fix for it, right?

30. 5:48

    You need to think about how do you authenticate but also authorize the request. It's really two checks, right? Make sure your, your token is valid. Good job, team. You got that.

31. 5:56

    And then the second thing is, like, this is what we see in the Superbase era with role-level security. Just make sure that you have some sort of access control matrix somewhere that checks that it matches up with whoever's making the request.

32. 6:09

    Okay? Super, super important. Authenticate and authorize. Now, you can see this was actually, you know, an issue that was kinda there, right? It's, it's not like around the LLM and the API server, it's really what is happening downstream.

33. 6:23

    And, um, yeah, there's a lot of arrows in this diagram. We're gonna look at all of them. So, the next thing is to remember, as you're thinking about these tools and how you're building it, like, agents actually act like users, um, not API servers.

34. 6:38

    When we were, like, debugging this issue, like we actually asked a bunch of Y Combinator companies, like, "Why, why did you build it this way?" Because clearly they can build a web app properly, right?

35. 6:48

    But it's just like, I think as developers, we have this natural pattern matching in our heads. It's like, "Oh yeah, this thing runs on a server, so it should be like a service, and then I'm gonna give it service-level permissions."

36. 6:58

    But actually, agents are like users, right? So everything that applies to users apply to agents, too. So make sure that, you know, your LLM should probably not determine your authorization pattern.

37. 7:09

    That, that, that's bad. That's a red flag. Uh, second thing is it should probably not act with service-level permission. Listen to your previous talk on OAuth, that's great. Um, and then just like users, you should make sure you, uh, don't just accept any input.

38. 7:21

    You should sanitize them. Same with outputs, right? A lot of these are like the traditional web application security things that you just need to like really, really internalize for this new world.

39. 7:32

    Now, that was interesting. And so the second one was even better. [laughs] Um, so this is not as common, but the damage is bigger. So it's... W- in pattern, we see, so there are a lot of code tools that agents use, and there's a, there's a, there's a Anthropic paper here.

40. 7:49

    It basically talks about what's the distribution of which industry and how much do they use Claude. And there's like this one outlier here. I'll zoom it in for you.

41. 7:58

    Um, yeah. W- so us nerds, we make up 3.4% of the world, but we're 37% of Claude's usage. Ooh, why is that? Because we love computers and we love coding, right?

42. 8:08

    And so we found immediately the value of it. But it's not just us that use agents with coding tools. In fact, many agents create code on demand to do some things, right?

43. 8:19

    Like, some agents just generate a calculator on demand to make a calculation, right? And so there's a lot of these code execution sandboxes out there that are interesting. And so if you, if you think about that, there's actually a critical path in your system because you've got a tool that talks to another container.

44. 8:37

    A container is arbitrary compute, and when you have arbitrary compute, many things can happen. Many bad things, many good things, right? But let's talk about the bad things today.

45. 8:46

    So we did the same script, did the system prompt. Again, the system prompt itself, great. I mean, doesn't cause any damage. But as an attacker, you always think about the fact, uh, the things that are like, "Huh, that's kind of suspicious," right?

46. 8:58

    It's like, "Oh wait, it, it, it runs code and never output it to the user. Okay, let's output it to the user. Oh yeah, and, and m- most- mostly run, run it mostly at most once.

47. 9:08

    Let's run it all the time." And so you try to basically invert what the system prompt is saying because that is exactly what the developer didn't want you to do.

48. 9:16

    And that is how bad actors think, right? So we figured out, oh, this thing does have a code tool, and so, you know, we tried, we tried running something.

49. 9:24

    It's like, ah, mm, it only allows me to write Python and, you know, I, I love JavaScript. And, um, yeah, and it doesn't allow me to run these really dangerous, you know, function calls.

50. 9:33

    Ah, okay, and it restricts like which Python files to run. That's also not good. So yeah. But we looked at what it could do, and it had two kinda innocent permissions: write a Python file and read some files.

51. 9:49

    You can do a lot with that. This is great because what if we just looked around the file system now, right? [laughs] We can read files. So we looked at, okay, build me a little tree functionality and, you know, return me the entire file system tree to see what's going on.

52. 10:03

    Oh my God, there's a app.py file. That's probably important. Um, and then we looked at, oh, it has two endpoints, write file and execute file. Ah, okay, these endpoints are hidden behind a VPC, so we cannot hit it directly.

53. 10:14

    That's okay. Um, but huh, we can write files. Huh, we can write files. There's a app.py file. Huh, let's look into that. Oh wait, that's where all the protections are for their code.

54. 10:29

    Uh, and so we can just override the app.py file with empty strings around the... all the security checks. And whoopsie, we got in. So now we can Bitcoin mine all day.

55. 10:42

    That's great, right? Yeah. No, it gets much worse. So the [laughs] thing with arbitrary code execution once you're inside a container is that you can do many things. Like, um, there's this thing called service endpoint discovery, metadata discovery.

56. 10:56

    Y'all heard of that? No? Okay. Basically allows you to discover what are other devices on the, uh, what are other devices on the network, what other resources are there on the network.

57. 11:05

    And, uh, you can also just, you know, fetch the user token, uh, sorry, the service token, you know, just see what's going on. What's the project name? Yeah, you know.

58. 11:12

    And you start looking around, it's like, oh, okay, yeah, okay, I, I, I can also fetch the scopes, so I can use, do many things with this token. That's awesome.

59. 11:20

    Um, who has really, really spent time configuring service-level tokens and their permissions in a granular manner and does it all the time and never forgets to set something wrong?

60. 11:33

    Okay, one guy, one guy there. [laughs] Okay. Whoopsie, we have access to all their customer data, so that's, uh... And we just queried BigQuery, which has a great interface for that.

61. 11:41

    Isn't that great? Yeah. So yeah, y- making sure you have code sandboxes correctly is very hard because you can move laterally across the infrastructure, and that is just very, very dangerous, okay?

62. 11:53

    And so kinda like don't roll your auth in the web world. Don't roll your own code sandboxes, please. Like, it's, it's just very hard. It's very, very hard. And so use a out-of-the-box solution.

63. 12:04

    There are many of them. E2B is, I think, a very popular one. Some, some folks have probably heard of it. Uh, there's one i- in our YC batch that I personally just genuinely really love.

64. 12:13

    They have observability built in. They boot up super quickly. And what I love about them is they have an MCP server that just is easy to plug into, right?

65. 12:19

    So just easier for your agents to work with. So please Do that. [laughs] Don't do, you know, your own Python app.py thing. Um, it's not good. Trust me. Um, so that leads into a third part of a attack vector around server-side request forgery.

66. 12:37

    This is... It's, it's a very long word, and it really bugs me that the SSRF didn't fit on the previous line. Just really triggers me. Um, yeah, I know.

67. 12:45

    So, um, this is what happens when you can kinda co- can kinda get a tool to call another endpoint that you didn't, and, you know, that the service itself didn't intend you to call, and you can pull out a lot of information just through that workflow.

68. 13:01

    So let me give you an example. So this is ex- exactly, extracted system prompt. Great. Oh, this thing can create databases. That sounds exciting. Um, and then you look into it, it's like, huh, it pulls a database schema from a private GitHub repository.

69. 13:18

    Isn't that great? That means whatever request goes to that private GitHub repo- repository must have the Git credentials, right? Otherwise, how can it pull that from a private repository?

70. 13:28

    So, um, yeah, and it's just a string, so I guess I can just put in whatever string I want and coerce it into providing that. So let's set up a badactor.com_test.gitrepo and just see what credentials come through.

71. 13:41

    And, yep, it comes across with the Git credentials. And so now you can actually take those Git credentials and just download their entire code base that was behind a private repo.

72. 13:51

    Isn't that crazy? Isn't that crazy? Yeah. This is... I mean, it's awesome for me to do this, right? [laughs] It's like you, you get paid to do th- oh, come on, it's amazing.

73. 14:01

    Now, um, we told our batch mates immediately, and they told us, "Don't worry, bro. It's already fixed. It's okay, guys." [laughs] That, that company's secure if you're a VC listening in.

74. 14:09

    Um, so, so but with that, though, it is really important to think about the implications of what your system is doing, right? I, I love vibe coding, not gonna lie, but, like, you gotta really think about where all these arrows are and if you've configured those things corre- correctly.

75. 14:26

    So with that, always sanitize your inputs and outputs. This could be like a web dev conference from 20 years ago. Um, [laughs]

76. 14:34

    but it, but it applies to agents too, right? Like, we just need to make sure we keep those good security practices that have, that we have learned to love, hopefully, [laughs] over the years to k- take it forward to a new technology paradigm.

77. 14:47

    And then ultimately, I want you to take away three things. So first thing is agent security is bigger than just LLM security. Make sure you understand how these threat vectors apply inside your overall system.

78. 15:00

    Second thing is treat agents as users, and that applies to authentication, to sanitization of user inputs, and many of the other things. And f- last thing, definitely don't roll your own code sandbox.

79. 15:11

    That is just so dangerous, and, you know, it, it, it very quickly turns from, like, an intern project into, like, a nightmare. So it... Be very, very careful of that.

80. 15:20

    And these are the most basic ones that we've seen come across, right? There's obviously many more security issues. And if you don't know exactly how your agent security posture is, you can go to Casco.com, you can book a demo with us.

81. 15:33

    We built an AI agent that actively attacks other AI agents and tells you where they break. Isn't that great? Um, and yeah, feel free to connect with me on LinkedIn or on Twitter.

82. 15:43

    And I have, uh, every now and then, some good stuff to post. Yeah. [clapping]

83. 15:50

    Awesome. Thanks, Rene. Does anyone have any questions? We have time for, like, one or two quick questions if you're in, if you're game for it. Sure.

84. 15:58

    How do you look at the system prompts?

85. 16:00

    Um, how do I look at system prompts? There's a lot of just, like, open techniques. The, the best one that I've seen is, uh, from hiddenlayer.com. Have you guys checked that, those guys out?

86. 16:08

    Yeah.

87. 16:08

    They have a great blog post on, like, um, the... It's a policy puppeteering attack. Yeah.

88. 16:13

    Mm.

89. 16:14

    It's great.

90. 16:16

    Very well.

91. 16:17

    Cool.

92. 16:17

    Awesome.

93. 16:18

    Oh.

94. 16:18

    Yeah.

95. 16:20

    Interesting, like, coding agents. Like, how do you make sure... Because a coding agent can recognize. Like, how do you make sure that it's actually not running, like, running the proper commands?

96. 16:29

    Because it's a super tough thing to do. Like, if you try whitelisting, like, there's so many creative ways that events can get around you- them. But, like, how, how would you-

97. 16:38

    Yeah. Are, are you talking about it locally or server side?

98. 16:41

    Um, they're on both. Like-

99. 16:43

    Yeah

100. 16:43

     ... I mean, locally is even more dangerous because they have-

101. 16:46

     Yeah

102. 16:46

     ... credentials of the user running them. [laughs]

103. 16:49

     Yeah, no, ve- very much so. So locally, uh, I think right now the industry is either you go full YOLO mode or you ask every time, right? Um, I mean, I'm not joking.

104. 16:58

     Cursor's thing is called YOLO mode, right? [laughs] Um, and then on server side, use a code sandbox, because ultimately they have constraints, uh, around the internal networks, but also they have constraints around, um, how long they can live as a sandbox.

105. 17:10

     Yeah.

106. 17:12

     Okay, so sandboxes that use VM is actually-

107. 17:14

     Um, yeah. So they, they typically use something called Firecracker under the hood-

108. 17:18

     Ah

109. 17:18

     ... which is a better isolation layer. Yeah, if you just use containers, by the way, that's not an isolation layer, in case anybody's wondering.

110. 17:23

     That's why I asked.

111. 17:23

     Yeah. Yeah, don't use containers for isolation. Yeah. [outro music]