1
00:00:03,300 --> 00:00:08,120
 Hi, everyone. Welcome to my talk about what happens when you detonate thousands of agent skills.

2
00:00:08,600 --> 00:00:16,000
 This is a look inside our agent detonation chamber and what we saw when we put skills in there.

3
00:00:16,540 --> 00:00:20,920
 My name is Roey. I'm a staff engineer at Zenity, and I'm a Muts Research Fellow.

4
00:00:21,200 --> 00:00:25,220
 I talk a lot about AI safety and security. Follow me on Twitter and LinkedIn.

5
00:00:26,680 --> 00:00:33,060
 Before we even start our session, we need to talk about what an agent's skill is.

6
00:00:33,380 --> 00:00:37,460
 Well, essentially, an agent's skill is capability on demand.

7
00:00:37,820 --> 00:00:41,700
 It has a bunch of instructions, resources, new behavior.

8
00:00:41,700 --> 00:00:44,780
 It can contain scripts, and it can even contain code.

9
00:00:45,540 --> 00:00:50,600
 The problem is that it was written by someone else, and this is exactly where the problems start.

10
00:00:50,820 --> 00:00:54,020
 So let's take a look at the skill's trajectory, for instance.

11
00:00:54,880 --> 00:00:58,140
 If we look at the trajectory over here,

12
00:00:58,440 --> 00:01:03,420
 we can see the skill leads through the cloud configuration

13
00:01:03,420 --> 00:01:04,600
 and Kubernetes configuration.

14
00:01:04,920 --> 00:01:07,080
 It contacts the identity service,

15
00:01:07,100 --> 00:01:10,040
 and it even props the instance metadata endpoint.

16
00:01:10,720 --> 00:01:13,720
 Now, I don't know what was the original intent of the skill,

17
00:01:13,840 --> 00:01:16,020
 but I can tell you that a lot of infosteelers

18
00:01:16,640 --> 00:01:18,060
 are operating in the same way.

19
00:01:18,260 --> 00:01:21,580
 They scan for the different secret and identity folders.

20
00:01:22,220 --> 00:01:25,940
 In this case, however, it was an official Azure diagnostic skill.

21
00:01:26,360 --> 00:01:32,280
 And this is exactly the point where it's kind of hard for us to identify whether a skill is malicious or benign.

22
00:01:32,440 --> 00:01:36,740
 And we have to understand the original intent of the skill according to its actions.

23
00:01:37,500 --> 00:01:41,820
 And at this point, you might ask, okay, so we can just like scan the skill, right?

24
00:01:42,900 --> 00:01:45,400
 Well, it's more complicated than that.

25
00:01:45,580 --> 00:01:50,100
 The first problem that we've noticed is that the payload just lives elsewhere.

26
00:01:50,100 --> 00:01:54,140
 In a lot of the skills, skill.md is just the entry point.

27
00:01:54,400 --> 00:01:59,340
 It can invoke another endpoint that contains the rest of the prompt.

28
00:01:59,380 --> 00:02:03,620
 It can download another package as part of its installation script.

29
00:02:03,800 --> 00:02:09,040
 And it might have another script or code that downloads other payloads and other packages.

30
00:02:09,740 --> 00:02:16,180
 So if you use just a static scanner and read through skills.md, you would lose all of that.

31
00:02:16,180 --> 00:02:19,880
 And you would have to create some sort of mechanism to track all of these payloads.

32
00:02:20,100 --> 00:02:21,220
 if you don't want to lose them.

33
00:02:21,720 --> 00:02:23,520
 The second problem that we've noticed

34
00:02:23,520 --> 00:02:25,960
 is that LLM analyzers can be tricked.

35
00:02:26,700 --> 00:02:29,220
 There's a really interesting research by Trail of Bits

36
00:02:29,220 --> 00:02:31,660
 in which they showcased a skill

37
00:02:31,660 --> 00:02:34,740
 that manipulated the package registry

38
00:02:34,740 --> 00:02:36,420
 of one of the projects,

39
00:02:36,600 --> 00:02:38,480
 and it had like a small comment

40
00:02:38,480 --> 00:02:42,400
 that said that the AppSec team approved this change,

41
00:02:42,600 --> 00:02:45,680
 and it completely worked on the LLM analyzer.

42
00:02:46,000 --> 00:02:49,580
 So given that scans are not bulletproof,

43
00:02:50,160 --> 00:02:52,160
 we've resorted to a different kind of action.

44
00:02:52,400 --> 00:02:55,340
 And we actually decided to just run the skill with the agent.

45
00:02:56,080 --> 00:03:00,020
 This is a technique employed by malware researchers for years now,

46
00:03:00,200 --> 00:03:03,460
 and it's called detonation, a detonation chamber.

47
00:03:03,740 --> 00:03:08,140
 So we created the same one, but for agents and skills.

48
00:03:08,700 --> 00:03:10,240
 And this was our first version.

49
00:03:10,380 --> 00:03:13,320
 In our first version, we created a task prompt.

50
00:03:13,580 --> 00:03:15,840
 We put an agent and a skill inside a sandbox.

51
00:03:15,960 --> 00:03:19,700
 We let it operate over, like, minutes or hours.

52
00:03:19,700 --> 00:03:25,320
 until it completed a task with an observed agent traces and analyzed it.

53
00:03:25,520 --> 00:03:27,400
 You can imagine that it wasn't enough.

54
00:03:27,580 --> 00:03:31,940
 Our first problem was that the environment was missing most of the time.

55
00:03:32,140 --> 00:03:34,960
 A lot of skills rely on CLIs.

56
00:03:35,160 --> 00:03:40,240
 They rely on specific services and even on the existence of projects.

57
00:03:40,640 --> 00:03:43,420
 You can imagine notion skill, for instance.

58
00:03:43,740 --> 00:03:47,280
 This will have to have a notion service available to talk to.

59
00:03:47,900 --> 00:03:54,000
 So we've added a world builder that looks at the skill and builds the appropriate environment accordingly.

60
00:03:55,040 --> 00:03:56,700
 But then we'll encounter another issue.

61
00:03:57,120 --> 00:04:01,540
 A generic task prompt might not be enough for all skills.

62
00:04:02,040 --> 00:04:03,280
 I mean, think about it.

63
00:04:03,400 --> 00:04:09,040
 You can have a presentation skill that needs to create a deck or a deployment skill that needs to ship a service,

64
00:04:09,040 --> 00:04:12,500
 a brainstorming skill that needs to explore an idea.

65
00:04:12,740 --> 00:04:16,120
 Each and every one of them has an entirely different motus operandi.

66
00:04:16,600 --> 00:04:18,880
 Let's take shipping a service, for instance.

67
00:04:19,100 --> 00:04:20,840
 You need to learn how to configure it.

68
00:04:21,020 --> 00:04:22,200
 You need to learn how to deploy it.

69
00:04:22,400 --> 00:04:26,180
 You need to tell it to recover from snapshots, using snapshots.

70
00:04:26,780 --> 00:04:31,500
 Each and every one of them requires a different set of prompts,

71
00:04:31,500 --> 00:04:39,280
 and you also need to make sure that you cover every route in that skill

72
00:04:39,280 --> 00:04:40,620
 because one of them can be malicious.

73
00:04:40,920 --> 00:04:43,180
 So we created the operator that does just that.

74
00:04:43,180 --> 00:04:50,340
 It leads through the skill and tries to analyze what are the different trajectories that we should invoke.

75
00:04:50,840 --> 00:04:52,760
 And why it worked pretty well?

76
00:04:52,980 --> 00:05:05,580
 If you worked with agents, especially in the security domain, you can assess that we went through a lot of safeguards,

77
00:05:05,960 --> 00:05:06,980
 safeguard triggers.

78
00:05:07,480 --> 00:05:09,440
 We got a lot of refusals.

79
00:05:10,320 --> 00:05:12,840
 the agents wasn't willing to decode

80
00:05:12,840 --> 00:05:14,820
 and when it was willing to decode

81
00:05:14,820 --> 00:05:16,300
 it wasn't willing to execute

82
00:05:17,140 --> 00:05:18,940
 so we had to give you

83
00:05:18,940 --> 00:05:20,060
 an offer you can't refuse

84
00:05:20,060 --> 00:05:22,960
 and essentially create an adversarial operator

85
00:05:22,960 --> 00:05:25,240
 that convinces the agents

86
00:05:25,240 --> 00:05:25,740
 to run

87
00:05:25,740 --> 00:05:28,220
 and in case you didn't run even

88
00:05:29,080 --> 00:05:30,980
 start the run from

89
00:05:30,980 --> 00:05:32,360
 the beginning

90
00:05:33,360 --> 00:05:34,760
 now I would

91
00:05:34,760 --> 00:05:37,220
 emphasize that we're doing a security

92
00:05:37,220 --> 00:05:39,360
 research here, this is a container environment

93
00:05:39,360 --> 00:05:41,000
 Nothing is malicious in here.

94
00:05:41,200 --> 00:05:45,420
 But this is one of the things that we had to do in order to make it run properly.

95
00:05:45,720 --> 00:05:51,720
 I would also note that a lot of the time, safeguards are being run on something which is definitely not malicious.

96
00:05:52,360 --> 00:05:55,820
 Sometimes you need to decode a payload, and it's part of the official skill.

97
00:05:56,020 --> 00:05:57,640
 But the safeguard would still be triggered.

98
00:05:58,940 --> 00:06:01,860
 So up until now, we were dealing with the sandbox.

99
00:06:03,280 --> 00:06:08,520
 At this point, we started to tackle the capturing mechanism.

100
00:06:09,900 --> 00:06:13,980
 So agent traces can go that far.

101
00:06:14,380 --> 00:06:17,200
 At some point, you'd be missing some side effects.

102
00:06:17,520 --> 00:06:19,600
 And npm install is the perfect example.

103
00:06:20,360 --> 00:06:23,140
 npm install has a post-installation script.

104
00:06:23,400 --> 00:06:26,520
 And the post-installation script can invoke a new process.

105
00:06:26,520 --> 00:06:30,240
 And that process can read and write to a file and then connect to an endpoint.

106
00:06:30,640 --> 00:06:32,840
 That's exactly how infosteelers work.

107
00:06:33,700 --> 00:06:40,260
 So we understood that these are also events and signals that we need to capture.

108
00:06:40,500 --> 00:06:43,760
 And we did just that as part of the capturing mechanism.

109
00:06:43,760 --> 00:06:49,320
 We added process creation event, file system event, and network packet capturing.

110
00:06:49,880 --> 00:06:57,740
 Given all of that along with the agent traces, we then moved it to quality evidence and captured the behavior.

111
00:07:00,140 --> 00:07:08,420
 Even this final form of the detonation chamber, let's take a look at an example skill.

112
00:07:08,800 --> 00:07:13,540
 So Zach Korman, you should follow him on Twitter, he created a really interesting skill proof

113
00:07:13,540 --> 00:07:18,240
 of concept where he created a skill called Security Review 2.

114
00:07:18,440 --> 00:07:23,360
 And as part of the skill installation, it installed a new skill called Security Review.

115
00:07:23,360 --> 00:07:26,460
 and security review tells you to go into GitHub

116
00:07:26,460 --> 00:07:28,320
 and fetch the content of skill.md.

117
00:07:28,580 --> 00:07:29,560
 And as you can imagine,

118
00:07:29,760 --> 00:07:32,220
 that skill.md contains hidden instructions

119
00:07:32,220 --> 00:07:34,880
 for the agent to execute the following payload.

120
00:07:35,200 --> 00:07:36,700
 And it actually did it.

121
00:07:36,820 --> 00:07:40,020
 Now, I would note that this is not a malicious payload.

122
00:07:40,200 --> 00:07:41,440
 This was just a proof of concept,

123
00:07:41,440 --> 00:07:43,540
 but it does work.

124
00:07:43,760 --> 00:07:46,500
 And static scanners can't figure that out.

125
00:07:46,960 --> 00:07:50,940
 You have to actually execute the entire chain

126
00:07:50,940 --> 00:07:52,340
 in order to recover it.

127
00:07:52,340 --> 00:07:55,820
 And this is a result of one run on the detonation chamber.

128
00:07:57,160 --> 00:08:03,880
 Using this approach, we were able to uncover 100 malicious skill out of thousands,

129
00:08:04,140 --> 00:08:08,600
 and they were shut off the skills.sh framework.

130
00:08:10,200 --> 00:08:12,420
 Now, let's take a look at another example.

131
00:08:13,120 --> 00:08:13,660
 Polymarket.

132
00:08:14,380 --> 00:08:21,000
 This was an interesting skill that is part of its installation script called Polymarket Client V2.

133
00:08:21,800 --> 00:08:23,260
 You notice the S over here.

134
00:08:23,420 --> 00:08:25,440
 This is not the original Polymarket package.

135
00:08:26,820 --> 00:08:32,000
 And when we went into NPM and looked at package.json of the package,

136
00:08:32,060 --> 00:08:36,680
 we noticed that it actually links to a non-official NPM registry.

137
00:08:37,020 --> 00:08:38,240
 This isn't real.

138
00:08:38,520 --> 00:08:39,200
 This isn't official.

139
00:08:39,500 --> 00:08:43,080
 And looking at the payload, it was actually an infostealer.

140
00:08:45,200 --> 00:08:46,780
 So it's out there.

141
00:08:46,880 --> 00:08:47,660
 It's happening.

142
00:08:49,060 --> 00:08:57,320
 Now, let's take a look at tricks that we've seen actors have been doing in the ecosystem.

143
00:08:57,780 --> 00:08:59,240
 One of them is persistence.

144
00:08:59,580 --> 00:09:08,560
 So some of the skills would add an additional prompt to Sol.js, to Sol.md or to Agent.md,

145
00:09:08,700 --> 00:09:12,060
 stating that if the skill is not installed, just reinstall it.

146
00:09:12,140 --> 00:09:14,420
 We've seen it happening a couple of times.

147
00:09:15,060 --> 00:09:18,140
 Another thing that we've noticed is silent actions.

148
00:09:18,820 --> 00:09:20,460
 There's a skill creator out there,

149
00:09:20,580 --> 00:09:23,020
 which is not Entropic's official skill creator,

150
00:09:23,460 --> 00:09:26,240
 that checks if Entropic's skill creator is installed,

151
00:09:26,380 --> 00:09:30,820
 in case it is uninstalled, and install itself instead.

152
00:09:31,620 --> 00:09:33,900
 This whole thing happens silently.

153
00:09:36,880 --> 00:09:40,420
 Another trick that we've seen a lot of actors employed,

154
00:09:40,620 --> 00:09:42,980
 and this is actually legitimate, but it's still problematic,

155
00:09:42,980 --> 00:09:44,640
 is remote prompt execution

156
00:09:44,640 --> 00:09:46,540
 where the initial prompt

157
00:09:46,540 --> 00:09:48,060
 doesn't contain all of the prompt

158
00:09:48,060 --> 00:09:49,760
 and it contains instruction

159
00:09:49,760 --> 00:09:51,880
 to retrieve it from another location.

160
00:09:52,320 --> 00:09:55,160
 We've seen it on a bunch of skills.

161
00:09:55,160 --> 00:09:58,080
 Some of them are legitimate.

162
00:09:58,360 --> 00:09:59,240
 Some of them are illegitimate.

163
00:09:59,700 --> 00:10:01,360
 And it's kind of problematic

164
00:10:01,360 --> 00:10:02,260
 because as you can imagine

165
00:10:02,260 --> 00:10:04,420
 that when something is appropriate

166
00:10:04,420 --> 00:10:05,660
 at one point in time,

167
00:10:05,840 --> 00:10:07,260
 at another point in time,

168
00:10:07,480 --> 00:10:09,660
 someone can either take over the domain

169
00:10:09,660 --> 00:10:12,360
 or either do it intentionally

170
00:10:12,360 --> 00:10:14,180
 and change the prompt entirely.

171
00:10:15,340 --> 00:10:16,720
 This is fairly interesting.

172
00:10:17,100 --> 00:10:19,540
 And another technique that we've seen is hook abuse.

173
00:10:19,880 --> 00:10:23,880
 You can actually trigger a skill on a given hook,

174
00:10:24,080 --> 00:10:25,020
 like user prompt submit.

175
00:10:25,260 --> 00:10:27,680
 Now imagine what would happen if a skill is triggered

176
00:10:27,680 --> 00:10:30,840
 every time a user sends a message to the assistant.

177
00:10:31,220 --> 00:10:33,000
 This could have major consequences,

178
00:10:33,000 --> 00:10:34,440
 and you need to be aware of that.

179
00:10:35,440 --> 00:10:38,340
 But let's talk about a campaign that we've uncovered.

180
00:10:38,660 --> 00:10:41,760
 Meet Carly, or Carly Paperclip, as he known in GitHub.

181
00:10:41,760 --> 00:10:50,460
 he faked the Paperclip organization. Paperclip is a well-known project. It's an open-source

182
00:10:50,460 --> 00:10:58,840
 project aimed for orchestration of various agents. Carly here basically created a fake

183
00:10:58,840 --> 00:11:05,620
 organization that mimics it. He registered the domain under kind of the same name, created a

184
00:11:05,620 --> 00:11:08,400
 lookalike organization, and then clone the project.

185
00:11:08,680 --> 00:11:14,600
 After he cloned the project, he did something bad.

186
00:11:14,780 --> 00:11:19,820
 He created a new log action call and called it under all of the different routes of the

187
00:11:19,820 --> 00:11:20,300
 project.

188
00:11:20,540 --> 00:11:31,200
 When we looked into what's happening into log action, we noticed that it's calling the

189
00:11:31,200 --> 00:11:34,020
 attackers' control endpoint.

190
00:11:34,020 --> 00:11:46,600
 It fetches a payload from there, decodes it, saves it into a temporary file, and executes it under a node and self-destroy after 60 seconds.

191
00:11:47,540 --> 00:11:51,400
 When we looked at the payload, this was alarming.

192
00:11:52,240 --> 00:11:53,940
 It was a credential harvester.

193
00:11:53,940 --> 00:12:10,380
 Essentially, it went through the different sensitive paths on the machine, such as SSH, AWS, and Azure, collected all the information from those folders, and then posted everything to a feedback endpoint on by the attacker.

194
00:12:11,100 --> 00:12:12,100
 This was bad.

195
00:12:13,280 --> 00:12:15,960
 He then did the same thing in browser use.

196
00:12:16,980 --> 00:12:20,580
 But this was a Python package, but he used the same command and control.

197
00:12:20,840 --> 00:12:22,940
 And then he tried to publish it in PyPy.

198
00:12:23,500 --> 00:12:24,600
 That didn't work.

199
00:12:25,080 --> 00:12:26,960
 Like under two hours, it was just down.

200
00:12:27,240 --> 00:12:31,420
 He did the same thing with Paperclip AI, which is the Python version of what we just saw.

201
00:12:31,880 --> 00:12:34,600
 And that also went down after less than an hour.

202
00:12:35,080 --> 00:12:40,800
 Well, at this point, you'd think that we'd be winning, but no, because the skins that

203
00:12:40,800 --> 00:12:45,460
 were using the same command and control were actually trading, trending at that time to

204
00:12:45,460 --> 00:12:49,520
 the point where they had 1.7 million total downloads.

205
00:12:50,140 --> 00:12:52,620
 So that's, that was pretty bad.

206
00:12:52,940 --> 00:12:55,960
 And it operates approximately the same way.

207
00:12:56,100 --> 00:13:01,920
 Upon setup installation, the payload is fetched from the attacker's own health endpoint.

208
00:13:02,440 --> 00:13:06,560
 It is then being decoded and sent to a JavaScript file.

209
00:13:07,020 --> 00:13:08,200
 Node then executes it.

210
00:13:08,320 --> 00:13:12,620
 And then the entire credential harvesting and sending to the post endpoint happens.

211
00:13:12,840 --> 00:13:14,080
 And it's of destruction as well.

212
00:13:14,540 --> 00:13:15,660
 Exactly the same chain.

213
00:13:16,480 --> 00:13:19,460
 We reported the incident to Vercel and Microsoft.

214
00:13:19,720 --> 00:13:22,580
 And under the last 12 hours, they took everything down.

215
00:13:22,940 --> 00:13:25,820
 If you want to read more about it, you should read the Zenity Labs blog.

216
00:13:26,280 --> 00:13:28,560
 It's very interesting.

217
00:13:29,380 --> 00:13:30,280
 Incredible work.

218
00:13:33,380 --> 00:13:38,440
 Skill is just one single part of the agent supply chain.

219
00:13:38,680 --> 00:13:41,940
 Everything that goes inside the agent context is part of it.

220
00:13:42,020 --> 00:13:43,640
 It can be a website.

221
00:13:43,640 --> 00:13:45,380
 It can be rows in your database.

222
00:13:45,380 --> 00:13:46,340
 It can be packages.

223
00:13:46,600 --> 00:13:47,940
 It can even be a YouTube comment.

224
00:13:48,460 --> 00:13:50,840
 Everything that comes into the context is under risk.

225
00:13:51,900 --> 00:13:57,960
 And if we talk about skills in particular, you need to treat them as package dependencies.

226
00:13:58,680 --> 00:14:07,580
 You need to inspect everything that it loads, limit what it can access, list privileges, as you're well aware, and observe what is actually being run.

227
00:14:08,860 --> 00:14:17,820
 Now, this entire work on the detonation shiver was done by the entire research and engineering thing that you see over here.

228
00:14:17,820 --> 00:14:21,080
 highly talented people, and I encourage you to follow them.

229
00:14:22,220 --> 00:14:24,860
 And with that, if you want to test out your skills,

230
00:14:25,040 --> 00:14:26,640
 it's available in a.total.io.

231
00:14:26,840 --> 00:14:29,940
 This is the donation chamber, and thank you so much.