1
07:00:53,120 --> 07:00:55,350
>> Hello, my name is Nikolai Petro and we

2
07:00:55,360 --> 07:00:57,830
will be talking about skills and agents

3
07:00:57,840 --> 07:01:01,430
knowledge of those skills.

4
07:01:01,440 --> 07:01:03,510
We'll start it with the simple prompts

5
07:01:03,520 --> 07:01:06,868
where we've been playing with the new

6
07:01:06,878 --> 07:01:08,230
stuff, new work being done by the

7
07:01:08,240 --> 07:01:10,310
agents, repeating those prompts over and

8
07:01:10,320 --> 07:01:12,468
over again. And then we start packing

9
07:01:12,478 --> 07:01:14,868
those into the skills,

10
07:01:14,878 --> 07:01:17,270
creating repeatable, reproducible,

11
07:01:17,280 --> 07:01:21,590
portable type of their functionality

12
07:01:21,600 --> 07:01:25,190
or capabilities that we can teach agents

13
07:01:25,200 --> 07:01:27,270
to work with. And some of those skills

14
07:01:27,280 --> 07:01:28,788
are very important. hundreds of the

15
07:01:28,798 --> 07:01:33,110
stars on GitHub.

16
07:01:33,120 --> 07:01:34,628
There are different types of their

17
07:01:34,638 --> 07:01:38,150
skills. Some teach models something new

18
07:01:38,160 --> 07:01:40,310
bringing new knowledge or capabilities

19
07:01:40,320 --> 07:01:43,910
to the model. But there are also types

20
07:01:43,920 --> 07:01:46,150
of their just steering or process skills

21
07:01:46,160 --> 07:01:48,468
where we

22
07:01:48,478 --> 07:01:50,150
do not bring any additional information

23
07:01:50,160 --> 07:01:53,510
but just getting repeatable

24
07:01:53,520 --> 07:01:55,830
flow of the actions that we want agents

25
07:01:55,840 --> 07:01:59,590
to perform. both are useful. In this

26
07:01:59,600 --> 07:02:01,350
session, we will be talking about new

27
07:02:01,360 --> 07:02:05,830
knowledge type of the skills and

28
07:02:05,840 --> 07:02:08,868
we'll see what we can

29
07:02:08,878 --> 07:02:10,708
get from the models understanding of

30
07:02:10,718 --> 07:02:13,910
those skills.

31
07:02:13,920 --> 07:02:16,948
So, we get our first skills packed,

32
07:02:16,958 --> 07:02:21,270
start using them. Um, basically

33
07:02:21,280 --> 07:02:24,230
starting with the the feel of does it

34
07:02:24,240 --> 07:02:26,788
work or not. But there is also the more

35
07:02:26,798 --> 07:02:28,628
technical way and more deterministic way

36
07:02:28,638 --> 07:02:30,788
for the checking if skills are working

37
07:02:30,798 --> 07:02:33,350
or not and what's the impacts of those

38
07:02:33,360 --> 07:02:36,548
skills are evolves. There are different

39
07:02:36,558 --> 07:02:39,270
formats where evolves can be defined.

40
07:02:39,280 --> 07:02:41,830
But at the sense this is task definition

41
07:02:41,840 --> 07:02:44,390
that you want your agent to be able to

42
07:02:44,400 --> 07:02:46,788
solve.

43
07:02:46,798 --> 07:02:48,948
You put it in the place ideally more

44
07:02:48,958 --> 07:02:51,830
deterministic so we can uh validate it

45
07:02:51,840 --> 07:02:56,070
and we can check two arms one there no

46
07:02:56,080 --> 07:02:58,390
skill loaded and there second one where

47
07:02:58,400 --> 07:02:59,910
we're loading the skill and providing

48
07:02:59,920 --> 07:03:01,270
this additional information to the

49
07:03:01,280 --> 07:03:04,868
context and we compare how the runs

50
07:03:04,878 --> 07:03:07,030
there are multiple things one is how

51
07:03:07,040 --> 07:03:08,948
many

52
07:03:08,958 --> 07:03:11,750
evolves been solved in this case with no

53
07:03:11,760 --> 07:03:14,310
skill we're solving three out of five

54
07:03:14,320 --> 07:03:16,310
with skill we're solving all five

55
07:03:16,320 --> 07:03:19,350
meaning that we get two more tasks.

56
07:03:19,360 --> 07:03:21,750
There also would be implications on the

57
07:03:21,760 --> 07:03:25,030
tokens usage or time that it took those

58
07:03:25,040 --> 07:03:28,070
tasks to be completed and other factors.

59
07:03:28,080 --> 07:03:29,990
But at the end we are comparing those

60
07:03:30,000 --> 07:03:32,868
and seeing what makes sense, what not.

61
07:03:32,878 --> 07:03:35,270
And this is one of the mains for

62
07:03:35,280 --> 07:03:38,390
evaluating the skills and uh saying that

63
07:03:38,400 --> 07:03:41,430
they not just nice ideas but actually

64
07:03:41,440 --> 07:03:43,430
working pieces of instructions and

65
07:03:43,440 --> 07:03:46,628
context for the agents.

66
07:03:46,638 --> 07:03:50,070
But what next? We get to the stage where

67
07:03:50,080 --> 07:03:52,868
we have their working skill. It solves

68
07:03:52,878 --> 07:03:57,030
some evolves. But we know that there

69
07:03:57,040 --> 07:03:59,350
should be a better way. We have models

70
07:03:59,360 --> 07:04:03,670
evolving. We have u some

71
07:04:03,680 --> 07:04:07,110
might be cases where we want

72
07:04:07,120 --> 07:04:09,990
to get skill more efficient meaning

73
07:04:10,000 --> 07:04:12,390
consuming less tokens. So what's their

74
07:04:12,400 --> 07:04:15,830
the options there? One is just change

75
07:04:15,840 --> 07:04:20,230
line by line and try brute force all the

76
07:04:20,240 --> 07:04:23,190
different combinations. Another would be

77
07:04:23,200 --> 07:04:25,510
to think through what makes sense. So

78
07:04:25,520 --> 07:04:29,270
maybe use something as the user feedback

79
07:04:29,280 --> 07:04:32,150
on the skills to see where the skills

80
07:04:32,160 --> 07:04:35,510
perform well. Uh there it's not. And

81
07:04:35,520 --> 07:04:38,868
with new models just running skill and

82
07:04:38,878 --> 07:04:41,510
seeing how it works, what's the options

83
07:04:41,520 --> 07:04:45,110
there? and how large those modifications

84
07:04:45,120 --> 07:04:47,270
should be just

85
07:04:47,280 --> 07:04:53,590
lines, sentences, ideas.

86
07:04:53,600 --> 07:04:57,350
So what we found to be useful is instead

87
07:04:57,360 --> 07:05:00,468
of relying on their luck, we can put

88
07:05:00,478 --> 07:05:03,670
more deterministic process here using

89
07:05:03,680 --> 07:05:06,310
idea of their items from their fact

90
07:05:06,320 --> 07:05:08,548
score. uh

91
07:05:08,558 --> 07:05:11,350
we can split their skill and

92
07:05:11,360 --> 07:05:16,310
instructions into those items and

93
07:05:16,320 --> 07:05:19,590
for each of those had the pair of

94
07:05:19,600 --> 07:05:22,708
question and answer that we expect.

95
07:05:22,718 --> 07:05:25,670
This could be either general ideas like

96
07:05:25,680 --> 07:05:28,468
proposition in this case or contracts

97
07:05:28,478 --> 07:05:32,230
there we put more definitive expectation

98
07:05:32,240 --> 07:05:37,110
on their action

99
07:05:37,120 --> 07:05:43,350
and in general we will be getting their

100
07:05:43,360 --> 07:05:46,390
some predefined amount of the items per

101
07:05:46,400 --> 07:05:50,070
skill and we will have let's say

102
07:05:50,080 --> 07:05:54,468
25 or 30 items arms per skill.

103
07:05:54,478 --> 07:05:57,510
Then for each of the items, we will

104
07:05:57,520 --> 07:06:00,868
again run two arms, one without skill

105
07:06:00,878 --> 07:06:03,430
and one with skill.

106
07:06:03,440 --> 07:06:07,270
And we will have judge or panel of

107
07:06:07,280 --> 07:06:09,910
judges that will be evaluating model

108
07:06:09,920 --> 07:06:11,670
answers. So we asking those questions to

109
07:06:11,680 --> 07:06:14,548
the model that we're interested in.

110
07:06:14,558 --> 07:06:17,910
We evalating those questions with the

111
07:06:17,920 --> 07:06:20,868
judge ideally different family of the

112
07:06:20,878 --> 07:06:24,390
models or panel of judges also ensures

113
07:06:24,400 --> 07:06:29,670
that the question do not leak answers.

114
07:06:29,680 --> 07:06:32,548
We depending on the case can compare

115
07:06:32,558 --> 07:06:35,830
with no tools or additional um

116
07:06:35,840 --> 07:06:37,670
information to check their internal

117
07:06:37,680 --> 07:06:40,310
model understanding or provided with

118
07:06:40,320 --> 07:06:43,030
their some tools like web access. So uh

119
07:06:43,040 --> 07:06:45,670
it would be more practical

120
07:06:45,680 --> 07:06:48,788
and we

121
07:06:48,798 --> 07:06:55,110
can get the simple four outcomes matrix

122
07:06:55,120 --> 07:06:57,350
basically where we are checking that the

123
07:06:57,360 --> 07:06:59,590
model

124
07:06:59,600 --> 07:07:03,110
uh knows or do not know something and

125
07:07:03,120 --> 07:07:05,830
how this maps to their skill impact on

126
07:07:05,840 --> 07:07:09,670
this. So the simplest option is where

127
07:07:09,680 --> 07:07:13,910
model was able to answer this particular

128
07:07:13,920 --> 07:07:16,150
questions even without skill or with

129
07:07:16,160 --> 07:07:18,150
skill. This means that skill have no

130
07:07:18,160 --> 07:07:20,310
impact on this and this was already part

131
07:07:20,320 --> 07:07:23,110
of the model's knowledge. The second one

132
07:07:23,120 --> 07:07:25,910
is where without skill actual model was

133
07:07:25,920 --> 07:07:28,708
able to answer this correctly

134
07:07:28,718 --> 07:07:32,868
and by adding skill we saw degradation

135
07:07:32,878 --> 07:07:35,830
in performance and model capability to

136
07:07:35,840 --> 07:07:40,230
talk or explain particular aspects.

137
07:07:40,240 --> 07:07:43,270
The ideal case is where without skill

138
07:07:43,280 --> 07:07:44,948
model wasn't able to answer something

139
07:07:44,958 --> 07:07:47,270
and with skill we start seeing that it

140
07:07:47,280 --> 07:07:49,510
can answer those new type of the

141
07:07:49,520 --> 07:07:51,110
questions and scenarios and this is what

142
07:07:51,120 --> 07:07:53,830
we were aiming for teach model something

143
07:07:53,840 --> 07:07:55,350
new

144
07:07:55,360 --> 07:07:57,670
but there also can be the fourth option

145
07:07:57,680 --> 07:08:03,430
where skills were written not ideally or

146
07:08:03,440 --> 07:08:08,230
in some maybe not clean way where model

147
07:08:08,240 --> 07:08:10,150
wasn't able to answer it either without

148
07:08:10,160 --> 07:08:12,628
skill or with skill. And while this

149
07:08:12,638 --> 07:08:14,390
topic might be relevant and important

150
07:08:14,400 --> 07:08:16,868
for the model, it still sort of misses

151
07:08:16,878 --> 07:08:21,110
some of those concepts.

152
07:08:21,120 --> 07:08:26,468
The simplest thing to fix are

153
07:08:26,478 --> 07:08:28,628
instructions or items that actually hurt

154
07:08:28,638 --> 07:08:31,670
performance. And

155
07:08:31,680 --> 07:08:34,868
for popular skills, those are criteria.

156
07:08:34,878 --> 07:08:37,350
So in the

157
07:08:37,360 --> 07:08:41,830
uh box below you see that it's just one

158
07:08:41,840 --> 07:08:45,430
case out of the mostly 400 items in

159
07:08:45,440 --> 07:08:48,390
entropic skills and others don't have

160
07:08:48,400 --> 07:08:51,990
those. But for their smaller skills that

161
07:08:52,000 --> 07:08:54,390
are less popular, you probably having

162
07:08:54,400 --> 07:08:55,830
more of those. This is what we're seeing

163
07:08:55,840 --> 07:08:59,830
with some of our experiments.

164
07:08:59,840 --> 07:09:01,510
And

165
07:09:01,520 --> 07:09:03,510
this is just straightforward the

166
07:09:03,520 --> 07:09:06,390
simplest thing to fix to ensure that

167
07:09:06,400 --> 07:09:08,788
your skill do not reduce model

168
07:09:08,798 --> 07:09:10,788
capabilities and performance and bring

169
07:09:10,798 --> 07:09:12,788
only relevant

170
07:09:12,798 --> 07:09:15,350
context there.

171
07:09:15,360 --> 07:09:17,510
And we also can actually color this. In

172
07:09:17,520 --> 07:09:20,390
this scenario, this is done by lines but

173
07:09:20,400 --> 07:09:22,310
actually going with it atoms. You might

174
07:09:22,320 --> 07:09:24,708
have their uh different coloring for a

175
07:09:24,718 --> 07:09:28,548
single line or a single sentence.

176
07:09:28,558 --> 07:09:30,868
And this just gives your example of

177
07:09:30,878 --> 07:09:34,310
their uh visualization where this would

178
07:09:34,320 --> 07:09:37,190
be helpful details and instructions in

179
07:09:37,200 --> 07:09:40,788
terms what new stuff you bring to the

180
07:09:40,798 --> 07:09:45,670
model and what is

181
07:09:45,680 --> 07:09:48,150
something that model already knows.

182
07:09:48,160 --> 07:09:49,670
Costwise

183
07:09:49,680 --> 07:09:51,590
we are running all those questions and

184
07:09:51,600 --> 07:09:53,590
asking model

185
07:09:53,600 --> 07:09:55,990
but still relatively this is uh cheap

186
07:09:56,000 --> 07:09:59,830
for 29 skills 14 bucks of their set and

187
07:09:59,840 --> 07:10:03,190
ha calls with their sonet used as a

188
07:10:03,200 --> 07:10:05,110
judge

189
07:10:05,120 --> 07:10:09,430
and

190
07:10:09,440 --> 07:10:12,868
there are couple ways how we also can

191
07:10:12,878 --> 07:10:16,310
check their strictness of their some

192
07:10:16,320 --> 07:10:20,788
knowledge. Um we have uh lang and strict

193
07:10:20,798 --> 07:10:23,350
ways of the checking this where first

194
07:10:23,360 --> 07:10:26,150
checks the general idea while the second

195
07:10:26,160 --> 07:10:30,390
one is checking for exact details and

196
07:10:30,400 --> 07:10:32,230
you can think about the difference

197
07:10:32,240 --> 07:10:33,510
between general knowledge and

198
07:10:33,520 --> 07:10:36,070
understanding of the topic to the

199
07:10:36,080 --> 07:10:38,548
understanding of the exact keys

200
07:10:38,558 --> 07:10:43,510
parameters and exact values or steps.

201
07:10:43,520 --> 07:10:46,628
And what we see across the models is the

202
07:10:46,638 --> 07:10:51,350
difference is large especially with the

203
07:10:51,360 --> 07:10:53,750
um larger frontier models where we see

204
07:10:53,760 --> 07:10:55,830
almost dropping from 70% understanding

205
07:10:55,840 --> 07:10:58,230
of their skill in case of the general

206
07:10:58,240 --> 07:11:02,230
understanding to just 22% of the

207
07:11:02,240 --> 07:11:05,190
specific details.

208
07:11:05,200 --> 07:11:07,990
And this is much our expectations. We

209
07:11:08,000 --> 07:11:10,628
see that new models are pretty well

210
07:11:10,638 --> 07:11:13,030
generalized to understand concepts but

211
07:11:13,040 --> 07:11:15,590
it's still helpful to provide their with

212
07:11:15,600 --> 07:11:17,590
exact details and data that will be

213
07:11:17,600 --> 07:11:23,830
useful for their solving those problems.

214
07:11:23,840 --> 07:11:28,150
Here is outlook of their um comparing

215
07:11:28,160 --> 07:11:30,230
those three popular repos with the

216
07:11:30,240 --> 07:11:33,990
skills across different models. So one

217
07:11:34,000 --> 07:11:36,468
of the interesting observations here is

218
07:11:36,478 --> 07:11:40,150
that smaller model have in general lower

219
07:11:40,160 --> 07:11:43,750
understanding of the

220
07:11:43,760 --> 07:11:45,430
skill

221
07:11:45,440 --> 07:11:47,750
with their larger sonet having better

222
07:11:47,760 --> 07:11:50,230
understanding which is

223
07:11:50,240 --> 07:11:53,030
why we have bigger models. But also we

224
07:11:53,040 --> 07:11:55,910
see that overall their models already

225
07:11:55,920 --> 07:12:01,030
understand a lot of their

226
07:12:01,040 --> 07:12:03,510
details and context of those skills. And

227
07:12:03,520 --> 07:12:05,110
you would expect that with the newer

228
07:12:05,120 --> 07:12:09,750
models and more advances in their uh

229
07:12:09,760 --> 07:12:11,590
better training capabilities we will

230
07:12:11,600 --> 07:12:14,788
have more and more

231
07:12:14,798 --> 07:12:17,110
uh in terms of the models understanding.

232
07:12:17,120 --> 07:12:19,110
But also we see that this pretty

233
07:12:19,120 --> 07:12:22,310
significantly differs skill to skill

234
07:12:22,320 --> 07:12:25,430
and it's not bad. Some skills are

235
07:12:25,440 --> 07:12:28,390
intended to be directional

236
07:12:28,400 --> 07:12:30,310
but it's also useful piece of

237
07:12:30,320 --> 07:12:33,750
information to see how those

238
07:12:33,760 --> 07:12:36,708
skills bring new knowledge uh to the

239
07:12:36,718 --> 07:12:38,390
different model and different families.

240
07:12:38,400 --> 07:12:40,548
And uh this should be run ideally in the

241
07:12:40,558 --> 07:12:47,990
broader set of the models.

242
07:12:48,000 --> 07:12:51,990
But just having their

243
07:12:52,000 --> 07:12:56,070
this novelty piece of information is

244
07:12:56,080 --> 07:12:57,670
just one of the signals. It's basically

245
07:12:57,680 --> 07:12:59,910
initial vector for optimization. At the

246
07:12:59,920 --> 07:13:02,070
end we still have to run evolves and

247
07:13:02,080 --> 07:13:05,990
validate that all the changes that we

248
07:13:06,000 --> 07:13:08,468
try make with their skill actually

249
07:13:08,478 --> 07:13:10,070
works.

250
07:13:10,080 --> 07:13:12,948
So the one of their simpler scenarios

251
07:13:12,958 --> 07:13:16,390
would be we can look on their skill and

252
07:13:16,400 --> 07:13:20,150
try to remove some or all of the already

253
07:13:20,160 --> 07:13:22,468
known pieces of information from the

254
07:13:22,478 --> 07:13:25,990
skill and run evalation.

255
07:13:26,000 --> 07:13:27,910
It might come up with their same

256
07:13:27,920 --> 07:13:29,990
scoring. Basically we will be getting

257
07:13:30,000 --> 07:13:31,830
their same results model still capable

258
07:13:31,840 --> 07:13:34,390
of solving those problems. Then great we

259
07:13:34,400 --> 07:13:37,910
can reduce skill size safe tokens safe

260
07:13:37,920 --> 07:13:39,830
context.

261
07:13:39,840 --> 07:13:42,628
And instead of there brute forcing and

262
07:13:42,638 --> 07:13:45,270
guessing what we can or should remove or

263
07:13:45,280 --> 07:13:49,430
add or how to fix their uh

264
07:13:49,440 --> 07:13:54,708
degradations, we can have a guidance.

265
07:13:54,718 --> 07:13:57,270
So all of those measurements experiments

266
07:13:57,280 --> 07:13:59,990
been done with the document which is a

267
07:14:00,000 --> 07:14:02,948
framework for the comparing

268
07:14:02,958 --> 07:14:04,788
everything related to the agents and

269
07:14:04,798 --> 07:14:06,390
context

270
07:14:06,400 --> 07:14:10,868
where it's is scale agents MD MCP tools

271
07:14:10,878 --> 07:14:14,230
whatever and novelty is one of their

272
07:14:14,240 --> 07:14:16,868
pieces of the functionality where you

273
07:14:16,878 --> 07:14:19,670
can get this initial vector for their

274
07:14:19,680 --> 07:14:21,830
skill optimization and evalation but

275
07:14:21,840 --> 07:14:25,350
there also uh evolves there Assessment

276
07:14:25,360 --> 07:14:27,430
of the context impact, routing,

277
07:14:27,440 --> 07:14:31,190
comparing the uh descriptions, naming,

278
07:14:31,200 --> 07:14:35,270
bunch of the things that related to the

279
07:14:35,280 --> 07:14:38,070
skills where it would be helpful to

280
07:14:38,080 --> 07:14:39,910
actually get there more deterministic

281
07:14:39,920 --> 07:14:42,868
relations to make conscious decisions in

282
07:14:42,878 --> 07:14:44,230
terms of what should be done with the

283
07:14:44,240 --> 07:14:47,270
skills.

284
07:14:47,280 --> 07:14:50,708
So, thank you for your time and please

285
07:14:50,718 --> 07:14:53,000
go ahead and write better skills.