Your SKILL.md is 70% of Things the Model Already Knows

7:00:53>> Hello, my name is Nikolai Petro and we

7:00:55will be talking about skills and agents

7:00:57knowledge of those skills.

7:01:01We'll start it with the simple prompts

7:01:03where we've been playing with the new

7:01:06stuff, new work being done by the

7:01:08agents, repeating those prompts over and

7:01:10over again. And then we start packing

7:01:12those into the skills,

7:01:14creating repeatable, reproducible,

7:01:17portable type of their functionality

7:01:21or capabilities that we can teach agents

7:01:25to work with. And some of those skills

7:01:27are very important. hundreds of the

7:01:28stars on GitHub.

7:01:33There are different types of their

7:01:34skills. Some teach models something new

7:01:38bringing new knowledge or capabilities

7:01:40to the model. But there are also types

7:01:43of their just steering or process skills

7:01:46where we

7:01:48do not bring any additional information

7:01:50but just getting repeatable

7:01:53flow of the actions that we want agents

7:01:55to perform. both are useful. In this

7:01:59session, we will be talking about new

7:02:01knowledge type of the skills and

7:02:05we'll see what we can

7:02:08get from the models understanding of

7:02:10those skills.

7:02:13So, we get our first skills packed,

7:02:16start using them. Um, basically

7:02:21starting with the the feel of does it

7:02:24work or not. But there is also the more

7:02:26technical way and more deterministic way

7:02:28for the checking if skills are working

7:02:30or not and what's the impacts of those

7:02:33skills are evolves. There are different

7:02:36formats where evolves can be defined.

7:02:39But at the sense this is task definition

7:02:41that you want your agent to be able to

7:02:44solve.

7:02:46You put it in the place ideally more

7:02:48deterministic so we can uh validate it

7:02:51and we can check two arms one there no

7:02:56skill loaded and there second one where

7:02:58we're loading the skill and providing

7:02:59this additional information to the

7:03:01context and we compare how the runs

7:03:04there are multiple things one is how

7:03:07many

7:03:08evolves been solved in this case with no

7:03:11skill we're solving three out of five

7:03:14with skill we're solving all five

7:03:16meaning that we get two more tasks.

7:03:19There also would be implications on the

7:03:21tokens usage or time that it took those

7:03:25tasks to be completed and other factors.

7:03:28But at the end we are comparing those

7:03:30and seeing what makes sense, what not.

7:03:32And this is one of the mains for

7:03:35evaluating the skills and uh saying that

7:03:38they not just nice ideas but actually

7:03:41working pieces of instructions and

7:03:43context for the agents.

7:03:46But what next? We get to the stage where

7:03:50we have their working skill. It solves

7:03:52some evolves. But we know that there

7:03:57should be a better way. We have models

7:03:59evolving. We have u some

7:04:03might be cases where we want

7:04:07to get skill more efficient meaning

7:04:10consuming less tokens. So what's their

7:04:12the options there? One is just change

7:04:15line by line and try brute force all the

7:04:20different combinations. Another would be

7:04:23to think through what makes sense. So

7:04:25maybe use something as the user feedback

7:04:29on the skills to see where the skills

7:04:32perform well. Uh there it's not. And

7:04:35with new models just running skill and

7:04:38seeing how it works, what's the options

7:04:41there? and how large those modifications

7:04:45should be just

7:04:47lines, sentences, ideas.

7:04:53So what we found to be useful is instead

7:04:57of relying on their luck, we can put

7:05:00more deterministic process here using

7:05:03idea of their items from their fact

7:05:06score. uh

7:05:08we can split their skill and

7:05:11instructions into those items and

7:05:16for each of those had the pair of

7:05:19question and answer that we expect.

7:05:22This could be either general ideas like

7:05:25proposition in this case or contracts

7:05:28there we put more definitive expectation

7:05:32on their action

7:05:37and in general we will be getting their

7:05:43some predefined amount of the items per

7:05:46skill and we will have let's say

7:05:5025 or 30 items arms per skill.

7:05:54Then for each of the items, we will

7:05:57again run two arms, one without skill

7:06:00and one with skill.

7:06:03And we will have judge or panel of

7:06:07judges that will be evaluating model

7:06:09answers. So we asking those questions to

7:06:11the model that we're interested in.

7:06:14We evalating those questions with the

7:06:17judge ideally different family of the

7:06:20models or panel of judges also ensures

7:06:24that the question do not leak answers.

7:06:29We depending on the case can compare

7:06:32with no tools or additional um

7:06:35information to check their internal

7:06:37model understanding or provided with

7:06:40their some tools like web access. So uh

7:06:43it would be more practical

7:06:45and we

7:06:48can get the simple four outcomes matrix

7:06:55basically where we are checking that the

7:06:57model

7:06:59uh knows or do not know something and

7:07:03how this maps to their skill impact on

7:07:05this. So the simplest option is where

7:07:09model was able to answer this particular

7:07:13questions even without skill or with

7:07:16skill. This means that skill have no

7:07:18impact on this and this was already part

7:07:20of the model's knowledge. The second one

7:07:23is where without skill actual model was

7:07:25able to answer this correctly

7:07:28and by adding skill we saw degradation

7:07:32in performance and model capability to

7:07:35talk or explain particular aspects.

7:07:40The ideal case is where without skill

7:07:43model wasn't able to answer something

7:07:44and with skill we start seeing that it

7:07:47can answer those new type of the

7:07:49questions and scenarios and this is what

7:07:51we were aiming for teach model something

7:07:53new

7:07:55but there also can be the fourth option

7:07:57where skills were written not ideally or

7:08:03in some maybe not clean way where model

7:08:08wasn't able to answer it either without

7:08:10skill or with skill. And while this

7:08:12topic might be relevant and important

7:08:14for the model, it still sort of misses

7:08:16some of those concepts.

7:08:21The simplest thing to fix are

7:08:26instructions or items that actually hurt

7:08:28performance. And

7:08:31for popular skills, those are criteria.

7:08:34So in the

7:08:37uh box below you see that it's just one

7:08:41case out of the mostly 400 items in

7:08:45entropic skills and others don't have

7:08:48those. But for their smaller skills that

7:08:52are less popular, you probably having

7:08:54more of those. This is what we're seeing

7:08:55with some of our experiments.

7:08:59And

7:09:01this is just straightforward the

7:09:03simplest thing to fix to ensure that

7:09:06your skill do not reduce model

7:09:08capabilities and performance and bring

7:09:10only relevant

7:09:12context there.

7:09:15And we also can actually color this. In

7:09:17this scenario, this is done by lines but

7:09:20actually going with it atoms. You might

7:09:22have their uh different coloring for a

7:09:24single line or a single sentence.

7:09:28And this just gives your example of

7:09:30their uh visualization where this would

7:09:34be helpful details and instructions in

7:09:37terms what new stuff you bring to the

7:09:40model and what is

7:09:45something that model already knows.

7:09:48Costwise

7:09:49we are running all those questions and

7:09:51asking model

7:09:53but still relatively this is uh cheap

7:09:56for 29 skills 14 bucks of their set and

7:09:59ha calls with their sonet used as a

7:10:03judge

7:10:05and

7:10:09there are couple ways how we also can

7:10:12check their strictness of their some

7:10:16knowledge. Um we have uh lang and strict

7:10:20ways of the checking this where first

7:10:23checks the general idea while the second

7:10:26one is checking for exact details and

7:10:30you can think about the difference

7:10:32between general knowledge and

7:10:33understanding of the topic to the

7:10:36understanding of the exact keys

7:10:38parameters and exact values or steps.

7:10:43And what we see across the models is the

7:10:46difference is large especially with the

7:10:51um larger frontier models where we see

7:10:53almost dropping from 70% understanding

7:10:55of their skill in case of the general

7:10:58understanding to just 22% of the

7:11:02specific details.

7:11:05And this is much our expectations. We

7:11:08see that new models are pretty well

7:11:10generalized to understand concepts but

7:11:13it's still helpful to provide their with

7:11:15exact details and data that will be

7:11:17useful for their solving those problems.

7:11:23Here is outlook of their um comparing

7:11:28those three popular repos with the

7:11:30skills across different models. So one

7:11:34of the interesting observations here is

7:11:36that smaller model have in general lower

7:11:40understanding of the

7:11:43skill

7:11:45with their larger sonet having better

7:11:47understanding which is

7:11:50why we have bigger models. But also we

7:11:53see that overall their models already

7:11:55understand a lot of their

7:12:01details and context of those skills. And

7:12:03you would expect that with the newer

7:12:05models and more advances in their uh

7:12:09better training capabilities we will

7:12:11have more and more

7:12:14uh in terms of the models understanding.

7:12:17But also we see that this pretty

7:12:19significantly differs skill to skill

7:12:22and it's not bad. Some skills are

7:12:25intended to be directional

7:12:28but it's also useful piece of

7:12:30information to see how those

7:12:33skills bring new knowledge uh to the

7:12:36different model and different families.

7:12:38And uh this should be run ideally in the

7:12:40broader set of the models.

7:12:48But just having their

7:12:52this novelty piece of information is

7:12:56just one of the signals. It's basically

7:12:57initial vector for optimization. At the

7:12:59end we still have to run evolves and

7:13:02validate that all the changes that we

7:13:06try make with their skill actually

7:13:08works.

7:13:10So the one of their simpler scenarios

7:13:12would be we can look on their skill and

7:13:16try to remove some or all of the already

7:13:20known pieces of information from the

7:13:22skill and run evalation.

7:13:26It might come up with their same

7:13:27scoring. Basically we will be getting

7:13:30their same results model still capable

7:13:31of solving those problems. Then great we

7:13:34can reduce skill size safe tokens safe

7:13:37context.

7:13:39And instead of there brute forcing and

7:13:42guessing what we can or should remove or

7:13:45add or how to fix their uh

7:13:49degradations, we can have a guidance.

7:13:54So all of those measurements experiments

7:13:57been done with the document which is a

7:14:00framework for the comparing

7:14:02everything related to the agents and

7:14:04context

7:14:06where it's is scale agents MD MCP tools

7:14:10whatever and novelty is one of their

7:14:14pieces of the functionality where you

7:14:16can get this initial vector for their

7:14:19skill optimization and evalation but

7:14:21there also uh evolves there Assessment

7:14:25of the context impact, routing,

7:14:27comparing the uh descriptions, naming,

7:14:31bunch of the things that related to the

7:14:35skills where it would be helpful to

7:14:38actually get there more deterministic

7:14:39relations to make conscious decisions in

7:14:42terms of what should be done with the

7:14:44skills.

7:14:47So, thank you for your time and please

7:14:50go ahead and write better skills.