# Your SKILL.md is 70% of Things the Model Already Knows

Source: https://www.youtube.com/watch?v=ZZ19zhYnDKY

Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session.

[07:00:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25253s) >> Hello, my name is Nikolai Petro and we

[07:00:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25255s) will be talking about skills and agents

[07:00:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25257s) knowledge of those skills.

[07:01:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25261s) We'll start it with the simple prompts

[07:01:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25263s) where we've been playing with the new

[07:01:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25266s) stuff, new work being done by the

[07:01:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25268s) agents, repeating those prompts over and

[07:01:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25270s) over again. And then we start packing

[07:01:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25272s) those into the skills,

[07:01:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25274s) creating repeatable, reproducible,

[07:01:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25277s) portable type of their functionality

[07:01:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25281s) or capabilities that we can teach agents

[07:01:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25285s) to work with. And some of those skills

[07:01:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25287s) are very important. hundreds of the

[07:01:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25288s) stars on GitHub.

[07:01:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25293s) There are different types of their

[07:01:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25294s) skills. Some teach models something new

[07:01:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25298s) bringing new knowledge or capabilities

[07:01:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25300s) to the model. But there are also types

[07:01:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25303s) of their just steering or process skills

[07:01:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25306s) where we

[07:01:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25308s) do not bring any additional information

[07:01:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25310s) but just getting repeatable

[07:01:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25313s) flow of the actions that we want agents

[07:01:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25315s) to perform. both are useful. In this

[07:01:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25319s) session, we will be talking about new

[07:02:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25321s) knowledge type of the skills and

[07:02:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25325s) we'll see what we can

[07:02:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25328s) get from the models understanding of

[07:02:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25330s) those skills.

[07:02:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25333s) So, we get our first skills packed,

[07:02:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25336s) start using them. Um, basically

[07:02:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25341s) starting with the the feel of does it

[07:02:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25344s) work or not. But there is also the more

[07:02:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25346s) technical way and more deterministic way

[07:02:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25348s) for the checking if skills are working

[07:02:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25350s) or not and what's the impacts of those

[07:02:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25353s) skills are evolves. There are different

[07:02:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25356s) formats where evolves can be defined.

[07:02:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25359s) But at the sense this is task definition

[07:02:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25361s) that you want your agent to be able to

[07:02:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25364s) solve.

[07:02:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25366s) You put it in the place ideally more

[07:02:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25368s) deterministic so we can uh validate it

[07:02:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25371s) and we can check two arms one there no

[07:02:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25376s) skill loaded and there second one where

[07:02:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25378s) we're loading the skill and providing

[07:02:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25379s) this additional information to the

[07:03:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25381s) context and we compare how the runs

[07:03:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25384s) there are multiple things one is how

[07:03:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25387s) many

[07:03:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25388s) evolves been solved in this case with no

[07:03:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25391s) skill we're solving three out of five

[07:03:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25394s) with skill we're solving all five

[07:03:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25396s) meaning that we get two more tasks.

[07:03:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25399s) There also would be implications on the

[07:03:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25401s) tokens usage or time that it took those

[07:03:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25405s) tasks to be completed and other factors.

[07:03:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25408s) But at the end we are comparing those

[07:03:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25410s) and seeing what makes sense, what not.

[07:03:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25412s) And this is one of the mains for

[07:03:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25415s) evaluating the skills and uh saying that

[07:03:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25418s) they not just nice ideas but actually

[07:03:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25421s) working pieces of instructions and

[07:03:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25423s) context for the agents.

[07:03:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25426s) But what next? We get to the stage where

[07:03:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25430s) we have their working skill. It solves

[07:03:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25432s) some evolves. But we know that there

[07:03:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25437s) should be a better way. We have models

[07:03:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25439s) evolving. We have u some

[07:04:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25443s) might be cases where we want

[07:04:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25447s) to get skill more efficient meaning

[07:04:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25450s) consuming less tokens. So what's their

[07:04:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25452s) the options there? One is just change

[07:04:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25455s) line by line and try brute force all the

[07:04:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25460s) different combinations. Another would be

[07:04:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25463s) to think through what makes sense. So

[07:04:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25465s) maybe use something as the user feedback

[07:04:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25469s) on the skills to see where the skills

[07:04:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25472s) perform well. Uh there it's not. And

[07:04:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25475s) with new models just running skill and

[07:04:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25478s) seeing how it works, what's the options

[07:04:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25481s) there? and how large those modifications

[07:04:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25485s) should be just

[07:04:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25487s) lines, sentences, ideas.

[07:04:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25493s) So what we found to be useful is instead

[07:04:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25497s) of relying on their luck, we can put

[07:05:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25500s) more deterministic process here using

[07:05:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25503s) idea of their items from their fact

[07:05:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25506s) score. uh

[07:05:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25508s) we can split their skill and

[07:05:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25511s) instructions into those items and

[07:05:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25516s) for each of those had the pair of

[07:05:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25519s) question and answer that we expect.

[07:05:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25522s) This could be either general ideas like

[07:05:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25525s) proposition in this case or contracts

[07:05:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25528s) there we put more definitive expectation

[07:05:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25532s) on their action

[07:05:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25537s) and in general we will be getting their

[07:05:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25543s) some predefined amount of the items per

[07:05:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25546s) skill and we will have let's say

[07:05:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25550s) 25 or 30 items arms per skill.

[07:05:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25554s) Then for each of the items, we will

[07:05:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25557s) again run two arms, one without skill

[07:06:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25560s) and one with skill.

[07:06:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25563s) And we will have judge or panel of

[07:06:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25567s) judges that will be evaluating model

[07:06:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25569s) answers. So we asking those questions to

[07:06:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25571s) the model that we're interested in.

[07:06:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25574s) We evalating those questions with the

[07:06:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25577s) judge ideally different family of the

[07:06:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25580s) models or panel of judges also ensures

[07:06:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25584s) that the question do not leak answers.

[07:06:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25589s) We depending on the case can compare

[07:06:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25592s) with no tools or additional um

[07:06:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25595s) information to check their internal

[07:06:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25597s) model understanding or provided with

[07:06:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25600s) their some tools like web access. So uh

[07:06:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25603s) it would be more practical

[07:06:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25605s) and we

[07:06:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25608s) can get the simple four outcomes matrix

[07:06:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25615s) basically where we are checking that the

[07:06:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25617s) model

[07:06:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25619s) uh knows or do not know something and

[07:07:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25623s) how this maps to their skill impact on

[07:07:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25625s) this. So the simplest option is where

[07:07:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25629s) model was able to answer this particular

[07:07:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25633s) questions even without skill or with

[07:07:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25636s) skill. This means that skill have no

[07:07:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25638s) impact on this and this was already part

[07:07:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25640s) of the model's knowledge. The second one

[07:07:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25643s) is where without skill actual model was

[07:07:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25645s) able to answer this correctly

[07:07:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25648s) and by adding skill we saw degradation

[07:07:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25652s) in performance and model capability to

[07:07:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25655s) talk or explain particular aspects.

[07:07:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25660s) The ideal case is where without skill

[07:07:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25663s) model wasn't able to answer something

[07:07:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25664s) and with skill we start seeing that it

[07:07:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25667s) can answer those new type of the

[07:07:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25669s) questions and scenarios and this is what

[07:07:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25671s) we were aiming for teach model something

[07:07:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25673s) new

[07:07:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25675s) but there also can be the fourth option

[07:07:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25677s) where skills were written not ideally or

[07:08:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25683s) in some maybe not clean way where model

[07:08:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25688s) wasn't able to answer it either without

[07:08:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25690s) skill or with skill. And while this

[07:08:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25692s) topic might be relevant and important

[07:08:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25694s) for the model, it still sort of misses

[07:08:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25696s) some of those concepts.

[07:08:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25701s) The simplest thing to fix are

[07:08:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25706s) instructions or items that actually hurt

[07:08:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25708s) performance. And

[07:08:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25711s) for popular skills, those are criteria.

[07:08:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25714s) So in the

[07:08:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25717s) uh box below you see that it's just one

[07:08:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25721s) case out of the mostly 400 items in

[07:08:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25725s) entropic skills and others don't have

[07:08:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25728s) those. But for their smaller skills that

[07:08:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25732s) are less popular, you probably having

[07:08:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25734s) more of those. This is what we're seeing

[07:08:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25735s) with some of our experiments.

[07:08:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25739s) And

[07:09:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25741s) this is just straightforward the

[07:09:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25743s) simplest thing to fix to ensure that

[07:09:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25746s) your skill do not reduce model

[07:09:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25748s) capabilities and performance and bring

[07:09:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25750s) only relevant

[07:09:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25752s) context there.

[07:09:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25755s) And we also can actually color this. In

[07:09:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25757s) this scenario, this is done by lines but

[07:09:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25760s) actually going with it atoms. You might

[07:09:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25762s) have their uh different coloring for a

[07:09:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25764s) single line or a single sentence.

[07:09:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25768s) And this just gives your example of

[07:09:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25770s) their uh visualization where this would

[07:09:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25774s) be helpful details and instructions in

[07:09:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25777s) terms what new stuff you bring to the

[07:09:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25780s) model and what is

[07:09:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25785s) something that model already knows.

[07:09:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25788s) Costwise

[07:09:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25789s) we are running all those questions and

[07:09:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25791s) asking model

[07:09:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25793s) but still relatively this is uh cheap

[07:09:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25796s) for 29 skills 14 bucks of their set and

[07:09:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25799s) ha calls with their sonet used as a

[07:10:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25803s) judge

[07:10:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25805s) and

[07:10:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25809s) there are couple ways how we also can

[07:10:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25812s) check their strictness of their some

[07:10:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25816s) knowledge. Um we have uh lang and strict

[07:10:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25820s) ways of the checking this where first

[07:10:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25823s) checks the general idea while the second

[07:10:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25826s) one is checking for exact details and

[07:10:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25830s) you can think about the difference

[07:10:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25832s) between general knowledge and

[07:10:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25833s) understanding of the topic to the

[07:10:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25836s) understanding of the exact keys

[07:10:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25838s) parameters and exact values or steps.

[07:10:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25843s) And what we see across the models is the

[07:10:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25846s) difference is large especially with the

[07:10:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25851s) um larger frontier models where we see

[07:10:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25853s) almost dropping from 70% understanding

[07:10:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25855s) of their skill in case of the general

[07:10:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25858s) understanding to just 22% of the

[07:11:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25862s) specific details.

[07:11:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25865s) And this is much our expectations. We

[07:11:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25868s) see that new models are pretty well

[07:11:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25870s) generalized to understand concepts but

[07:11:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25873s) it's still helpful to provide their with

[07:11:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25875s) exact details and data that will be

[07:11:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25877s) useful for their solving those problems.

[07:11:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25883s) Here is outlook of their um comparing

[07:11:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25888s) those three popular repos with the

[07:11:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25890s) skills across different models. So one

[07:11:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25894s) of the interesting observations here is

[07:11:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25896s) that smaller model have in general lower

[07:11:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25900s) understanding of the

[07:11:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25903s) skill

[07:11:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25905s) with their larger sonet having better

[07:11:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25907s) understanding which is

[07:11:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25910s) why we have bigger models. But also we

[07:11:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25913s) see that overall their models already

[07:11:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25915s) understand a lot of their

[07:12:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25921s) details and context of those skills. And

[07:12:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25923s) you would expect that with the newer

[07:12:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25925s) models and more advances in their uh

[07:12:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25929s) better training capabilities we will

[07:12:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25931s) have more and more

[07:12:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25934s) uh in terms of the models understanding.

[07:12:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25937s) But also we see that this pretty

[07:12:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25939s) significantly differs skill to skill

[07:12:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25942s) and it's not bad. Some skills are

[07:12:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25945s) intended to be directional

[07:12:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25948s) but it's also useful piece of

[07:12:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25950s) information to see how those

[07:12:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25953s) skills bring new knowledge uh to the

[07:12:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25956s) different model and different families.

[07:12:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25958s) And uh this should be run ideally in the

[07:12:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25960s) broader set of the models.

[07:12:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25968s) But just having their

[07:12:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25972s) this novelty piece of information is

[07:12:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25976s) just one of the signals. It's basically

[07:12:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25977s) initial vector for optimization. At the

[07:12:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25979s) end we still have to run evolves and

[07:13:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25982s) validate that all the changes that we

[07:13:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25986s) try make with their skill actually

[07:13:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25988s) works.

[07:13:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25990s) So the one of their simpler scenarios

[07:13:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25992s) would be we can look on their skill and

[07:13:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25996s) try to remove some or all of the already

[07:13:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26000s) known pieces of information from the

[07:13:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26002s) skill and run evalation.

[07:13:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26006s) It might come up with their same

[07:13:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26007s) scoring. Basically we will be getting

[07:13:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26010s) their same results model still capable

[07:13:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26011s) of solving those problems. Then great we

[07:13:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26014s) can reduce skill size safe tokens safe

[07:13:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26017s) context.

[07:13:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26019s) And instead of there brute forcing and

[07:13:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26022s) guessing what we can or should remove or

[07:13:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26025s) add or how to fix their uh

[07:13:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26029s) degradations, we can have a guidance.

[07:13:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26034s) So all of those measurements experiments

[07:13:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26037s) been done with the document which is a

[07:14:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26040s) framework for the comparing

[07:14:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26042s) everything related to the agents and

[07:14:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26044s) context

[07:14:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26046s) where it's is scale agents MD MCP tools

[07:14:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26050s) whatever and novelty is one of their

[07:14:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26054s) pieces of the functionality where you

[07:14:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26056s) can get this initial vector for their

[07:14:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26059s) skill optimization and evalation but

[07:14:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26061s) there also uh evolves there Assessment

[07:14:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26065s) of the context impact, routing,

[07:14:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26067s) comparing the uh descriptions, naming,

[07:14:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26071s) bunch of the things that related to the

[07:14:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26075s) skills where it would be helpful to

[07:14:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26078s) actually get there more deterministic

[07:14:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26079s) relations to make conscious decisions in

[07:14:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26082s) terms of what should be done with the

[07:14:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26084s) skills.

[07:14:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26087s) So, thank you for your time and please

[07:14:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=26090s) go ahead and write better skills.