# Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel Nikolay Petrov, Intel [Talk page](https://ai.engineer/talks/yzx4IuEh_tU-your-skill-md-is-70-stuff-model) · [Video](https://www.youtube.com/watch?v=yzx4IuEh_tU) ## Your SKILL.md Is 70% Stuff the Model Already Knows In the first skill Nikolay Petrov’s team tested, the model already knew 70% of the material. Testing individual claims helps an editor find the details worth keeping. Running the original tasks shows whether the shorter file still works. ### What bought the last two task passes? [0:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=0s) A prompt you keep typing is a good candidate for a skill. Pack the instructions into a file and another agent can use them, too. Nikolay Petrov of Intel distinguishes two reasons to do this. A process skill gets an agent to perform familiar actions in a repeatable order. A knowledge skill supplies information the agent needs to solve a problem. For that second kind, it makes sense to ask how much of the file the model already knows. Start with tasks whose results you can check. Give the agent the same tasks in two conditions, once without the skill and once with its instructions in context. In Petrov’s example, the agent solves three of five tasks without the file and all five with it. Loading the file buys two additional successes. Token use and time to completion are separate costs you can compare across the runs. Now suppose you want to shorten the file or move it to a newer model. The five passes tell you the skill works as written. Which instructions earned their place? User feedback can point to a task that went badly. Trying combinations of line edits can narrow the search, at the price of running each combination. There’s also a choice about what to edit. A line, a sentence and an idea can be three different units. ### Give each claim a question and an expected answer [4:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=240s) Petrov borrows an idea from **FActScore**: split the text into claims small enough to check individually. In a skill, an item might express a proposition about how something works. It might also specify what action the agent should take, which Petrov calls a contract. For each item, write a question and the answer you expect. He gives 25 or 30 items as an example budget for one skill. Who gets to see which information? The target model gets the same question in both runs. Only one run gets the skill. A judge, ideally from a different model family, or a panel of judges grades each response against the expected answer. Keep that answer out of the question itself. If the question gives away the detail, even the run without the skill can get credit for repeating it. Tool access changes the meaning of a correct answer. With tools and other information withheld, the baseline probes what the model can supply on its own. Give both runs web access and the comparison answers a practical question about a working agent. The baseline can now succeed by looking something up. In either setting, each claim gets a pair of grades, while the input that varies is the whole skill. Finding out what happens when you delete a particular sentence takes another experiment. Which inputs reach the model, and which reach the judge? Follow the question through both runs. The skill enters only one branch. The branches meet at grading, where the same expected answer applies to both responses. - Item question — Ask about one claim or required action. - Skill text — The full instructions available as context. - Run without skill — The target model answers the item question. - Run with skill — The same model answers with the skill loaded. - Expected answer — Defines what counts as correct for this item. - Judge or panel — Grades each response to produce the pair of results. - Item question → Run without skill (same question) - Item question → Run with skill (same question) - Skill text → Run with skill (additional context) - Run without skill → Judge or panel (answer without skill) - Run with skill → Judge or panel (answer with skill) - Expected answer → Judge or panel (grading reference) ### Four ways a single item can score [5:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=355s) A pass with the skill loaded could mean the instructions taught the model something. It could also mean the model could already answer the question. Read the baseline grade first, then the grade with the skill. - **Pass → pass.** The model could already give an acceptable answer. The corresponding passage is a candidate for shortening. - **Pass → fail.** Loading the skill spoiled a previously correct answer. Inspect the instructions associated with that regression. - **Fail → pass.** With the skill, the model can answer a question it previously missed. Preserve the detail that enabled the answer. - **Fail → fail.** The model still misses the concept with the instructions in hand. If the task needs it, check whether the wording makes the expected answer or action clear. ### Put the grades back on the lines they describe [7:28](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=448s) Petrov starts with regressions. Here the model has demonstrated that it can answer the question, and loading the skill makes it stumble. These cases were rare in the popular skills tested. In one collection, only one item out of roughly 400 showed a regression. Even that single item gives the editor a specific passage to investigate. Where in the file did the skill supply something useful? Petrov’s example colors the source lines by outcome so an editor can locate passages that helped and passages the model already understood. The distinction can be finer than the display suggests. Two claims in one sentence can earn different colors. Deleting the whole sentence could take the one needed detail with it. Each question costs model calls in both conditions, plus grading. Petrov reports spending about $14 across 29 skills in a run using **Haiku** and **Sonnet**, with Sonnet as judge. ### Knowing the idea and supplying the right key [9:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=556s) How precise does an answer have to be to pass? Lenient grading accepts the general idea. Strict grading checks the details the task calls for, such as a particular key, parameter, value or sequence of steps. An answer can explain what a parameter does and still leave the agent without the value it needs to use. In the larger-model comparison Petrov highlights, the model met the general-idea standard on 70% of the material. When the judge required the specific details, the score fell to 22%. That gap explains the risk in shortening a skill by eye. The introductory explanation may be easy for the model to reconstruct. A key or value that barely registers in a summary may be the part the agent needs from the file. ### The same file can teach two models different amounts [10:30](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=630s) How well does a result for one model carry over to another? Across three popular skill repositories, the smaller model generally knew less of the material than the larger Sonnet model, though both already understood a substantial amount. A passage that looks redundant in a Sonnet run may still supply something the smaller model needs. The recording’s description places the headline’s 70% figure on the first skill the team tested. Results varied considerably from skill to skill, and the purpose of each file matters. A process skill can do its job by getting the agent to choose a particular workflow even when every action is familiar. For skills meant to teach, Petrov would run the comparison across a broader set of models and model families before deciding which details to remove. ### Try the deletion on the original tasks [11:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=715s) A model can explain an idea when asked directly and still benefit from a reminder while doing the work. Petrov proposes a deletion experiment. Remove some or all of the material the model could explain unaided, load the shorter skill, and rerun the original task evaluations. Compare which problems the agent solves with each version. If it still solves the same problems, the shorter file saves input tokens and leaves more room in context. The 70% knowledge result identifies candidates for that experiment. How much text can go depends on the task results. The team ran the experiments in a framework for comparing how agents use context, including skills, `AGENTS.md` instructions and MCP tools. It also supports task evaluations, routing checks and comparisons of names and descriptions. Those checks address different decisions. A description can affect which skill the agent selects. Once the skill is loaded, the item questions locate details worth editing, and the task evaluation shows whether the agent can still finish the job. ## Key takeaways - Ask one question per claim in both conditions. Keeping the expected answer out of the question makes the baseline informative. - Grade at the precision the task requires. In the highlighted comparison, 70% under a general-idea check fell to 22% when the judge required specific details. - Use the paired grades to choose a deletion. Test its effect by comparing the shortened file with the original on the tasks the skill was written to solve. ## Slides ### Your SKILL.md is 70% of Things the Model Already Knows Petrov quizzes the model on every line of a SKILL.md to find which instructions it already knows. That raises a second question: which of those familiar lines can you cut? [0:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=0s) [The headline highlights “Already Knows” in blue. The subtitle introduces a line-by-line quiz to find what the model knows and which instructions can be cut.](https://assets.ai.engineer/sites/aiecode2025/sha256/c75a4b097888009014e5ef38575eee8b401f86b915b83a060ab95ef9d68b9d6c.webp) Your SKILL.md is 70% of Things the Model Already Knows — AIE CODE SUMMIT 2026 · NIKOLAY PETROV We quiz the model on every line of a skill: which lines are new to it, which it already knows, and which of those you can really cut. Interface text: OVERVIEW 01 / 14 ### We packaged how we work into skills Packaging repeated chat instructions in a skill lets agents reuse the same steps across sessions. An agent loads the PDF skill when a task matches its description and reads forms.md only when needed. [0:07](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=7s) [Arrows connect repeated chat instructions to a skill folder and then to agents that load it for matching tasks. Examples below cover PDF processing, test-driven development, and interviewing a user about a plan.](https://assets.ai.engineer/sites/aiecode2025/sha256/3633cb173d882aedf0c2927800954b0696b14b6ec834c5077a88fe852bcfb44a.webp) — 01 · SKILLS We packaged how we work into skills Before In your head, in the chat The same steps, typed again every session, a little different each time → A skill pdf/ ├─ SKILL.md when + how ├─ scripts/ code it runs └─ forms.md read on demand → After Every agent, every session Loaded only when the task matches its description anthropics/skills · ★ 188k pdf Fill forms, extract tables obra/superpowers · ★ 314k test-driven-development Red, green, refactor mattpocock/skills · ★ 317k grilling Interview me about my plan Interface text: OVERVIEW 02 / 14 ### Some skills teach, some steer A teaching skill supplies details like the format of a grilling round. Current models already know 96% of the test-driven-development skill, yet it can still help them follow the steps in order. [0:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=37s) [The teaching column lists project-specific facts such as formats, names, and thresholds. The steering column uses test-driven development to illustrate how a skill sets the order of familiar steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/862be750be6431a7d240fd715291f23bcbf404d909a49aa770e2a52c34309dfd.webp) — 02 · TWO KINDS OF SKILLS Some skills teach, some steer Teach: new knowledge Facts the model can't guess Your formats, names, thresholds, project rules e.g. grilling's exact round format Steer: process Order for things it already knows Follow the cycle, don't skip a step e.g. test-driven-development: 96% already known by current models This talk measures knowledge. For a steering skill, "already known" is expected and fine: knowing the rule is not following it. Interface text: OVERVIEW 03 / 14 ### A skill is just text. An eval checks whether it improves the work. When asked to grill a user about moving CI to self-hosted runners, the agent numbered its questions and waited for answers only with the skill. Across five tasks, it passed every check on 3/5 without the skill and 5/5 with it. [1:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=78s) [An example checks whether the agent numbers questions, recommends an answer for each, and waits for replies. Across five tasks, the bars show all checks passing on three without the skill and five with it.](https://assets.ai.engineer/sites/aiecode2025/sha256/f8589c0b52ca1c0ee67a6a737ff070b636fa6c8ecc75771c7658935fb765cbb3.webp) — 03 · DOES IT WORK? A skill is just text. An eval checks whether it improves the work. One eval case task: "Grill me on moving our CI to self-hosted runners" run: agent without the skill, then agent with it checks: questions numbered? ✗ ✓ a recommendation each? ✓ ✓ waits for my answers? ✗ ✓ Checks = what the skill promises, as pass/fail. You write them, or a model drafts them from the skill and you review. 5 such tasks, all checks pass? no skill 3/5 with skill 5/5 Δ +2 tasks · plus tokens and time from the trace Tests Code runs: tests pass or fail Judge model A model checks the output against a rubric: the written checks Trace Tools called, tokens, time Interface text: OVERVIEW 04 / 14 ### It works. Now, how do we make the skill better? The whole skill scores 5/5. Which lines earned that score? Testing 30 lines one at a time takes 30 eval runs. If two lines give the same instruction, removing either one leaves the score unchanged. [2:51](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=171s) [Question marks beside individual lines of SKILL.md accompany four editing decisions: cut, keep, fix, and re-check. A callout explains that removing either of two equivalent lines alone can leave the score unchanged.](https://assets.ai.engineer/sites/aiecode2025/sha256/62c56f17318dd476ded45c54a8fe8fdad82c1b7f42487379aa89df8043bbe33d.webp) 04 · THE PROBLEM It works. Now, how do we make the skill better? Eval: 5/5 ✓ for the whole file SKILL.md ? ? ? ? ? ? ? Which part did the work? A line? A sentence? An idea? • Cut: what does the model already know? • Keep: what must stay word for word? • Fix: what confuses it? • Re-check: when a new model ships, what does it know now? Brute force: delete a line, rerun the whole eval. 30 eval runs, one per line And lines interact Two lines can say the same thing: cut either alone, nothing changes Interface text: OVERVIEW 05 / 14 ### Step 1 · Split into atoms One line of the grilling skill becomes four quiz questions, each with an expected answer. Two test ideas about rounds and which decisions are ready. Two test rules for asking every question in the round and waiting for answers. [3:58](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=238s) [Four numbered highlights in one skill passage map to four question-and-answer cards. Two propositions describe rounds and eligible decisions. Two contracts specify how to ask the questions and when to wait for answers.](https://assets.ai.engineer/sites/aiecode2025/sha256/9825d20773c77d0f79eece7dad4c95ab45b2144d7e8f282ab2d433ec919428b2.webp) — 05 · STEP 1 · SPLIT INTO ATOMS To quiz the model, split the skill into atoms: one checkable fact each grilling/SKILL.md, line 8 → four atoms Work the tree ① in rounds. ② The frontier is every decision whose prerequisites are settled. ③ Ask the whole frontier in one round: number each question and give your recommended answer. ④ Then wait for the user's answers before the next round. ① proposition Q: In what units is the tree worked? A: In rounds ② proposition Q: What is the "frontier"? A: Decisions whose prerequisites are settled ③ contract Q: What is asked each round, and how? A: All of it, numbered, each with a recommendation ④ contract Q: What before the next round? A: Wait for the answers Proposition an idea or advice Contract an exact rule: a format, name, number or order · mattpocock: 367 + 245 of 661 atoms Atoms: idea from FActScore (Min et al., EMNLP 2023): it splits a model's answer, we split the skill Interface text: OVERVIEW 06 / 14 ### Testing each question with and without the skill The model answers the same question twice, once from memory and once with SKILL.md. A judge grades one answer at a time against the same key. Repeating each call shows how much the answers vary. [4:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=297s) [The same question goes to the model twice, once alone and once with the skill file. A judge sees the question, the answer key and one response at a time.](https://assets.ai.engineer/sites/aiecode2025/sha256/a54197a3184e5ff3f6584ffe0c2b2c8c23305a2d2dbe67119ab15be7df29e0aa.webp) — 05 · STEP 2 · ASK TWICE, JUDGE BOTH Ask each atom without and with the skill; a judge grades both Atom ③ · question "What is asked each round, and how?" Answer key "All of it, numbered, each with a recommendation." Without the skill question only With the skill SKILL.md + question Judge Sees question, key, one answer Picks a verdict: complete, missing a detail, partial, wrong one model or a panel Fair questions The question never contains the answer. No tools: we test memory, not search; the eval covers real runs. Fair grading Repeat each call to measure noise. Judge from another model family. Interface text: OVERVIEW 07 / 14 ### Mapping quiz results back to skill-file lines Each quiz result colours its source line. In the grilling skill, green marks rules new to both models, including how to word a question so “yes” accepts the recommendation. Grey marks interviewing advice the models already knew. [8:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=500s) [Green highlights identify newly learned rules about round format, numbering and how to word a recommendation. Grey marks familiar interviewing advice. Side panels report a cost of $14 for 5,360 calls across 29 skills and show that a 60% estimate from 25 atoms has a 40–80% interval.](https://assets.ai.engineer/sites/aiecode2025/sha256/ed0f6aa5b1dbad23be5a8dedc370d8e4d15f419a5331f4d9040a664b0bc6ebb8.webp) 05 · STEP 4 · MAP IT BACK Each atom points to its line: colour the lines grilling/SKILL.md mattpocock/skills · without the skill Haiku 5.5 knew 31% and Sonnet 5.5 62% of 16 probed facts · 25% new to both Known to both models · Known to one model only · New to both · Hurts · No probed fact 1 --- 2 name: grilling 3 description: Grill the user relentlessly about a plan, decision, or idea. Use when the user wants to stress-test their thinking, or uses any 'grill' trigger phrases. 4 --- 5 6 Interview the user relentlessly until you reach a shared understanding. Map this as a **design tree**: every decision branches into the decisions that hang off it. 7 8 Work the tree in **rounds**. The **frontier** is every decision whose prerequisites are already settled: the questions you can ask _now_ without guessing at answers you haven't heard yet. Ask the whole frontier in one round: number each question and give your recommended answer. Then wait for the user's answers before the next round. 9 10 Format a round like so: 11 12 ``` 13 ❓ **Q1** - ****: 14 15 ➡️ 16 17 --- 18 19 ❓ **Q2** - ****: 20 21 ➡️ 22 ``` 23 24 Word each question so "yes" accepts your recommended answer. 25 26 Each round the user answers reshapes the tree: settled decisions push the frontier outward and unblock questions that depended on them. Recompute the frontier and ask the next round. A question whose answer depends on another question still open in this round belongs to a _later_ round, not this one. 27 28 Finding _facts_ is your job, never the user's. When a frontier question needs a fact from the environment (filesystem, tools, etc.), dispatch a sub-agent with a fresh context window to find it; don't ask the user for anything you could look up yourself. Don't block on it: a running exploration is an unsettled prerequisite, so only the questions downstream of it wait for the sub-agent to report; ask the rest of the frontier now. The _decisions_ are the user's: put each to them and wait. 29 30 The session is done when the frontier is empty: every branch of the design tree visited, nothing left silently assumed. Do not act on it until the user confirms you have reached a shared understanding. Line pills: 6 (1), 8 (4), 10 (1), 12 (1), 24 (1), 26 (3), 28 (4), 30 (2) Each line takes the colour of the facts drawn from it; when several facts share a line the strongest wins (hurts > new > one model > both). Pill = number of facts on the line. Grey lines are candidates to shorten only after an eval confirms behaviour does not change. Green · new Round format, numbering, "yes" wording Grey · known General interviewing advice Cost 29 skills: $14, 5,360 calls. Sonnet 5.5 extracts and judges; Haiku 5.5 and Sonnet 5.5 answer. Precision "60% known" for one skill (25 atoms) means 40–80%. For the repo (661 atoms): ±4. grilling: 16 atoms · at most 25 atoms per skill · 95% intervals · mattpocock/skills study Interface text: ▦ OVERVIEW 10 / 14 ### Baseline quiz accuracy across three skill repositories With no skill loaded, Sonnet outscored Haiku on all three repos under lenient grading. Within obra, both got under 25% right on diagnosing-superpowers and over 90% on systematic-debugging. Sonnet graded both models. [10:29](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=629s) [With no skill loaded and lenient grading, Sonnet and Haiku respectively score 71% and 65% for obra, 64% and 54% for anthropics, and 53% and 45% for mattpocock. A paired dot plot shows how the two models’ scores vary across 15 obra skills.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e57666f6f34061d86fe6178b10a9728d77dc89d6b37547971926f09955859ed.webp) 07 · THREE PUBLIC SKILL REPOS Models already know half to two-thirds, and it varies by skill Share of atoms the model answers right with no skill loaded, lenient grading Sonnet 5.5 Haiku 5.5 obra 71% 65% anthropics 64% 54% mattpocock 53% 45% Per skill · obra/superpowers How much of each skill the model already knows obra/superpowers · share of the skill's facts answered correctly without the skill · 477 facts, 15 skills, same facts for both models Haiku 5.5 (mean 65%) Sonnet 5.5 (mean 71%) diagnosing-superpowers · 25 executing-plans · 60 writing-plans · 25 subagent-driven-development · 60 brainstorming · 24 dispatching-parallel-agents · 25 finishing-a-development-branch · 25 using-git-worktrees · 25 requesting-code-review · 31 verification-before-completion · 45 using-superpowers · 32 writing-skills · 25 receiving-code-review · 25 systematic-debugging · 25 test-driven-development · 25 X-axis ticks: 0%, 25%, 50%, 75%, 100% Number after the name = facts probed (25 per triage skill, ±0.2 at 95%). Lenient scoring. Judge: Sonnet 5.5, so Sonnet grades its own family and its rate is likely inflated. Knowing a fact ≠ following it: this measures knowledge, not behaviour. 60 skills, 1,537 atoms · judge Sonnet 5.5 (Sonnet self-judged) · lab study, Nov 2026 Interface text: OVERVIEW 12 / 14 ### Novelty is one signal: it shows where to experiment Haiku knows it should wait for answers. Would it actually wait without the reminder? Rerun the same tasks with that line removed. Cut it if the score holds. If the score drops, keep the line and record why. [11:51](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=711s) [Haiku answers that it should wait for the next round, but the quiz leaves open whether it actually waits in a session. Three boxes show how to test a known instruction: remove it, rerun the same tasks, and cut it if the score holds. If the score falls, keep it and record why.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ae2774ea9f059724aa4dc372b180ea63bf9113e026931f8d0224b8d39528082.webp) 08 · ONE OPTIMISATION VECTOR Novelty is one signal: it shows where to experiment What the quiz sees Q: "What must you do before the next round?" Haiku, no skill: "Wait for the answers." ✓ known What it can't see In a real session without the skill, does the agent actually stop and wait, or ask everything and move on? Known lines cut candidates → Eval without them same tasks → Same score: cut Worse: keep, mark why Interface text: OVERVIEW 13 / 14 ### Dokimi: test your SKILL.md like code Dokimi maps what each model already knows to suggest where a skill could be shorter. Its A/B tests compare quality, token use and time with and without the skill. Routing tests check whether the agent picks it at all. [13:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=780s) [Below the Dokimi repository link, two rows of five cards describe novelty mapping, A/B evaluations, context impact, routing, cost estimates, shortening and rewriting skills, deciding whether to merge or split them, runtime security, and a panel of judges.](https://assets.ai.engineer/sites/aiecode2025/sha256/96927364980fa22080b66bee09db180dab702c6af11cfdf7e4890a92b5c20487.webp) — TRY IT Dokimi: test your SKILL.md like code github.com/intel/dokimi Does a skill, MCP doc source or agent profile improve the agent? A scorecard. Novelty map What each model already knows A/B evals With vs without: quality, tokens, time Context impact What the run held: start-up load, growth, tool results Routing Will the agent ever pick your skill? Cost estimate The bill before any model call Shorten Cut text, keep the knowledge; rules that clash Evolve GEPA rewrites the skill, a sealed eval set gates it Merge or split One skill or several? Decided by routing Runtime security Sandboxed runs, trap tasks, trace diff Judge panel Several judges, checked evidence Full mattpocock/skills report: github.com/mattpocock/skills/issues/1384 Interface text: OVERVIEW 14 / 14 ## Full transcript [0:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=0s) Hello, my name is Nikolai Petro and we [0:02](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=2s) Hello, my name is Nikolai Petro and we [0:02](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=2s) Hello, my name is Nikolai Petro and we will be talking about skills and agents [0:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=4s) will be talking about skills and agents [0:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=4s) will be talking about skills and agents knowledge of those skills. [0:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=8s) knowledge of those skills. [0:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=8s) knowledge of those skills. We'll start it with the simple prompts [0:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=10s) We'll start it with the simple prompts [0:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=10s) We'll start it with the simple prompts where we've been [0:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=12s) where we've been [0:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=12s) where we've been playing with the new stuff, new work [0:14](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=14s) playing with the new stuff, new work [0:14](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=14s) playing with the new stuff, new work being done by the agents, repeating [0:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=16s) being done by the agents, repeating [0:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=16s) being done by the agents, repeating those prompts over and over again. And [0:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=18s) those prompts over and over again. And [0:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=18s) those prompts over and over again. And then we start packing those into the [0:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=20s) then we start packing those into the [0:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=20s) then we start packing those into the skills, [0:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=21s) skills, [0:22](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=22s) skills, creating repeatable, reproducible, [0:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=24s) creating repeatable, reproducible, [0:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=24s) creating repeatable, reproducible, portable type of their functionality [0:28](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=28s) portable type of their functionality [0:28](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=28s) portable type of their functionality or capabilities that we can teach agents [0:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=32s) or capabilities that we can teach agents [0:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=32s) or capabilities that we can teach agents to work with. And some of those skills [0:34](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=34s) to work with. And some of those skills [0:34](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=34s) to work with. And some of those skills are very important. hundreds of the [0:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=35s) are very important. hundreds of the [0:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=35s) are very important. hundreds of the stars on GitHub. [0:40](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=40s) There are different types of their [0:41](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=41s) There are different types of their [0:41](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=41s) There are different types of their skills. Some teach models something new [0:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=45s) skills. Some teach models something new [0:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=45s) skills. Some teach models something new bringing new knowledge or capabilities [0:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=47s) bringing new knowledge or capabilities [0:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=47s) bringing new knowledge or capabilities to the model. But there are also types [0:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=50s) to the model. But there are also types [0:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=50s) to the model. But there are also types of their just steering or process skills [0:53](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=53s) of their just steering or process skills [0:53](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=53s) of their just steering or process skills where we [0:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=55s) where we [0:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=55s) where we do not bring any additional information [0:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=57s) do not bring any additional information [0:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=57s) do not bring any additional information but just getting repeatable [1:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=60s) but just getting repeatable [1:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=60s) but just getting repeatable flow of the actions that we want agents [1:02](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=62s) flow of the actions that we want agents [1:02](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=62s) flow of the actions that we want agents to perform. both are useful. In this [1:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=66s) to perform. both are useful. In this [1:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=66s) to perform. both are useful. In this session, we will be talking about new [1:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=68s) session, we will be talking about new [1:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=68s) session, we will be talking about new knowledge type of the skills [1:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=70s) knowledge type of the skills [1:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=70s) knowledge type of the skills and [1:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=72s) and [1:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=72s) and we'll see what we can [1:15](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=75s) we'll see what we can [1:15](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=75s) we'll see what we can get from the models understanding of [1:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=77s) get from the models understanding of [1:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=77s) get from the models understanding of those skills. [1:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=80s) those skills. [1:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=80s) those skills. So, we get our first skills packed, [1:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=84s) So, we get our first skills packed, [1:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=84s) So, we get our first skills packed, start using them. Um, basically [1:28](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=88s) start using them. Um, basically [1:28](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=88s) start using them. Um, basically starting with the the feel of does it [1:31](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=91s) starting with the the feel of does it [1:31](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=91s) starting with the the feel of does it work or not. But there is also the more [1:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=93s) work or not. But there is also the more [1:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=93s) work or not. But there is also the more technical way and more deterministic way [1:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=95s) technical way and more deterministic way [1:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=95s) technical way and more deterministic way of the checking if skills are working or [1:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=97s) of the checking if skills are working or [1:38](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=98s) of the checking if skills are working or not and what's the impacts of those [1:40](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=100s) not and what's the impacts of those [1:40](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=100s) not and what's the impacts of those skills are evolves. There are different [1:43](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=103s) skills are evolves. There are different [1:43](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=103s) skills are evolves. There are different formats where evolves can be defined but [1:46](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=106s) formats where evolves can be defined but [1:46](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=106s) formats where evolves can be defined but at the sense this is task definition [1:48](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=108s) at the sense this is task definition [1:48](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=108s) at the sense this is task definition that you want your agent to be able to [1:51](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=111s) that you want your agent to be able to [1:51](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=111s) that you want your agent to be able to solve. [1:53](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=113s) solve. [1:53](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=113s) solve. You put it in the place ideally more [1:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=115s) You put it in the place ideally more [1:56](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=116s) You put it in the place ideally more deterministic so we can uh validate it [1:58](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=118s) deterministic so we can uh validate it [1:58](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=118s) deterministic so we can uh validate it and we can check two arms. One they're [2:02](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=122s) and we can check two arms. One they're [2:02](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=122s) and we can check two arms. One they're no skill loaded and there second one [2:05](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=125s) no skill loaded and there second one [2:05](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=125s) no skill loaded and there second one where we're loading the skill and [2:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=126s) where we're loading the skill and [2:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=126s) where we're loading the skill and providing this additional information to [2:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=128s) providing this additional information to [2:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=128s) providing this additional information to the context and we compare how the runs. [2:11](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=131s) the context and we compare how the runs. [2:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=132s) the context and we compare how the runs. There are multiple things. One is how [2:14](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=134s) There are multiple things. One is how [2:14](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=134s) There are multiple things. One is how many [2:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=136s) many [2:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=136s) many evolves been solved. In this case with [2:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=138s) evolves been solved. In this case with [2:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=138s) evolves been solved. In this case with no skill we're solving three out of [2:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=140s) no skill we're solving three out of [2:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=140s) no skill we're solving three out of five. With skill we're solving all five. [2:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=143s) five. With skill we're solving all five. [2:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=143s) five. With skill we're solving all five. meaning that we get two more tasks. [2:26](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=146s) meaning that we get two more tasks. [2:26](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=146s) meaning that we get two more tasks. There also would be implications on the [2:28](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=148s) There also would be implications on the [2:28](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=148s) There also would be implications on the tokens usage or time that it took those [2:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=152s) tokens usage or time that it took those [2:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=152s) tokens usage or time that it took those tasks to be completed and other factors. [2:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=155s) tasks to be completed and other factors. [2:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=155s) tasks to be completed and other factors. But at the end, we are comparing those [2:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=157s) But at the end, we are comparing those [2:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=157s) But at the end, we are comparing those and seeing what makes sense, what not. [2:39](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=159s) and seeing what makes sense, what not. [2:40](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=160s) and seeing what makes sense, what not. And this is one of the main tools for [2:42](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=162s) And this is one of the main tools for [2:42](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=162s) And this is one of the main tools for evaluating the skills and uh saying that [2:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=165s) evaluating the skills and uh saying that [2:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=165s) evaluating the skills and uh saying that they not just nice ideas but actually [2:48](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=168s) they not just nice ideas but actually [2:48](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=168s) they not just nice ideas but actually working pieces of instructions and [2:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=170s) working pieces of instructions and [2:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=170s) working pieces of instructions and context for the agents. [2:53](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=173s) context for the agents. [2:53](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=173s) context for the agents. But what next? We get to the stage where [2:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=177s) But what next? We get to the stage where [2:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=177s) But what next? We get to the stage where we have their working skill. It solves [2:59](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=179s) we have their working skill. It solves [2:59](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=179s) we have their working skill. It solves all some evolves. But we know that there [3:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=184s) all some evolves. But we know that there [3:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=184s) all some evolves. But we know that there should be a better way. We have models [3:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=186s) should be a better way. We have models [3:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=186s) should be a better way. We have models evolving. We have u some [3:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=190s) evolving. We have u some [3:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=190s) evolving. We have u some might be cases where we want [3:14](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=194s) might be cases where we want [3:14](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=194s) might be cases where we want to get skill more efficient meaning [3:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=197s) to get skill more efficient meaning [3:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=197s) to get skill more efficient meaning consuming less tokens. So what's there [3:19](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=199s) consuming less tokens. So what's there [3:19](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=199s) consuming less tokens. So what's there the options there? One is just change [3:22](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=202s) the options there? One is just change [3:22](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=202s) the options there? One is just change line by line and try brute force all the [3:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=207s) line by line and try brute force all the [3:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=207s) line by line and try brute force all the different combinations. Another would be [3:30](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=210s) different combinations. Another would be [3:30](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=210s) different combinations. Another would be to think through what makes sense. So [3:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=212s) to think through what makes sense. So [3:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=212s) to think through what makes sense. So maybe use something as the user feedback [3:36](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=216s) maybe use something as the user feedback [3:36](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=216s) maybe use something as the user feedback on the skills to see where the skills [3:39](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=219s) on the skills to see where the skills [3:39](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=219s) on the skills to see where the skills perform well. Uh there it's not. And [3:42](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=222s) perform well. Uh there it's not. And [3:42](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=222s) perform well. Uh there it's not. And with new models just running scale and [3:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=225s) with new models just running scale and [3:46](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=226s) with new models just running scale and seeing how it works. What's the options [3:48](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=228s) seeing how it works. What's the options [3:48](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=228s) seeing how it works. What's the options there? And how large those modification [3:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=232s) there? And how large those modification [3:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=232s) there? And how large those modification should be just [3:54](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=234s) should be just [3:54](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=234s) should be just lines, sentences, ideas. [4:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=240s) So what we found to be useful is instead [4:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=244s) So what we found to be useful is instead [4:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=244s) So what we found to be useful is instead of relying on their luck we can put more [4:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=248s) of relying on their luck we can put more [4:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=248s) of relying on their luck we can put more deterministic process here using idea of [4:11](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=251s) deterministic process here using idea of [4:11](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=251s) deterministic process here using idea of their items from their fact score. U [4:15](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=255s) their items from their fact score. U [4:15](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=255s) their items from their fact score. U we can split their skill and [4:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=258s) we can split their skill and [4:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=258s) we can split their skill and instructions into those items and [4:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=263s) instructions into those items and [4:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=263s) instructions into those items and for each of those had the pair of [4:26](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=266s) for each of those had the pair of [4:26](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=266s) for each of those had the pair of question and answer that we expect. [4:29](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=269s) question and answer that we expect. [4:29](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=269s) question and answer that we expect. This could be either general ideas like [4:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=272s) This could be either general ideas like [4:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=272s) This could be either general ideas like proposition in this case or contracts. [4:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=275s) proposition in this case or contracts. [4:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=275s) proposition in this case or contracts. There we put more definitive expectation [4:39](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=279s) There we put more definitive expectation [4:39](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=279s) There we put more definitive expectation on their action [4:44](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=284s) and in general we will be getting the u [4:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=290s) some predefined amount of the items per [4:53](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=293s) some predefined amount of the items per [4:53](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=293s) some predefined amount of the items per skill and we will have let's say [4:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=297s) skill and we will have let's say [4:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=297s) skill and we will have let's say 25 or 30 items per skill. [5:01](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=301s) 25 or 30 items per skill. [5:01](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=301s) 25 or 30 items per skill. Then for each of the items we will again [5:05](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=305s) Then for each of the items we will again [5:05](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=305s) Then for each of the items we will again run two arms, one without skill and one [5:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=308s) run two arms, one without skill and one [5:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=308s) run two arms, one without skill and one with skill [5:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=310s) with skill [5:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=310s) with skill and we will have judge or panel of [5:14](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=314s) and we will have judge or panel of [5:14](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=314s) and we will have judge or panel of judges that will be evaluating model [5:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=316s) judges that will be evaluating model [5:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=316s) judges that will be evaluating model answers. So we asking those questions to [5:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=318s) answers. So we asking those questions to [5:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=318s) answers. So we asking those questions to the model that we're interested in. [5:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=321s) the model that we're interested in. [5:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=321s) the model that we're interested in. We evaluating those questions with the [5:25](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=325s) We evaluating those questions with the [5:25](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=325s) We evaluating those questions with the judge ideally different family of the [5:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=327s) judge ideally different family of the [5:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=327s) judge ideally different family of the models or panel of judges. We also [5:31](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=331s) models or panel of judges. We also [5:31](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=331s) models or panel of judges. We also ensures that the question do not leak [5:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=333s) ensures that the question do not leak [5:34](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=334s) ensures that the question do not leak answers. [5:36](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=336s) answers. [5:36](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=336s) answers. We depending on the case can compare [5:39](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=339s) We depending on the case can compare [5:39](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=339s) We depending on the case can compare with no tools or additional um [5:42](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=342s) with no tools or additional um [5:42](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=342s) with no tools or additional um information to check their internal [5:44](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=344s) information to check their internal [5:44](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=344s) information to check their internal model understanding or provide it with [5:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=347s) model understanding or provide it with [5:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=347s) model understanding or provide it with there some tools like web access. So uh [5:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=350s) there some tools like web access. So uh [5:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=350s) there some tools like web access. So uh it would be more practical [5:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=352s) it would be more practical [5:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=352s) it would be more practical and we [5:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=355s) and we [5:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=355s) and we can get the simple for outcomes matrix [6:02](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=362s) basically where we are checking that the [6:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=364s) basically where we are checking that the [6:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=364s) basically where we are checking that the model [6:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=366s) model [6:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=366s) model um knows or do not know something and [6:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=370s) um knows or do not know something and [6:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=370s) um knows or do not know something and how this maps to their skill impact on [6:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=372s) how this maps to their skill impact on [6:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=372s) how this maps to their skill impact on this. So the simplest option is where [6:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=376s) this. So the simplest option is where [6:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=376s) this. So the simplest option is where model was able to answer this particular [6:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=381s) model was able to answer this particular [6:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=381s) model was able to answer this particular questions even without skill or with [6:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=383s) questions even without skill or with [6:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=383s) questions even without skill or with skill. This means that skill have no [6:25](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=385s) skill. This means that skill have no [6:25](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=385s) skill. This means that skill have no impact on this and this was already part [6:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=387s) impact on this and this was already part [6:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=387s) impact on this and this was already part of the model's knowledge. The second one [6:30](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=390s) of the model's knowledge. The second one [6:30](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=390s) of the model's knowledge. The second one is where without skill actual model was [6:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=393s) is where without skill actual model was [6:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=393s) is where without skill actual model was able to answer this correctly [6:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=395s) able to answer this correctly [6:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=395s) able to answer this correctly and by adding skill we saw the tradition [6:39](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=399s) and by adding skill we saw the tradition [6:40](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=400s) and by adding skill we saw the tradition in performance and model capability to [6:42](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=402s) in performance and model capability to [6:42](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=402s) in performance and model capability to talk or explain particular aspects. [6:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=407s) talk or explain particular aspects. [6:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=407s) talk or explain particular aspects. The ideal case is where without skill [6:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=410s) The ideal case is where without skill [6:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=410s) The ideal case is where without skill model wasn't able to answer something [6:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=412s) model wasn't able to answer something [6:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=412s) model wasn't able to answer something and with skill we start seeing that it [6:54](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=414s) and with skill we start seeing that it [6:54](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=414s) and with skill we start seeing that it can answer those new type of the [6:56](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=416s) can answer those new type of the [6:56](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=416s) can answer those new type of the questions and scenarios and this is what [6:58](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=418s) questions and scenarios and this is what [6:58](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=418s) questions and scenarios and this is what we were aiming for teach model something [7:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=420s) we were aiming for teach model something [7:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=420s) we were aiming for teach model something new [7:02](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=422s) new [7:02](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=422s) new but there also can be the fourth option [7:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=424s) but there also can be the fourth option [7:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=424s) but there also can be the fourth option where skills were written not ideally or [7:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=430s) where skills were written not ideally or [7:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=430s) where skills were written not ideally or in some maybe not clean way [7:14](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=434s) in some maybe not clean way [7:14](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=434s) in some maybe not clean way model wasn't able to answer it either [7:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=436s) model wasn't able to answer it either [7:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=436s) model wasn't able to answer it either without skill or with skill. And while [7:19](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=439s) without skill or with skill. And while [7:19](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=439s) without skill or with skill. And while this topic might be relevant and [7:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=441s) this topic might be relevant and [7:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=441s) this topic might be relevant and important for the model, it still sort [7:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=443s) important for the model, it still sort [7:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=443s) important for the model, it still sort of misses some of those concepts. [7:28](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=448s) The simplest thing to fix are [7:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=453s) The simplest thing to fix are [7:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=453s) The simplest thing to fix are instructions or items that actually hurt [7:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=455s) instructions or items that actually hurt [7:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=455s) instructions or items that actually hurt performance. And [7:38](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=458s) performance. And [7:38](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=458s) performance. And for popular skills, those are criteria. [7:41](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=461s) for popular skills, those are criteria. [7:41](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=461s) for popular skills, those are criteria. So in the [7:44](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=464s) So in the [7:44](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=464s) So in the uh box below you see that it's just one [7:48](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=468s) uh box below you see that it's just one [7:48](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=468s) uh box below you see that it's just one case out of the mostly 400 items in [7:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=472s) case out of the mostly 400 items in [7:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=472s) case out of the mostly 400 items in entropic skills and others don't have [7:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=475s) entropic skills and others don't have [7:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=475s) entropic skills and others don't have those. But for their smaller skills that [7:59](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=479s) those. But for their smaller skills that [7:59](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=479s) those. But for their smaller skills that are less popular you probably haven't [8:01](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=481s) are less popular you probably haven't [8:01](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=481s) are less popular you probably haven't those. This is what we're seeing with [8:03](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=483s) those. This is what we're seeing with [8:03](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=483s) those. This is what we're seeing with some of our experiments. [8:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=486s) some of our experiments. [8:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=486s) some of our experiments. And [8:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=488s) And [8:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=488s) And this is just straightforward the [8:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=490s) this is just straightforward the [8:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=490s) this is just straightforward the simplest thing to fix to ensure that [8:13](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=493s) simplest thing to fix to ensure that [8:13](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=493s) simplest thing to fix to ensure that your skill do not reduce model [8:15](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=495s) your skill do not reduce model [8:15](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=495s) your skill do not reduce model capabilities and performance and bring [8:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=497s) capabilities and performance and bring [8:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=497s) capabilities and performance and bring only relevant [8:19](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=499s) only relevant [8:19](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=499s) only relevant context there. [8:22](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=502s) context there. [8:22](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=502s) context there. And we also can actually color this in [8:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=504s) And we also can actually color this in [8:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=504s) And we also can actually color this in this scenario. This is done by lines but [8:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=507s) this scenario. This is done by lines but [8:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=507s) this scenario. This is done by lines but actually going with its you might have [8:29](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=509s) actually going with its you might have [8:29](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=509s) actually going with its you might have their uh different coloring for a single [8:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=512s) their uh different coloring for a single [8:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=512s) their uh different coloring for a single line or a single sentence. [8:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=515s) line or a single sentence. [8:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=515s) line or a single sentence. And this just gives your example there [8:38](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=518s) And this just gives your example there [8:38](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=518s) And this just gives your example there uh visualization where this would be [8:41](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=521s) uh visualization where this would be [8:41](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=521s) uh visualization where this would be helpful details and instructions in [8:44](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=524s) helpful details and instructions in [8:44](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=524s) helpful details and instructions in terms what new stuff you bring to the [8:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=527s) terms what new stuff you bring to the [8:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=527s) terms what new stuff you bring to the model and what is [8:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=532s) something that model already knows. [8:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=535s) something that model already knows. [8:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=535s) something that model already knows. Costwise [8:56](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=536s) Costwise [8:56](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=536s) Costwise we are running all those questions and [8:58](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=538s) we are running all those questions and [8:58](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=538s) we are running all those questions and asking model [9:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=540s) asking model [9:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=540s) asking model but still relatively this is uh cheap [9:03](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=543s) but still relatively this is uh cheap [9:03](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=543s) but still relatively this is uh cheap for 29 skills 14 bucks of their set and [9:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=546s) for 29 skills 14 bucks of their set and [9:06](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=546s) for 29 skills 14 bucks of their set and haiku calls with their sonet used as a [9:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=550s) haiku calls with their sonet used as a [9:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=550s) haiku calls with their sonet used as a judge [9:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=552s) judge [9:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=552s) judge and [9:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=556s) there are couple ways how we also can [9:19](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=559s) there are couple ways how we also can [9:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=560s) there are couple ways how we also can check their strictness of their some [9:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=563s) check their strictness of their some [9:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=563s) check their strictness of their some knowledge. Um we have uh lenient and [9:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=567s) knowledge. Um we have uh lenient and [9:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=567s) knowledge. Um we have uh lenient and strict ways of the checking this where [9:29](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=569s) strict ways of the checking this where [9:30](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=570s) strict ways of the checking this where first checks the general idea while the [9:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=572s) first checks the general idea while the [9:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=572s) first checks the general idea while the second one is checking for exact details [9:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=577s) second one is checking for exact details [9:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=577s) second one is checking for exact details and you can think about the difference [9:39](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=579s) and you can think about the difference [9:39](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=579s) and you can think about the difference between general knowledge and [9:40](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=580s) between general knowledge and [9:40](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=580s) between general knowledge and understanding of the topic to the [9:43](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=583s) understanding of the topic to the [9:43](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=583s) understanding of the topic to the understanding of the exact keys [9:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=585s) understanding of the exact keys [9:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=585s) understanding of the exact keys parameters and exact values or steps. [9:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=590s) parameters and exact values or steps. [9:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=590s) parameters and exact values or steps. And what we see across the models is the [9:53](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=593s) And what we see across the models is the [9:53](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=593s) And what we see across the models is the difference is large especially with the [9:58](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=598s) difference is large especially with the [9:58](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=598s) difference is large especially with the um larger frontier models where we see [10:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=600s) um larger frontier models where we see [10:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=600s) um larger frontier models where we see all dropping from 70% understanding of [10:03](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=603s) all dropping from 70% understanding of [10:03](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=603s) all dropping from 70% understanding of their skill in case of the general [10:05](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=605s) their skill in case of the general [10:05](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=605s) their skill in case of the general understanding to just 22% of the [10:09](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=609s) understanding to just 22% of the [10:09](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=609s) understanding to just 22% of the specific details [10:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=612s) specific details [10:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=612s) specific details and this is much our expectations. We [10:15](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=615s) and this is much our expectations. We [10:15](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=615s) and this is much our expectations. We see that new models are pretty well [10:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=617s) see that new models are pretty well [10:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=617s) see that new models are pretty well generalized to understand concepts but [10:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=620s) generalized to understand concepts but [10:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=620s) generalized to understand concepts but it's still helpful to provide their with [10:22](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=622s) it's still helpful to provide their with [10:22](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=622s) it's still helpful to provide their with exact details and data that will be [10:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=624s) exact details and data that will be [10:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=624s) exact details and data that will be useful for their solving those problems. [10:30](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=630s) Here is outlook of their um comparing [10:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=635s) Here is outlook of their um comparing [10:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=635s) Here is outlook of their um comparing those three popular repos with the [10:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=637s) those three popular repos with the [10:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=637s) those three popular repos with the skills across different models. So one [10:41](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=641s) skills across different models. So one [10:41](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=641s) skills across different models. So one of the interesting observations here is [10:43](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=643s) of the interesting observations here is [10:43](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=643s) of the interesting observations here is that smaller model have in general lower [10:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=647s) that smaller model have in general lower [10:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=647s) that smaller model have in general lower understanding of the [10:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=650s) understanding of the [10:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=650s) understanding of the skill [10:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=652s) skill [10:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=652s) skill with their larger sonet having better [10:54](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=654s) with their larger sonet having better [10:54](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=654s) with their larger sonet having better understanding which is [10:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=657s) understanding which is [10:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=657s) understanding which is why we have bigger models. But also we [11:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=660s) why we have bigger models. But also we [11:00](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=660s) why we have bigger models. But also we see that overall their models already [11:03](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=663s) see that overall their models already [11:03](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=663s) see that overall their models already understand a lot of their [11:08](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=668s) details and context of those skills. And [11:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=670s) details and context of those skills. And [11:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=670s) details and context of those skills. And you would expect that with the newer [11:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=672s) you would expect that with the newer [11:12](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=672s) you would expect that with the newer models and more advances in their uh [11:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=676s) models and more advances in their uh [11:16](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=676s) models and more advances in their uh better training capabilities we will [11:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=678s) better training capabilities we will [11:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=678s) better training capabilities we will have more and more [11:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=681s) have more and more [11:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=681s) have more and more uh in terms of the models understanding. [11:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=684s) uh in terms of the models understanding. [11:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=684s) uh in terms of the models understanding. But also we see that this pretty [11:26](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=686s) But also we see that this pretty [11:26](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=686s) But also we see that this pretty significantly differs skill to skill. [11:29](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=689s) significantly differs skill to skill. [11:29](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=689s) significantly differs skill to skill. And it's not bad. Some skills are [11:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=692s) And it's not bad. Some skills are [11:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=692s) And it's not bad. Some skills are intended to be directional, [11:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=695s) intended to be directional, [11:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=695s) intended to be directional, but it's also useful piece of [11:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=697s) but it's also useful piece of [11:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=697s) but it's also useful piece of information to see how those [11:40](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=700s) information to see how those [11:40](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=700s) information to see how those skills bring new knowledge uh to the [11:43](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=703s) skills bring new knowledge uh to the [11:43](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=703s) skills bring new knowledge uh to the different model and different families. [11:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=705s) different model and different families. [11:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=705s) different model and different families. And uh this should be run ideally in the [11:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=707s) And uh this should be run ideally in the [11:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=707s) And uh this should be run ideally in the broader set of the models. [11:55](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=715s) But just having their [11:59](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=719s) But just having their [11:59](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=719s) But just having their This novelty piece of information is [12:03](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=723s) This novelty piece of information is [12:03](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=723s) This novelty piece of information is just one of the signals. It's basically [12:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=724s) just one of the signals. It's basically [12:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=724s) just one of the signals. It's basically initial vector for optimization. At the [12:07](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=727s) initial vector for optimization. At the [12:07](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=727s) initial vector for optimization. At the end, we still have to run evolves and [12:09](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=729s) end, we still have to run evolves and [12:09](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=729s) end, we still have to run evolves and validate that all the changes that we [12:13](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=733s) validate that all the changes that we [12:13](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=733s) validate that all the changes that we try make with their skill actually [12:15](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=735s) try make with their skill actually [12:15](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=735s) try make with their skill actually works. [12:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=737s) works. [12:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=737s) works. So the one of their simpler scenarios [12:19](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=739s) So the one of their simpler scenarios [12:20](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=740s) So the one of their simpler scenarios would be we can look on their skill and [12:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=743s) would be we can look on their skill and [12:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=743s) would be we can look on their skill and try to remove some or all of the already [12:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=747s) try to remove some or all of the already [12:27](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=747s) try to remove some or all of the already known pieces of information from the [12:29](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=749s) known pieces of information from the [12:29](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=749s) known pieces of information from the skill and run aation. [12:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=753s) skill and run aation. [12:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=753s) skill and run aation. It might come up with their same [12:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=755s) It might come up with their same [12:35](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=755s) It might come up with their same scoring. Basically we will be getting [12:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=757s) scoring. Basically we will be getting [12:37](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=757s) scoring. Basically we will be getting their same results model still capable [12:38](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=758s) their same results model still capable [12:38](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=758s) their same results model still capable of solving those problems. Then great, [12:41](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=761s) of solving those problems. Then great, [12:41](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=761s) of solving those problems. Then great, we can reduce skill size, safe tokens, [12:44](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=764s) we can reduce skill size, safe tokens, [12:44](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=764s) we can reduce skill size, safe tokens, safe context. [12:46](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=766s) safe context. [12:46](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=766s) safe context. And instead of their brute forcing and [12:49](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=769s) And instead of their brute forcing and [12:49](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=769s) And instead of their brute forcing and guessing what we can or should remove or [12:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=772s) guessing what we can or should remove or [12:52](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=772s) guessing what we can or should remove or add or how to fix their uh [12:56](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=776s) add or how to fix their uh [12:56](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=776s) add or how to fix their uh degradations, we can have a guidance. [13:01](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=781s) degradations, we can have a guidance. [13:01](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=781s) degradations, we can have a guidance. So all of those measurements experiments [13:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=784s) So all of those measurements experiments [13:04](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=784s) So all of those measurements experiments been done with the document which is a [13:07](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=787s) been done with the document which is a [13:07](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=787s) been done with the document which is a framework for the comparing [13:09](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=789s) framework for the comparing [13:10](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=790s) framework for the comparing everything related to the agents and [13:11](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=791s) everything related to the agents and [13:11](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=791s) everything related to the agents and context [13:13](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=793s) context [13:13](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=793s) context where it's is scale agents MD MCP tools [13:17](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=797s) where it's is scale agents MD MCP tools [13:18](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=798s) where it's is scale agents MD MCP tools whatever and novelty is one of their [13:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=801s) whatever and novelty is one of their [13:21](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=801s) whatever and novelty is one of their pieces of the functionality where you [13:23](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=803s) pieces of the functionality where you [13:24](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=804s) pieces of the functionality where you can get this initial vector for their [13:26](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=806s) can get this initial vector for their [13:26](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=806s) can get this initial vector for their skill optimization and evalation but [13:28](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=808s) skill optimization and evalation but [13:28](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=808s) skill optimization and evalation but there also uh evolves there This [13:31](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=811s) there also uh evolves there This [13:32](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=812s) there also uh evolves there This assessment of the context impact, [13:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=813s) assessment of the context impact, [13:33](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=813s) assessment of the context impact, routing, comparing their uh [13:36](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=816s) routing, comparing their uh [13:36](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=816s) routing, comparing their uh descriptions, naming a bunch of the [13:38](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=818s) descriptions, naming a bunch of the [13:38](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=818s) descriptions, naming a bunch of the things that related to their skills [13:43](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=823s) things that related to their skills [13:43](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=823s) things that related to their skills where it would be helpful to actually [13:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=825s) where it would be helpful to actually [13:45](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=825s) where it would be helpful to actually get there more deterministic revelations [13:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=827s) get there more deterministic revelations [13:47](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=827s) get there more deterministic revelations to make conscious decisions in terms of [13:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=830s) to make conscious decisions in terms of [13:50](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=830s) to make conscious decisions in terms of what should be done with the skills. [13:54](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=834s) what should be done with the skills. [13:54](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=834s) what should be done with the skills. So, thank you for your time and please [13:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=837s) So, thank you for your time and please [13:57](https://www.youtube.com/watch?v=yzx4IuEh_tU&t=837s) So, thank you for your time and please go ahead and write better skills.