Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel

A prompt you keep typing is a good candidate for a skill. Pack the instructions into a file and another agent can use them, too. Nikolay Petrov of Intel distinguishes two reasons to do this. A process skill gets an agent to perform familiar actions in a repeatable order. A knowledge skill supplies information the agent needs to solve a problem. For that second kind, it makes sense to ask how much of the file the model already knows.

Start with tasks whose results you can check. Give the agent the same tasks in two conditions, once without the skill and once with its instructions in context. In Petrov’s example, the agent solves three of five tasks without the file and all five with it. Loading the file buys two additional successes. Token use and time to completion are separate costs you can compare across the runs.

Now suppose you want to shorten the file or move it to a newer model. The five passes tell you the skill works as written. Which instructions earned their place? User feedback can point to a task that went badly. Trying combinations of line edits can narrow the search, at the price of running each combination. There’s also a choice about what to edit. A line, a sentence and an idea can be three different units.

Petrov borrows an idea from FActScore: split the text into claims small enough to check individually. In a skill, an item might express a proposition about how something works. It might also specify what action the agent should take, which Petrov calls a contract. For each item, write a question and the answer you expect. He gives 25 or 30 items as an example budget for one skill.

Who gets to see which information? The target model gets the same question in both runs. Only one run gets the skill. A judge, ideally from a different model family, or a panel of judges grades each response against the expected answer. Keep that answer out of the question itself. If the question gives away the detail, even the run without the skill can get credit for repeating it.

Tool access changes the meaning of a correct answer. With tools and other information withheld, the baseline probes what the model can supply on its own. Give both runs web access and the comparison answers a practical question about a working agent. The baseline can now succeed by looking something up. In either setting, each claim gets a pair of grades, while the input that varies is the whole skill. Finding out what happens when you delete a particular sentence takes another experiment.

A pass with the skill loaded could mean the instructions taught the model something. It could also mean the model could already answer the question. Read the baseline grade first, then the grade with the skill.

• Pass → pass. The model could already give an acceptable answer. The corresponding passage is a candidate for shortening.
• Pass → fail. Loading the skill spoiled a previously correct answer. Inspect the instructions associated with that regression.
• Fail → pass. With the skill, the model can answer a question it previously missed. Preserve the detail that enabled the answer.
• Fail → fail. The model still misses the concept with the instructions in hand. If the task needs it, check whether the wording makes the expected answer or action clear.

Petrov starts with regressions. Here the model has demonstrated that it can answer the question, and loading the skill makes it stumble. These cases were rare in the popular skills tested. In one collection, only one item out of roughly 400 showed a regression. Even that single item gives the editor a specific passage to investigate.

Where in the file did the skill supply something useful? Petrov’s example colors the source lines by outcome so an editor can locate passages that helped and passages the model already understood. The distinction can be finer than the display suggests. Two claims in one sentence can earn different colors. Deleting the whole sentence could take the one needed detail with it.

Each question costs model calls in both conditions, plus grading. Petrov reports spending about $14 across 29 skills in a run using Haiku and Sonnet, with Sonnet as judge.

How precise does an answer have to be to pass? Lenient grading accepts the general idea. Strict grading checks the details the task calls for, such as a particular key, parameter, value or sequence of steps. An answer can explain what a parameter does and still leave the agent without the value it needs to use.

In the larger-model comparison Petrov highlights, the model met the general-idea standard on 70% of the material. When the judge required the specific details, the score fell to 22%. That gap explains the risk in shortening a skill by eye. The introductory explanation may be easy for the model to reconstruct. A key or value that barely registers in a summary may be the part the agent needs from the file.

How well does a result for one model carry over to another? Across three popular skill repositories, the smaller model generally knew less of the material than the larger Sonnet model, though both already understood a substantial amount. A passage that looks redundant in a Sonnet run may still supply something the smaller model needs.

The recording’s description places the headline’s 70% figure on the first skill the team tested. Results varied considerably from skill to skill, and the purpose of each file matters. A process skill can do its job by getting the agent to choose a particular workflow even when every action is familiar. For skills meant to teach, Petrov would run the comparison across a broader set of models and model families before deciding which details to remove.

A model can explain an idea when asked directly and still benefit from a reminder while doing the work. Petrov proposes a deletion experiment. Remove some or all of the material the model could explain unaided, load the shorter skill, and rerun the original task evaluations. Compare which problems the agent solves with each version. If it still solves the same problems, the shorter file saves input tokens and leaves more room in context. The 70% knowledge result identifies candidates for that experiment. How much text can go depends on the task results.

The team ran the experiments in a framework for comparing how agents use context, including skills, AGENTS.md instructions and MCP tools. It also supports task evaluations, routing checks and comparisons of names and descriptions. Those checks address different decisions. A description can affect which skill the agent selects. Once the skill is loaded, the item questions locate details worth editing, and the task evaluation shows whether the agent can still finish the job.