AI Engineer World's Fair 2026
Don't Ship Skills Without Evals
Read the talk
Don't Ship Skills Without Evals
A skill is useful only if it changes an agent’s behavior for the better. Small, repeatable evaluations reveal missed triggers, regressions, wasted context and when a skill can retire.
From a talk by Philipp Schmid
Before you start: Familiarity with coding agents, prompts and basic Python is helpful; no prior evaluation framework is required.
Everyone uses skills. Who tests them?
Do you use coding agents? Do you use skills with them? Do you have evaluations for those skills? Philipp Schmid, introducing himself as working on the Gemini API and agents at Google DeepMind, opens with that sequence of audience questions. Hands go up for coding agents and skills; very few remain for evals. The gap matters because accepting a plausible SKILL.md file is much easier than establishing that it helps.
Schmid describes SkillsBench as having indexed more than 50,000 skills from GitHub, with almost none carrying evaluations and many written by AI. Those are his ecosystem observations. The immediate engineering problem is independent of the inventory size: agent nondeterminism makes a failed task hard to diagnose. Was the skill wrong, did the agent fail to discover it, or was the task beyond the model’s capabilities? One successful run cannot separate those possibilities.
The distinction becomes sharper when moving from agents developers use to agents developers build. In Antigravity, Cursor or Claude Code, an engineer asking for a Gemini API feature can notice that the relevant skill was missed, stop the task, re-prompt or invoke a slash command. A customer asking for a refund does not know that a refund skill exists. They ask for help with their problem, not for a particular piece of the agent’s implementation. Customer-facing agents must discover the right instructions from ordinary requests without relying on the user to repair routing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A skill has layers—and a lifetime
A skill is a folder containing SKILL.md and supporting assets. Its defining mechanism is progressive disclosure: the agent starts with enough information to discover the skill, then loads instructions and deeper references only as needed.
| Layer | Contents | When it enters context |
|---|---|---|
| Discovery | Name and description | Available for skill selection |
| Instructions | SKILL.md body | When the agent reads the skill |
| Supporting material | References and other assets | When the task needs their detail |
The description therefore has a different job from the body: it helps the model decide whether to retrieve the instructions at all.
Schmid distinguishes two reasons for supplying those instructions:
- Capability skills teach something the model cannot yet do consistently, such as tracing logs or creating a React application. As models improve, an evaluation may show that the skill is no longer necessary.
- Preference skills encode team workflows, style conventions or company-specific requirements. These are more durable because a foundation model is unlikely to acquire those private preferences through general training. Evals protect them against regressions when the agent changes.
The same evaluation machinery supports two different decisions: retiring temporary assistance and preserving behavior that remains specific to the organization.
There is evidence that skills can help. Schmid summarizes SkillsBench 1.1 as roughly a fifteen-percent average improvement across open and closed models in different harnesses, covering around a hundred coding and productivity tasks across languages. The published release makes the unit and scope precise: mean task resolution rose from 33.9% to 50.5%, a 16.6-percentage-point increase, across 87 tasks and 18 model–harness configurations. Its public leaderboard and contribution process make the benchmark useful for investigating where those gains occur.
Generating a skill and accepting it after a quick skim is a different proposition. Schmid emphasizes the stronger results from human-written skills and the possibility that generated skills reduce performance. In the release’s self-generation experiment, all three tested configurations fell below their no-skill baselines; that does not establish that all AI-assisted authoring is harmful. His practical authoring recommendation is to keep the SKILL.md body under 500 lines and move detail into references, rather than treating a long generated document as inherently more capable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make discovery precise and context selective
First choose how the skill should be invoked. A model-triggered skill depends on the model recognizing a match between the request and the description. A user-invoked skill starts with an explicit instruction, which is often a better fit for familiar developer workflows such as creating a pull request or staging documentation. Explicit invocation can avoid keeping routine workflow guidance in the agent’s standing context. For the customer-facing scenario in this talk, the evaluation problem centers on model-invoked skills: the user will not name the mechanism.
The first of Schmid’s eight writing tips is to treat the description as functional routing logic. Often only a couple of sentences in the system instructions, it must communicate why the skill is useful, when it applies and how it should be used. A weak description can fail in either direction: never activating when needed or activating for too many tasks. A React skill, for example, needs a trigger that identifies React work rather than vaguely advertising development expertise.
Inside the instructions, prefer directives over explanatory essays. The talk contrasts passive information about the Interactions API being recommended for multi-turn chat because it handles session state with a direct instruction to use that API when working on a chat application. The purpose is to change the agent’s choice, so the instruction should state the choice and its condition.
Keep each disclosure layer lean. Schmid uses a description of roughly 100–200 tokens to illustrate a cost paid on every model call where it is included. The body adds another context cost when read. Detailed references should therefore remain accessible without being loaded indiscriminately. For a deployment skill spanning AWS, Google Cloud and Azure, put each provider’s instructions in a separate reference file; the skill body should help the agent find the relevant provider, not force it to read all three deployment guides.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the agent useful freedom, then test the boundaries
A skill should also leave the right amount of discretion. If a workflow always consists of the same steps in the same order, put those steps in a script and let the agent invoke it. There is little benefit in spending model tokens reconstructing an invariant procedure. Where judgment is useful, describe the goal and constraints: how documentation can be deployed, updated or staged, for example. For a database configuration change, identify the configuration file and the requested modification instead of spelling out every read, port edit and subsequent action.
Boundaries matter just as much as positive triggers. A description that says to use a skill for web development can cause React-specific guidance to load during Angular work. Narrowing the trigger to React components or Tailwind CSS gives the model a more useful decision boundary. The corresponding evaluation needs examples where the skill should remain unloaded, not just examples where it should activate.
Start testing while writing the skill. Schmid recommends an initial set of 10–20 prompts: five happy-path requests, five requests that should not trigger the skill, and customer or production traces where available. Real requests expose wording and missing context that a skill author may unconsciously avoid when inventing tests.
Then remove behavioral no-ops. Schmid credits educator Matt and his skills repository for this technique: delete instructions that do not change what the agent does. Generic requests to be thorough, make an implementation easy to read or write clear, high-quality code may merely restate behavior already expected from the model. Useful skill content supplies a missing constraint, decision or piece of knowledge.
Finally, compare runs with the skill enabled and disabled. Models, environments and expectations change. If the model reaches the same performance without the skill, retirement can remove both token cost and maintenance work. This ablation test is the bridge from writing plausible instructions to measuring whether those instructions still earn their place.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A small harness for a newly released API
The practical example is a skill for the Gemini Interactions API, an interface for working with models and agents. Schmid describes a training-knowledge gap: the API arrived after the relevant training, leaving Gemini 3, Gemini 3.1 and Gemini 3.5 without the context needed to use it reliably. The skill’s job was to supply that missing API knowledge and steer generated code toward the latest models. This is the talk’s historical motivation, rather than a claim about what every current model knows.
The team assembled 117 test cases from real-user code-generation requests, synthetic cases and feedback about outdated model choices—for example, code still selecting Gemini 2.0 when Gemini 3.0 was available. Schmid reports almost 90% performance on generating valid Interactions API code with the latest Gemini models in this 117-case evaluation. The spoken account does not supply the baseline or a complete scoring definition, so the result is best read as the outcome of this particular skill-development exercise.
The harness needed two assets: a JSON file of cases and a basic Python script. Each case recorded the user prompt, the target language—TypeScript or Python—a should_trigger flag and expected checks. The runner invoked Gemini CLI, parsed its output and made the result available for validation. Keeping the trigger expectation separate from output checks lets the evaluator distinguish a discovery problem from incorrect code after discovery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use cheap checks for concrete requirements
Many of the case study’s requirements were mechanically detectable: the right SDK, the right model identifier, the right methods and the absence of obsolete patterns. Regex assertions made these checks cheap enough to repeat frequently, and coding agents could help author the patterns. When a new model arrived, the team could update the expected identifiers rather than introduce an LLM judge for a simple string-level requirement.
A compact Python validator can keep those requirements in the case data. Here, required and forbidden map readable check names to regex patterns; the harness supplies generated output and whether the trace showed a skill read.
python
import re
def evaluate_case(case: dict, output: str, skill_read: bool) -> dict:
checks = {
"trigger": skill_read == case["should_trigger"],
}
expected = case["expected_checks"]
for name, pattern in expected.get("required", {}).items():
checks[f"required:{name}"] = bool(re.search(pattern, output))
for name, pattern in expected.get("forbidden", {}).items():
checks[f"forbidden:{name}"] = not re.search(pattern, output)
output_checks = {
name: passed
for name, passed in checks.items()
if name != "trigger"
}
return {
"checks": checks,
"trigger_pass": checks["trigger"],
"output_pass": all(output_checks.values()),
}
This separates trigger diagnostics from output requirements. Regex can establish that a required pattern appears or an obsolete one does not; execution-level correctness needs a validator that actually checks that property.
For more complex skills, the relevant evidence may be the whole trace: which steps the agent took, what it inspected and whether the final result met a qualitative requirement. Use an LLM judge with an explicit rubric for those cases. Feed it the output or trace, request a pass/fail decision, then inspect failures and revise the skill. The distinction is about what must be evaluated, not a requirement to use a model judge for every test.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put evaluations beside the skill
Schmid describes Google DeepMind’s internal approach as keeping evaluations alongside every skill. The YAML shown in the presentation illustrates the structure, rather than representing the team’s actual storage format. Each evaluation has multiple prompt cases and a clean workspace that can include application files. Startup commands install or preload the libraries needed for the task. Script validators inspect traces for skill reads, commands and CLI execution; LLM judges assess expectations that need broader interpretation.
Every change to a skill file runs the evaluations. Schmid describes a merge rule under which a change must improve the evaluated cases or add new evaluations. That makes the test suite part of the skill’s maintenance contract: changing instructions requires evidence about the resulting behavior, rather than a judgment that the new wording sounds better.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure outcomes without prescribing a path
Discovery remains a major source of failure. Schmid reports that 50% of the failures his team observed involved incorrect skill triggering when user prompts lacked enough detail. He does not specify the sample behind that figure. Customers do not know the descriptions in the agent’s context, so short or underspecified requests must be part of the evaluation set. Clear directives, negative tests and a small initial collection of prompts can expose these problems before a large suite exists.
But triggering is a diagnostic, not the final objective. Do not require the model to load a skill on its first turn merely because that is the path the author expected. Loading it after five turns can be acceptable. Completing the task without loading it can also be acceptable. Keep the trigger result visible, as in the Python validator, while defining success in terms of the requested outcome. Otherwise the evaluation can punish a capable agent for taking a different successful route.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test cleanly, repeat, and keep the eval
Isolation prevents an apparently successful run from borrowing the answer. In an existing development environment, an agent may find previous chats or artifacts from earlier executions and recover the skill’s context without reading the skill. A clean workspace removes that accidental assistance. Repeat the experiment as well: Schmid recommends three to six trials per case because a first-run success may fail to recur.
Cover the harnesses people actually use. A skill evaluated only in Claude Code or Antigravity may behave differently in Cursor. The model is another variable: good results with Gemini do not establish good results for customers using Codex. Reliability belongs to the combination of task, skill, model, harness and environment, so the evaluation matrix should reflect the combinations that matter to the deployment.
When an improved model makes a skill unnecessary, retain its eval. The test still expresses a behavior the product needs. If performance later degrades, it can reveal the need to reintroduce the skill or adjust another tool or agent component. A capability that required explicit guidance six months ago may now work without it; the durable asset is the test that tells you whether it still works.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with the skill you use most
Schmid closes with an exercise small enough to do on returning from the holiday:
- Pick the most-used skill. If that is unclear, ask a coding agent to inspect your trajectories and identify frequent skill use.
- Write five test prompts for it.
- Store the cases in JSON or YAML and use a Python script to run the agent or harness and inspect the outcomes.
- Remove no-op instructions. Even when scores stay unchanged, unnecessary tokens still cost money.
- Run the same evaluations with the skill loaded and without it.
That last comparison establishes whether the skill is helping and whether it is ready to retire. The instruction to stop shipping untested skills does not require a large evaluation platform: it starts with a few representative requests and a repeatable way to check what happens.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Benchmark release covering paired evaluations with and without curated skills across 87 tasks, with results and links to evaluation artifacts.
Further reading
Philipp Schmid's practical guidance on descriptions, reference layers, negative cases, repeated evaluation and skill retirement.
Updates since the talk
Matt Pocock's guide to removing ineffective instructions and structuring agent-facing documents, including installation instructions for the skill formerly called writing-great-skills.
Current official documentation for calling Gemini models and agents, managing conversation state and finding supported SDKs.
Read the complete timestamped transcript
- 0:00
[upbeat music] Yeah. So hi, everyone. My name is Philipp. I'm based out of Germany.
- 0:16
I am part of the Google DeepMind team, mostly working on Gemini API and agents, and we are going to talk about why you should not ship skills without evals.
- 0:25
And maybe before we start, I need a little bit of your help. So if you could raise your hands if you use coding agents to write code. So yeah, hopefully every, every hand goes up, right?
- 0:35
And do you use skills with it? Okay. Do you have evals for those skills? Okay, yeah. That's, um, not a lot of hands. Everyone uses skills. No one has evals.
- 0:46
Hopefully, we can fix that today. And, like, very important is, like, why checks fail in productions. And, um, Skills Bench is a very popular and nice, like, eval or benchmark which, um, indexed, like, over, like, fifty thousand skills from, like, GitHub and, like, tried to look into them, and almost none of those skills had evals.
- 1:04
Most of them were AI written, um, not really tested, and it's very hard to know if your skill is good or bad because, like, agents are really non-deterministic. So you might not know if your, uh, task fails because your skill is bad or if your task fails because it's way too challenging for the model.
- 1:21
So, um, very important, um, before we go into it is I want to, like, really make sure that we know the difference between the agents we use and the agents we build.
- 1:31
Uh, most of us use agents for writing code, doing productivity work. That's the agents we use. It's like Antigravity, Cursor, Claude Code, and there you are the engineer, and you have context about skills, right?
- 1:43
If you write some prompt to, I don't know, like, "Help me build a new Gemini API feature," and if your agent does not invoke the skill on the first time, you will notice it very quickly.
- 1:54
You stop your, your task and re-prompt it or, like, use slash commands for, for triggering those skills. When you build an agent inside your application for consumer or cus-customers, they have no idea about what a skill is.
- 2:07
They don't start their prompt with, "Use customer support skill to, like, help me refund," or, "Use refund skill to help me solve my problem." So there's a big difference between the agents we use and how we use skills and the agents we build and how our customers might want to use skills in, like, the context there.
- 2:27
And what is a skill? I mean, every one of us knows hopefully by now what a skill is. It's, like, basically really a, a folder with a skills.md file in it and then some additional assets, um, to make that skill really work.
- 2:38
And the big difference with skills is that they work on progressive disclosure. So most of the skills start very small. So you have the title and the description. The sc-description is normally part of, um, the model's context, so the model knows when to use the skill.
- 2:54
Second layer is we have a skills body with more instructions, more details, and hopefully more references to external files. And then you can really go deep in those reference files where there's all of the context the model needs to discover to, to solve the task.
- 3:08
And I like to differentiate between two kinds of skills. So there are capability skills and preference skills. Capability skills teach models something they cannot do consistently at the moment.
- 3:19
Maybe it's like, I don't know, like tracing some logs, creating a new React app. And those capability skills are temporary. So the better our model gets, the more likely it is that we can remove those skills.
- 3:32
And evals will tell us when we can retire a skill and when not. And then we have preference skills. Those are more durable, mostly encode some preferences. So if you have a specific workflow in your team or a specific style language or other preferences which are very, um, specific to your company, um, you will have or create
- 3:53
preference skills. And those, uh, preference skills are then protected with evals. Well, because most of, like, the foundation models might not, uh, integrate the knowledge which is very specific to your use case or your domain.
- 4:06
And preference skills are very valuable, so we really want to make sure that those are working and we don't, like, update our agents to, uh, degrade performance. So do skills work?
- 4:17
Yes, they do work. And I-- going back to Skills Bench, which has an update of one point one, which has evaluated all kinds of open and closed models in different harnesses, showing that skills on average improve the performance by roughly fifteen percent.
- 4:32
Skills Bench covers around one hundred different tasks, uh, based on, like, coding and also productivity across different languages. It's, uh, openly available. They have a very nice website, a very nice leaderboard, are also very open for, uh, community contributions.
- 4:47
And then they did a second analysis on self-generated or AI-generated skills, right? It's very easy if you are in a coding agent and you work on something, tell the model, "Create a skill," and then it writes a skill.md file.
- 5:00
We maybe look at it very closely. It roughly covers what we want to do, and then we just accept it and start using it. And what I found out is that, um, human-written skills are the best we can provide.
- 5:15
Uh, AI-generated skills can, uh, impact performance negatively and that skills or skills.md files should be below five hundred lines of words. So if you have your lo-laptop open [chuckles] and have a skill available, if you open that and if it's above five hundred lines, you should definitely look at the skill, uh, after our session.
- 5:34
And, um, the last topic about what is a skill and how a skill works, uh, we have different ways of triggering our skill, right? We can have a model-triggered skill, meaning, uh, based on the context and the description the model decides to use or read a skill to, uh, get more context to solve a task.
- 5:52
And then there are user-invoked skills. And I think people underestimate how powerful user-invoked skills are. Um, and they most of the time just accept, um, the
- 6:04
Overhead by pro-- like adding it into the context. I have like many user-invoked skills for more workflow type of tasks, like creating a pull request, uh, staging documentation, and all like of the very, uh, normal dev work which could be run in a script should most likely be a user-invoked, uh, skill.
- 6:25
And when you build agents for customer, you don't have those user-invoked skills. We are only working in the model-invoked skills, and that's where we are like focusing on for the small eval section we are going to look in a second.
- 6:36
So writing skills, um, is an important topic. Uh, we're going to look at eight, um, examples on how or tips on how you can write good skills. And most importantly, if you work with model-invoked skills, is the description.
- 6:52
Because the description are most of the time two sentences we provide to the system instruction to help the model know when it should use a skill or not. And it's bad if your s-description is too weak because then it might trigger too often, or it m-it might not be triggered if you need it.
- 7:09
So very important is the, the why and the how for the model, so why it should use that skill and then how it should use that skill. Um, very common is like use that skill if you are working on a React application, for example.
- 7:21
And then of course, the when. And we should write directives instead of essays. So we should not say something like, "Hey, the Interactions A-API is recommended for multi-chat, um, multi-chat because it handles like session state," and it's like where you should be way more directive.
- 7:38
Like use the Interactions API if you are working on like a chat application. So you need to give the model like clear instructions and directives on when it should use the skill and how it should use the skill.
- 7:50
And similar to what we have seen in the skills be-uh, bench, uh, results, we should keep the skill lean and layer information. So the description is the cost you always pay on every model invocation.
- 8:02
So on every model call, the description is part of the model context, so you always pay that one hundred, two hundred tokens cost, and you don't want to have a super long description because then you always have to pay that.
- 8:13
When you have a very long skill in the file, it will be always read into context when the model decides to read the, the, the skill or to use the skill, uh, which can be expensive as well.
- 8:22
That's why we want to keep it as concise as possible, but still include all of the reference and details for the model to solve that task. And then of course, the layer three is like we can have those reference files where the model needs to like, um, go really deep into a very specific task.
- 8:37
And a good example for this is like if you are working in like maybe a multi-cloud environment and you have a skill to deploy your application, you might need instruction to deploying to AWS and deploying to Google Cloud.
- 8:48
Those should not be part of your skill in the file. Those should be references, that you have a reference for AWS, reference for Google Cloud, maybe a reference for Azure, so that the model can basically explore based on the context where it should go to get all of that information.
- 9:03
Then we should set the right level of freedom. Um, I see many people very clearly describing the exact workflow in a skill. Step one, go there. Step two, do this.
- 9:13
Step three, do this. If you have those type of use cases, you should not use skills. You-- Maybe you should write a script because if the, the process or the workflow is always the same, you don't need to waste models and tokens for that exercise.
- 9:27
You can create a script. You can tell the model, "Use that script to run a specific workflow." So rather define goals and constraints. So if you need to like deploy to your or update or stage your documentation, describe how the model can do that.
- 9:42
Or like for your database, uh, updating a config, you should not say like, "Read the config, update the port," and then like pl- deploy again. The model knows what to do.
- 9:50
Just like, "Hey, if we need to change the config, here's the file, make the change." Then, uh, don't skip negative cases. So we always look at the when we want to use the skill, but most of the time we don't look at when we don't want to use the skill.
- 10:04
So if we have a description for our skill which says, "Use it for, uh, web development tasks," uh, it might over-trigger. Uh, maybe you work with Ran-React, maybe you also work with Angular, and the model always loads the skill if you're working in like a web development environment.
- 10:20
But if you are very specific for like, "Hey, only use that skill for React components or for Tailwind CSS," then the model knows, "Hey, that's very specific for one, uh, to use."
- 10:31
And with evals, we can also, uh, identify those. Um, and then test early. So that's what we are going to look at. We should really try to test when you create a new skill, always try to create ten of twenty prompts.
- 10:43
I like to create five for like the happy path. So when do I want to use that skill? Five when I don't want to use that skill, just to make sure the model is not over-triggering the skill and confusing itself.
- 10:55
And then if you have already some, uh, customer or production traces, try to include those as well because nothing is better than, than real world data. And then, uh, tip seven, which is quite new and I have to give all credits to Matt.
- 11:07
So if you don't know Matt, he's a great AI educator and you should definitely follow him. He, uh, published a tweet and also a skill on like killing all of the no-ops.
- 11:15
And what he found is that AI-generated skills tend to include a lot of no-ops. And no-ops basically is an instruction which does nothing to change the agent's behavior. It's like before or make an implementation easy to read.
- 11:29
Like the model knows how or when it should make something easy to read or write clear high quality code. I mean like that's what we expect from the model to do without telling it really.
- 11:39
So, um, definitely look at those no-ops. He have pu- has published a very good skill in, in his like skills repository. Uh, and then last but not least, uh, know when you should retire a skill.
- 11:51
Um, skills are not there to live forever. Models get better, behaviors change, expectation change, um, the environment changes. So, um, always try to run evals with and without the skill enabled.
- 12:05
And if the model achieves the performance without even like triggering the skill, you know you can retire that skill, save the cost, uh, for your tokens. And then also, um-
- 12:16
Don't keep, like, it redundant, so save cost at the end and maintenance also as well. And to look at a little bit of a practical example and also how you can create your own small eval or eval harness for skills, um, earlier this year we wanted to create a new skill for the Gemini Interactions API.
- 12:35
So the Gemini Interactions API is our new interface for working with Gemini models and with agents, and the Interactions API was released after the last training of Gemini. So the model or Gemini 3 and, like, 3.1 or even 3.5 has no context about what is the React...
- 12:52
uh, the Gemini Interactions API. So we decided, okay, let's look, uh, at creating a skill to help the model create good code for the Interactions API to use the latest models.
- 13:03
And to do that, we created 117 test cases. Those are, uh, based on, like, uh, data we see, uh, from real users trying to generate, uh, Gemini code from, um, synthetic generated, uh, test cases and also from, like, feedback we see people like, "Hey, the model is, like, using Gemini 2.0 even if we are already on 3.0."
- 13:24
And the, the end result was that we improved the, the performance up to, like, almost 90% for, um, generating valid Interactions API code with the latest Gemini models. And to do this, we basically only needed, like, two very simple, uh, assets.
- 13:43
So one of that was a, a JSON file with all of our test cases. And it's very, like, no clear structure. It's like, hey, we have a prompt. That's basically what we expect the user to provide.
- 13:55
We have a language because we wanted to test the skill against TypeScript and Python. Uh, we have a should trigger. That's basically there to tell us if the agent should read the skill or not read the skill.
- 14:07
And then we have different expected checks. Uh, we look at them in a little bit. Those are basically, uh, very simple asserts, uh, for that prompt if it should trigger or not.
- 14:17
And then we have a very basic Python script which runs a coding agent, in this case it was the Gemini CLI, which, uh, parses the output and returns it so we can, like, take a look at the, the outcome, whether we have valid code for the Interactions API or not.
- 14:32
And, uh, most of the tests or evals for skills can be regex. It's, like, very amazing how good of re-regex you can write using coding agents. And it's really, for us it was all about, okay, do we use the correct SDK?
- 14:49
Do we use the correct, um, model? Do we use the correct methods? Do we use any old patterns? And we created, um, very basic asserts for all of those cases, which are very cheap to run, so we can run our skill, uh, against the evals many times.
- 15:04
So if a new model releases, we have a very easy way to update those asserts to the latest, uh, model IDs, and it's very cheap to run for... it well because we don't need to use, like, LLM-as-a-judge.
- 15:15
But of course you can use LLM-as-a-judge if you have, like, more complex skills which need to look at the whole traces or the whole steps taken. And a very easy case is, like, you just create LLM-as-a-judge with a rubric on, like, what you want to look at, and then, like, take the output, put it through the LLM-as-a-judge,
- 15:33
try to get a pass or a fail. And then if it fails, look at the data, and then, like, try to, um, improve your skill based on that. And that's also how we now, uh, eval skills at Go- Google DeepMind.
- 15:45
So the-- We don't use YAML, but it's, like, just as, like, uh, an example. Uh, we have, uh, tests or evals alongside every skill we have internally at Google DeepMind.
- 15:59
Um, every test has multiple cases with, like, a prompt. Uh, we all run them in, like, clear workspaces. So you can define your workspace or environment, if it should include additional files like your application environments.
- 16:12
You have, uh, startup commands, which basically preloads or installs libraries into the environment. And then you have script validators. Those are those regex where we look at all of the traces to see was the skill triggered, was a certain command run, was a certain CLI run.
- 16:29
And then we ha- also have LLM-as-a-judge where we have some expectations which are basically matched against, like, hey, did it trigger the skill? Did it run a certain bash command to, like, also evaluate it?
- 16:41
And we run them on every change to the skill. So if a change happens to, or, like, a diff to the skill file, the eval will be run, and there will also be a result, and the change will not be merged if it is not improving the test cases.
- 16:56
So we always have those regression tests for every change to the skill, and you can only change the skill if it improves the eval or add new evals. And, um,
- 17:08
yeah, that, that's how we, we, we basically manage it. And then last but not least, uh, ten examples for best practices for skills. You don't need to take photos.
- 17:17
They are in the blog post I can share later. So, um,
- 17:22
the-- I mean, we had it many, many times. The, the s- skill description is very important. Uh, we have seen 50% of the failures, uh, because the skill was not triggered correctly because the prompt of the user was not, uh, detailed enough for the model to understand, "Hey, I need to use that skill to solve the task."
- 17:38
And especially if you build agents for others, they are not aware of the skill descriptions you have for your model and for your skill, so they might write something very, um, shallow, and then the model needs to know, "Okay, I need to trigger that skill."
- 17:54
Um, we should write directives over passive information, so we should always think about it. You should tell the agent what to do or not what to do and not just like, "Hey, if you feel happy today, please use the skill."
- 18:06
Um, include negative tests. Uh, we always forget negative tests. Start small. Even, like, 10 to 20 skill eval samples are better than nothing. You will be surprised on how much you will find even from, like, five to ten examples.
- 18:20
And then, like, definitely create outcomes, not paths. Uh, we don't want to- Test if the model loads the skill on, like, the first turn. We really want to test if it can achieve the task based on the prompt, and if it loads the skill, it loads the skill.
- 18:33
If not, then not. If it, um, loads the skill after five turns, that's also okay. Then we want to have isolated runs because coding agents are very good at finding or cheating.
- 18:46
So if you run inside, uh, your existing environment, it might look up previous chats or it might look up some other executions and then, like, try to cheat it and get the context from the skill without even using the skill.
- 18:59
Then definitely run more than one trial when running evals, like agents. Our models are non-deterministic. Maybe the first run works, the second one doesn't, so always run three to six, uh, trials per case.
- 19:12
And to measure reliability, um, test across different harnesses if you work with, like... or if you have employees or, um, people working with, like, different harnesses, not only just evaluate against Claude Code or AntiGravity.
- 19:25
If you have people working with Cursor, try to include them as well because agent harnesses behave differently and, of course, model behaves differently. So maybe your skill is very good with a Gemini, but very bad with Codex, and then you have, uh, customers, consumers using your harness with Codex, and then it fails.
- 19:41
And then, um, create your eval. So if your, um, model is good enough, does-- it doesn't need the skill anymore, keep that eval. You don't need to throw that eval away because you throw the skill away.
- 19:53
You can keep that eval to make sure that the model or the agent keeps the performance, and as soon as you start seeing some degradation, you can reintroduce a skill.
- 20:01
You can maybe tweak some other tools or pieces to keep, like, the, the performance up, and then really detect when you can retire a skill, and you will be very surprised with all of the model updates, how fast you can retire a skill which you might needed, like, six months ago, but not today anymore.
- 20:18
And I have some homework for you. So if you are back from holiday on Monday, um, pick, uh, the most used skill and write five test prompts. Uh, you can also use your coding agent and ask it to see, look at your tra- uh, trajectories, which are my most used skills, and then try to create some, some
- 20:36
skills. As you have seen, it's, like, very easy to write your eval harness. It's like a JSON or YAML file and then, like, some Python script which runs your coding agent or your agent harness and then, like, look at the outcome.
- 20:48
Definitely, uh, try to look at, uh, removing no-ops. Maybe it does not change the eval performance, but it helps you save cost because all of the tokens which are not helpful or not changing the agent behavior are money you will, like, spend.
- 21:03
So look at, um, writing create skills, uh, from Matt. It's-- You can find it on, on GitHub. And then also run ablation tests. So run always evals with your skill loaded and without your skill loaded.
- 21:16
Only that way you will know when you can retire a skill or if a skill is really helpful for your performance. So don't ship skills without evals. Thank you. [audience applauding] [outro music]