Don't Ship Skills Without Evals

Do you use coding agents? Do you use skills with them? Do you have evaluations for those skills? Philipp Schmid, introducing himself as working on the Gemini API and agents at Google DeepMind, opens with that sequence of audience questions. Hands go up for coding agents and skills; very few remain for evals. The gap matters because accepting a plausible SKILL.md file is much easier than establishing that it helps.

Schmid describes SkillsBench as having indexed more than 50,000 skills from GitHub, with almost none carrying evaluations and many written by AI. Those are his ecosystem observations. The immediate engineering problem is independent of the inventory size: agent nondeterminism makes a failed task hard to diagnose. Was the skill wrong, did the agent fail to discover it, or was the task beyond the model’s capabilities? One successful run cannot separate those possibilities.

The distinction becomes sharper when moving from agents developers use to agents developers build. In Antigravity, Cursor or Claude Code, an engineer asking for a Gemini API feature can notice that the relevant skill was missed, stop the task, re-prompt or invoke a slash command. A customer asking for a refund does not know that a refund skill exists. They ask for help with their problem, not for a particular piece of the agent’s implementation. Customer-facing agents must discover the right instructions from ordinary requests without relying on the user to repair routing.

A skill is a folder containing SKILL.md and supporting assets. Its defining mechanism is progressive disclosure: the agent starts with enough information to discover the skill, then loads instructions and deeper references only as needed.

The description therefore has a different job from the body: it helps the model decide whether to retrieve the instructions at all.

Schmid distinguishes two reasons for supplying those instructions:

• Capability skills teach something the model cannot yet do consistently, such as tracing logs or creating a React application. As models improve, an evaluation may show that the skill is no longer necessary.
• Preference skills encode team workflows, style conventions or company-specific requirements. These are more durable because a foundation model is unlikely to acquire those private preferences through general training. Evals protect them against regressions when the agent changes.

The same evaluation machinery supports two different decisions: retiring temporary assistance and preserving behavior that remains specific to the organization.

There is evidence that skills can help. Schmid summarizes SkillsBench 1.1 as roughly a fifteen-percent average improvement across open and closed models in different harnesses, covering around a hundred coding and productivity tasks across languages. The published release makes the unit and scope precise: mean task resolution rose from 33.9% to 50.5%, a 16.6-percentage-point increase, across 87 tasks and 18 model–harness configurations. Its public leaderboard and contribution process make the benchmark useful for investigating where those gains occur.

Generating a skill and accepting it after a quick skim is a different proposition. Schmid emphasizes the stronger results from human-written skills and the possibility that generated skills reduce performance. In the release’s self-generation experiment, all three tested configurations fell below their no-skill baselines; that does not establish that all AI-assisted authoring is harmful. His practical authoring recommendation is to keep the SKILL.md body under 500 lines and move detail into references, rather than treating a long generated document as inherently more capable.

First choose how the skill should be invoked. A model-triggered skill depends on the model recognizing a match between the request and the description. A user-invoked skill starts with an explicit instruction, which is often a better fit for familiar developer workflows such as creating a pull request or staging documentation. Explicit invocation can avoid keeping routine workflow guidance in the agent’s standing context. For the customer-facing scenario in this talk, the evaluation problem centers on model-invoked skills: the user will not name the mechanism.

The first of Schmid’s eight writing tips is to treat the description as functional routing logic. Often only a couple of sentences in the system instructions, it must communicate why the skill is useful, when it applies and how it should be used. A weak description can fail in either direction: never activating when needed or activating for too many tasks. A React skill, for example, needs a trigger that identifies React work rather than vaguely advertising development expertise.

Inside the instructions, prefer directives over explanatory essays. The talk contrasts passive information about the Interactions API being recommended for multi-turn chat because it handles session state with a direct instruction to use that API when working on a chat application. The purpose is to change the agent’s choice, so the instruction should state the choice and its condition.

Keep each disclosure layer lean. Schmid uses a description of roughly 100–200 tokens to illustrate a cost paid on every model call where it is included. The body adds another context cost when read. Detailed references should therefore remain accessible without being loaded indiscriminately. For a deployment skill spanning AWS, Google Cloud and Azure, put each provider’s instructions in a separate reference file; the skill body should help the agent find the relevant provider, not force it to read all three deployment guides.

A skill should also leave the right amount of discretion. If a workflow always consists of the same steps in the same order, put those steps in a script and let the agent invoke it. There is little benefit in spending model tokens reconstructing an invariant procedure. Where judgment is useful, describe the goal and constraints: how documentation can be deployed, updated or staged, for example. For a database configuration change, identify the configuration file and the requested modification instead of spelling out every read, port edit and subsequent action.

Boundaries matter just as much as positive triggers. A description that says to use a skill for web development can cause React-specific guidance to load during Angular work. Narrowing the trigger to React components or Tailwind CSS gives the model a more useful decision boundary. The corresponding evaluation needs examples where the skill should remain unloaded, not just examples where it should activate.

Start testing while writing the skill. Schmid recommends an initial set of 10–20 prompts: five happy-path requests, five requests that should not trigger the skill, and customer or production traces where available. Real requests expose wording and missing context that a skill author may unconsciously avoid when inventing tests.

Then remove behavioral no-ops. Schmid credits educator Matt and his skills repository for this technique: delete instructions that do not change what the agent does. Generic requests to be thorough, make an implementation easy to read or write clear, high-quality code may merely restate behavior already expected from the model. Useful skill content supplies a missing constraint, decision or piece of knowledge.

Finally, compare runs with the skill enabled and disabled. Models, environments and expectations change. If the model reaches the same performance without the skill, retirement can remove both token cost and maintenance work. This ablation test is the bridge from writing plausible instructions to measuring whether those instructions still earn their place.

The practical example is a skill for the Gemini Interactions API, an interface for working with models and agents. Schmid describes a training-knowledge gap: the API arrived after the relevant training, leaving Gemini 3, Gemini 3.1 and Gemini 3.5 without the context needed to use it reliably. The skill’s job was to supply that missing API knowledge and steer generated code toward the latest models. This is the talk’s historical motivation, rather than a claim about what every current model knows.

The team assembled 117 test cases from real-user code-generation requests, synthetic cases and feedback about outdated model choices—for example, code still selecting Gemini 2.0 when Gemini 3.0 was available. Schmid reports almost 90% performance on generating valid Interactions API code with the latest Gemini models in this 117-case evaluation. The spoken account does not supply the baseline or a complete scoring definition, so the result is best read as the outcome of this particular skill-development exercise.

The harness needed two assets: a JSON file of cases and a basic Python script. Each case recorded the user prompt, the target language—TypeScript or Python—a should_trigger flag and expected checks. The runner invoked Gemini CLI, parsed its output and made the result available for validation. Keeping the trigger expectation separate from output checks lets the evaluator distinguish a discovery problem from incorrect code after discovery.

Many of the case study’s requirements were mechanically detectable: the right SDK, the right model identifier, the right methods and the absence of obsolete patterns. Regex assertions made these checks cheap enough to repeat frequently, and coding agents could help author the patterns. When a new model arrived, the team could update the expected identifiers rather than introduce an LLM judge for a simple string-level requirement.

A compact Python validator can keep those requirements in the case data. Here, required and forbidden map readable check names to regex patterns; the harness supplies generated output and whether the trace showed a skill read.

import re


def evaluate_case(case: dict, output: str, skill_read: bool) -> dict:
    checks = {
        "trigger": skill_read == case["should_trigger"],
    }
    expected = case["expected_checks"]

    for name, pattern in expected.get("required", {}).items():
        checks[f"required:{name}"] = bool(re.search(pattern, output))

    for name, pattern in expected.get("forbidden", {}).items():
        checks[f"forbidden:{name}"] = not re.search(pattern, output)

    output_checks = {
        name: passed
        for name, passed in checks.items()
        if name != "trigger"
    }
    return {
        "checks": checks,
        "trigger_pass": checks["trigger"],
        "output_pass": all(output_checks.values()),
    }

This separates trigger diagnostics from output requirements. Regex can establish that a required pattern appears or an obsolete one does not; execution-level correctness needs a validator that actually checks that property.

For more complex skills, the relevant evidence may be the whole trace: which steps the agent took, what it inspected and whether the final result met a qualitative requirement. Use an LLM judge with an explicit rubric for those cases. Feed it the output or trace, request a pass/fail decision, then inspect failures and revise the skill. The distinction is about what must be evaluated, not a requirement to use a model judge for every test.

Schmid describes Google DeepMind’s internal approach as keeping evaluations alongside every skill. The YAML shown in the presentation illustrates the structure, rather than representing the team’s actual storage format. Each evaluation has multiple prompt cases and a clean workspace that can include application files. Startup commands install or preload the libraries needed for the task. Script validators inspect traces for skill reads, commands and CLI execution; LLM judges assess expectations that need broader interpretation.

Every change to a skill file runs the evaluations. Schmid describes a merge rule under which a change must improve the evaluated cases or add new evaluations. That makes the test suite part of the skill’s maintenance contract: changing instructions requires evidence about the resulting behavior, rather than a judgment that the new wording sounds better.

Discovery remains a major source of failure. Schmid reports that 50% of the failures his team observed involved incorrect skill triggering when user prompts lacked enough detail. He does not specify the sample behind that figure. Customers do not know the descriptions in the agent’s context, so short or underspecified requests must be part of the evaluation set. Clear directives, negative tests and a small initial collection of prompts can expose these problems before a large suite exists.

But triggering is a diagnostic, not the final objective. Do not require the model to load a skill on its first turn merely because that is the path the author expected. Loading it after five turns can be acceptable. Completing the task without loading it can also be acceptable. Keep the trigger result visible, as in the Python validator, while defining success in terms of the requested outcome. Otherwise the evaluation can punish a capable agent for taking a different successful route.

Isolation prevents an apparently successful run from borrowing the answer. In an existing development environment, an agent may find previous chats or artifacts from earlier executions and recover the skill’s context without reading the skill. A clean workspace removes that accidental assistance. Repeat the experiment as well: Schmid recommends three to six trials per case because a first-run success may fail to recur.

Cover the harnesses people actually use. A skill evaluated only in Claude Code or Antigravity may behave differently in Cursor. The model is another variable: good results with Gemini do not establish good results for customers using Codex. Reliability belongs to the combination of task, skill, model, harness and environment, so the evaluation matrix should reflect the combinations that matter to the deployment.

When an improved model makes a skill unnecessary, retain its eval. The test still expresses a behavior the product needs. If performance later degrades, it can reveal the need to reintroduce the skill or adjust another tool or agent component. A capability that required explicit guidance six months ago may now work without it; the durable asset is the test that tells you whether it still works.

Schmid closes with an exercise small enough to do on returning from the holiday:

1. Pick the most-used skill. If that is unclear, ask a coding agent to inspect your trajectories and identify frequent skill use.
2. Write five test prompts for it.
3. Store the cases in JSON or YAML and use a Python script to run the agent or harness and inspect the outcomes.
4. Remove no-op instructions. Even when scores stay unchanged, unnecessary tokens still cost money.
5. Run the same evaluations with the skill loaded and without it.

That last comparison establishes whether the skill is helping and whether it is ready to retire. The instruction to stop shipping untested skills does not require a large evaluation platform: it starts with a few representative requests and a repeatable way to check what happens.