AI Engineer Europe 2026
Malleable Evals: Why Are We Still Evaluating Adaptive Systems with Static Tests?
Read the talk
Malleable Evals: Testing Agents That Keep Changing
As agents adapt their tools, behavior and user experience, evaluation needs to evolve with them—from fixed examples to trace-derived tests and telemetry-guided feedback.
From a talk by Vincent Koc
Before you start: Familiarity with unit tests, LLM tool calling and application traces will help you follow the evaluation mechanisms.
When intended use stops describing actual use
Around 2013, Vincent Koc tried VR goggles whose warning label limited use to five minutes. He recalls using them for three hours, then vomiting for three hours afterward. The device’s intended use and his actual behavior were very different—and that difference is where testing becomes interesting. Emerging technology invites experimentation, including uses its designers would rather people avoid.
Koc brings that experimental perspective to evaluation research at Comet. He describes working with universities and benchmarking systems for organizations including Uber, Netflix and UK banks. Against that background, the joke that evaluations are dead raises a practical question: what must evaluation become as the systems it measures grow more adaptive?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The missing chaos-engineering layer
Software testing starts with an expectation: this thing is meant to do something. Examples become unit tests. A manual regression suite records that doing A and B sometimes produces an unwanted C. CI/CD checks help keep that behavior from shipping again. Beyond those known cases, chaos engineering deliberately stresses and breaks systems to discover their limits, while observability helps engineers understand what happened.
AI evaluation often stops earlier. Consider a risk-compliance check intended to prevent an assistant from selling financial services. A team handcrafts questions, tunes the system until it handles them correctly, and cycles through those questions offline before deployment. That establishes behavior on the selected cases, but leaves the exploratory layer underdeveloped.
| Practice | What it addresses |
|---|---|
| Static benchmark | Performance on a fixed collection of tasks |
| Hand-curated evaluation | Specific behaviors the team anticipates |
| Pre-deployment evaluation | Whether the current system passes before release |
| Chaos engineering and observability | What happens when behavior stretches beyond those expectations |
Koc’s gap is the last row: how do teams discover where an agent breaks, rather than only checking the examples they already know?
Adding more datasets does not automatically close that gap. A benchmark about adding numbers may measure a capability, but its relevance to a particular application still needs explaining. Teams can accumulate a huge collection of evaluations and nevertheless return to diagnosis when an unexpected production failure arrives.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When the harness changes, what does a passing test mean?
The application being evaluated is itself changing. Ordinary software releases can require revised tests, but adaptive agents make that relationship more immediate. Koc identifies himself as a core contributor to OpenClaw and describes harnesses that can shift their behavior and create skills. As those capabilities change, a benchmark tied to an earlier application can become less representative.
An adaptive-testing paper he introduces asks whether benchmarks can change with applications. He describes its approach as more selective, smarter testing: choosing what to test more intelligently. That is a step toward adaptive evaluation, but it is narrower than maintaining an evaluation system that follows changes in real usage. Selecting better questions from an existing test collection and changing what the collection needs to cover are different problems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From wordsmithing prompts to testing components
The route to this problem starts with prompt engineering. In a banking or creative application, a developer changes instructions, tries more words, and watches what comes out. Koc compares this to developing a medicine for liver disease and discovering that it relieves pain: a useful effect can emerge without the original experiment explaining it well. His rhetorical claim that prompt engineering died in 2023 comes with an acknowledgment that people still work this way. The target of the criticism is unguided wordsmithing.
Context engineering makes the system’s moving parts more explicit. Incoming data, search, retrieval-augmented generation and tool calls create steps that can be inspected separately. A large agent might contain an MCP sales tool; that tool can be tested for its intended behavior independently of the whole application. Decomposition gives evaluation clearer targets and makes failures easier to understand, even though passing component tests does not settle the behavior of the complete agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More generation, more adaptation
Greater access to tokens accelerates this process. Koc concedes that whether tokens are cheap is debatable, but their availability makes more generation possible. His phrase “tokens become fast food” captures a feedback effect: developers consume more tokens, generate more code, and increase the velocity of application development.
Model capability expands what that generated software can do. Koc describes optimization work in which general-purpose models solve individual ARC-AGI-2 puzzles. He also recalls trying ARC-AGI-3 and finding some puzzles difficult for humans, while emphasizing models’ ability to recognize useful patterns. ARC-AGI-2 uses input-output tasks; ARC-AGI-3 introduces interactive environments with unstated rules and goals. His examples include no model identity, harness, budget or measured score, so they illustrate capability rather than establish completion of either benchmark.
That leads to intent engineering: systems that adapt toward what a user wants. Koc points to OpenClaw and harnesses around Claude and Codex as examples of systems trying to understand users and improve their experience. Personalization creates a new testing difficulty. If the system adapts differently to you and to me, a single expected interaction may no longer describe both experiences.
This increases the need for evaluation and observability. Teams need to understand what happens inside the different layers of an agentic application. A statement such as agents being insecure is too broad to guide a fix; visibility into behavior is what can turn that concern into a specific, actionable problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define outcomes, then let traces reshape the tests
An intent-based outcome requires a different evaluation target. For arithmetic, an exact-answer check can establish that one plus one equals two. For an assistant, the questions may instead concern how it handles ambiguity, expresses personality or meets an organization’s expectations. Rubrics make those criteria explicit, much as a school can evaluate artwork without requiring every student to produce the same picture.
The test suite can also change as usage changes. Koc imagines a workload in which 80% of traced activity is familiar, then asks what happens when the customer base changes. New customers ask different questions and use the agent differently. Those traces provide evidence that the evaluation suite may no longer represent the application’s workload.
His proposed trace-driven process is:
- Collect traces of the application’s behavior.
- Feed those traces to an evaluation agent that looks for changes in usage.
- Notify users or agent owners when something has changed.
- Revise the evaluation suite to cover the changed behavior.
The agent helps curate the tests rather than leaving that work entirely to people. This is a proposed maintenance loop: the detection of a change supplies a reason to update coverage.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put operational feedback inside the loop
Once agents read traces and perform evaluations, the process can operate as an always-on evaluation and optimization service. Evaluation no longer has to wait for someone to assemble a dataset and launch an offline run.
The next step is telemetry in the loop. Koc mentions a paper he has written on this idea: expose operational information to the harness running the software, MCP tools or agentic system. Two kinds of feedback matter in his description:
- Failures: Give the harness information about what broke so it can attempt a repair and continue.
- Cost: Give it information about resource spending, together with conditions that guide acceptable behavior.
The feedback changes what the harness can respond to. Instead of relying only on failures anticipated during development, it can use evidence from the current run. Koc describes this as enabling self-correction to some degree, not guaranteed recovery from every error.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the desired end state the evaluation target
Koc tentatively calls the alternative eval calcification: evaluation becomes increasingly rigid while the system it describes continues to change. To explore a different pattern, he points to Karpathy’s autoresearch, a Python-based optimization loop that tunes and tweaks toward a goal. The transferable idea is to specify a target, evaluate a candidate change, and use the resulting signal to guide another iteration.
His deliberately everyday example is barbecue. Optimizing for the tastiest mix and optimizing for the cheapest mix are different tasks. The user’s intent determines which reward signal makes sense. A small Python example makes that distinction concrete with illustrative candidate values:
python
from dataclasses import dataclass
@dataclass(frozen=True)
class Mix:
name: str
cost: float
taste: float
mixes = [
Mix("smoky", cost=6.0, taste=8.0),
Mix("pepper", cost=4.0, taste=7.0),
Mix("sweet", cost=5.0, taste=9.0),
]
def reward(mix: Mix, intent: str) -> float:
if intent == "tastiest":
return mix.taste
if intent == "cheapest":
return -mix.cost
raise ValueError(f"Unknown intent: {intent}")
def choose_mix(intent: str) -> Mix:
return max(mixes, key=lambda mix: reward(mix, intent))
for intent in ("tastiest", "cheapest"):
print(intent, choose_mix(intent).name)
The candidates stay the same; changing the objective changes the preferred result. The taste values would have to come from some assessment—the function cannot discover what tastes good merely by naming that objective.
An adaptive loop would go further than choosing among those fixed candidates: it would propose changes, assess them against the objective, and use the feedback to correct its next attempt. Evaluation becomes a specification of the desired end state, rather than merely the dataset used at the beginning. That is the broader direction Koc draws from automated optimization: people define what they want the system to reach, and the machine performs the iterative work toward it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Maintain the changing edge of behavior
Koc closes with an illustrative split: 80% of behavior is stable and defined through intent, while 20% keeps changing. Those proportions are a thought experiment, not measured workload statistics. The changing portion contains the strange question or unexpected use that can cause serious business problems. His proposal is to have agents monitor and maintain that portion, adapting evaluations as those behaviors emerge.
This makes evaluation something to maintain as code, software or a living agent: a growing, self-optimizing system rather than a point-in-time dataset. The implementation was still in progress at the time of the talk. Koc did not show the planned in-depth Comet demonstration because its end state was unfinished; he expected it to be ready in the following weeks.
The conceptual framework is meant to transfer to other teams’ systems. Agents can shift their own behavior; the surrounding problem and datasets can change too. Evaluation therefore needs its own capacity to observe those changes, revise what it tests and guide correction toward the intended outcome.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and setup instructions for the personal assistant that connects models, tools and messaging channels.
A single-GPU experiment loop in which an agent edits training code and evaluates changes against validation bits per byte within a fixed training budget.
Further reading
The benchmark's original paper explains its input-output tasks, abstract-reasoning goals and human testing.
- Introducing ARC-AGI-3Article
The March 2026 introduction to interactive environments where agents must discover rules and goals through exploration, with links to the games.
- How ARC-AGI-3 is scoredDocumentation
Explains how completed levels and action efficiency relative to human players contribute to benchmark scores.
- Mind the MetricsPaper
Koc and collaborators describe an architecture for bringing telemetry, traces and evaluation feedback into IDE and agent development workflows through MCP.
ATLAS uses psychometric models to select informative benchmark questions and estimate model ability with fewer items. The linked paper includes a July 2026 revision.
Read the complete timestamped transcript
- 0:00
[upbeat music] Cool.
- 0:16
Hey, everyone. Uh, thanks for joining this session. Sorry if my sound's a little croaky. I've done three talks back to back. So one on Wednesday, one yesterday, keynote, and then a workshop style session today.
- 0:29
So I'm Vincent. I'm gonna be talking about malleable evals, um, from static AI measuring, uh, to adaptive systems. Now, let's jump into who I am, what I do. I call myself the friendly cancer.
- 0:43
I use AI, I use technology. I'm always on the edge. Um, for those of you that haven't seen my keynote, I do, um,
- 0:51
yeah, I just live on the edge and, and just do some fun stuff. So this is me using VR goggles in, um, like, back in twenty thirteen when, like, people hadn't even heard of VR.
- 0:59
It came with a warning label, said, "Only use it for five minutes." I used it for three hours, then I vomited for three hours after that. [laughs] So measurement, anything we do in technology, anything on the edge is gonna be janky, it's gonna be weird, and that's kind of fun in my opinion.
- 1:14
Now, whenever we talk about evals to people, and, and a little bit of pretext, like my, my role at, uh, Comet, um, work in evals, I do an eval research.
- 1:24
I work with universities. We benchmark and run evals for large set of companies and organizations, everything from like Uber to Netflix to, to banks even in the UK. Um, but the, the thing that's been going on right now is that, hey, like this kind of joke that like evals is a little bit dead.
- 1:40
Um, and it's a little bit of a joke, but there's a little bit of truth to it as well. And I'm gonna hopefully like kind of walk you through the mindset shift and hopefully explain a little bit less about evals, but like what's actually happening, um, in the sort of agentic AI space, and then how do we
- 1:54
then translate that back to evals. So when we think about like software engineering as a practice, when we're thinking about like how do we measure things, we kind of look at it from the sense that, you know, we're gonna start with like this thing is meant to do something.
- 2:07
Um, so we-we'd start with a set of examples and write some unit tests. Um, we might do like a manual regression suite which is like, hey, when we do A and B, sometimes C happens and C is unfavorable.
- 2:19
Like let's not do that. Let's not make people vomit when they put their VR goggles on. Um, we could do things like CI/CD pipelines to like make sure that like the thing ships out and works the way it's meant to and intended to.
- 2:33
But mostly like we do things in engineering known as chaos engineering and observability. Um, for those of you that are in-unfamiliar with the term chaos engineering, it's basically where you're like doing all kinds of random stuff and just breaking it and just having fun with the technology and just seeing where you can stretch it and where you
- 2:48
can go. Now, when we apply this to AI and data science space as you-- we traditionally know it in the last little while, twenty twenty five included, uh, we do things like static benchmarks.
- 3:00
Like we have these like evaluations. It's like, oh, um, I'll give you an example. It's like my, how, how compliant is my AI in like risk? I'm gonna ask it a bunch of questions and make sure it doesn't talk about, you know, selling me some financial services 'cause that's a big no-no.
- 3:17
Um, we will then handcraft like a set of questions and examples and sit there and like tune this thing up, make sure it's absolutely perfect. Um, before we deploy the, the AI system or the, the models, we'll do some like some sort of offline evaluation where we're just kind of cycling through those tests.
- 3:35
But we're missing that sort of chaos engineering space. We're missing that, you know, like what comes next and how do we mess up with it and how do we know where we can stretch this thing?
- 3:44
And I think that's like a, an honest gap that we see in this space. And that's why we're just so hyper-fixated on benchmarks and evaluations. If you go to any AI conference, uh, in the academic space, all people talk about is like benchmarks.
- 3:56
I created a benchmark for like, um, ad-adding numbers and what LLMs think about it. It's like well great, but like how is this actually helping me? So then you end up with like this huge humongous set of like data sets to try and somewhat explain what is happening with your agent until something goes wrong.
- 4:12
And it's a matter of time before something goes wrong, and it will, and you're kind of back to the drawing board and trying to figure out what's going on.
- 4:21
And the reason for that is that our AI applications are not static, but we're treating them like they're static software. Um, yes, when we ship software we might change unit tests.
- 4:30
They're a little bit quicker to do. But realistically speaking, even software is becoming malleable. Um, so flip to my keynote I gave, uh, yesterday where I'm one of the core contributors of something called OpenClaw.
- 4:42
The harness changes itself. Like the harness will shift. Like you wanna create skills, you wanna do other things, like it will adapt, right? So that adaption that we're seeing inside of things where software is being shipped at lightning speed, how does your benchmarks keep up with that?
- 4:55
Like how does your benchmarks adapt to that space? Um, this is one of many papers that are out there. Um, I don't remember when this one was published, but this concept of like adaptive testing for LLM evals.
- 5:08
This concept is like somewhat revolutionary maybe, but like what happens if our benchmarks would change with our, with our applications? Uh, I didn't write this, but you know, great that someone did.
- 5:18
But it just kind of poses the question that like why are sta-benchmarks static? Like why don't we test in a more sort of adaptive manner? So this could be great.
- 5:26
This is more like selectively testing and just being a bit more smart about how we test. But it's still like, you know, taking us in that journey. I think it's like a mindset shift.
- 5:36
Now rewind to like what we're seeing in the AI space for a minute. Um, we had prompt engineering if we like focused purely on LLM, uh, space. Um, we had this prompt engineering world where it's like, hey, I'm gonna like doom scroll, uh, wordsmith instructions.
- 5:50
I'm gonna just like bash random words into an AI and hope it improves. So if I'm building this like banking app or creative app, I'm like gonna stick all kinds of random words and see what comes out the other end and makes it creative.
- 6:02
It's a little bit akin, and I'm not trying to downplay medicine in any way, it's like, hey, I'm gonna make medication for like, uh, I don't know, liver disease, and turns out it cures pain.
- 6:13
Okay, these are painkillers now. That's great. And the same thing we're doing with, like, prompt engineering. We're just, like, bashing words into it and hope it changes. And for some reason, this died in like 2023, but peop-people still do it.
- 6:23
It's kind of in-in-in-intense. Um, and then we kind of went into this, like, world of context engineering, and I think this started making evals a little bit more relevant because it was a little bit more complicated.
- 6:34
You know, there were steps involved. There was, like, data coming in and search was, like, a thing. And we're starting to steer the agents in a direction, um, with things like RAG and tool calling.
- 6:47
And the beautiful part is there's like, well, okay, I'm a-- I'm an organization of this big agentic system. Maybe I can break this agent up into its parts. Maybe I can go, oh, I have this MCP tool that does like some sales agent thing.
- 6:59
I can test that thing is doing what it's meant to, right? I can just go off and like be sure that that thing is happening. So this come o-- this, this process of like, you know, tool calling and, and kind of breaking this larger agentic piece up into its, its sum of its parts, made evaluation somewhat steerable
- 7:16
and a little bit more un-understand, but it still didn't kind of hit the, the head on it, on its head. Um, but then in 2025, like, where are we going next?
- 7:25
I mean, if you look around, we, we can see that code is cheap. Um, that's not a changing thing. Like tokens are available. Debatable, uh, if you think tokens are cheap or not, but you know, they're, they're there, which makes tokens cheap.
- 7:37
And then tokens become fast food. Essentially, we can consume more tokens, therefore we generate more. We, we-- the velocity of creating software and applications increases, um, and models become really good.
- 7:48
And I think this is the thing that I think, I think a lot of people just have not yet comprehended that a lot of the AI applications that are now running, the models can do absolutely amazing things.
- 7:59
Um, I've been working on a lot of like optimization problems, and we can take these models that are somewhat like seen as like these generic systems and be able to do like amazing things like solve ArcadeGI2, which is like puzzles.
- 8:12
And if-- I recommend anyone who's like interested in evals, look at the ArcadeGI2 puzzles. I've tried the, uh, ArcadeGI3. Some of these puzzles are like really hard for humans to solve, but like machines can like pattern recognize, and LLM can actually pattern recognize and, and start to solve those.
- 8:27
So what that brings us to is like intent engineering. Um, this kind of concept that like machines can self-optimize based on intent, right? And we're seeing this with the harnesses that we're seeing coming out where, you know, we've got this with like OpenClaw, but we're also getting this with like other types of harnesses inside of Claude and
- 8:44
Codex, where it's trying to understand you and it's trying to adapt to you and give you a better experience. Now, the problem with this is that when we have intentful machines, the evaluations become even more complicated because it's like, how do I know my experience is different from your experience and different from someone else's experience?
- 9:01
Like, how do we start to build testing around this sort of methodology and understanding? And I think the complicated part of this is that it, it just kind of exacerbates the, the, the kind of need for evaluation even more.
- 9:15
Um, there was this kind of joke, like I was saying earlier, like people saying, "Oh, evaluations are dead. They're gonna go away. Observability is dead, they're gonna go away."
- 9:21
But realistically, now more than ever, people wanna know what's happening inside of these agentic applications within their different layers because then it gives them some understanding of what's, what's going on.
- 9:31
Like we, we use words like, "Oh, these, these agents are insecure," or, um, "We're not sure what's happening." So how do we actually-- how do we actually turn that into something meaningful?
- 9:42
So going back to my earlier slides on kind of this concept, let me just, let me just recap where I was for everyone. You know, we said there was this like...
- 9:52
We had these static benchmarks. We hand-created evaluations. We would do like these offline evaluations, and we had this big gap.
- 10:00
Uh, what we're actually moving towards is like this intent-based outcome. So if you-- if we think about like there's this, this concept of like intent engineering that, that I'm, I'm mentioning, like how do we actually map that to something?
- 10:11
So instead of saying, you know, one plus one equals two, or users asking this very specific question and this is the answer and this is what we're gonna compare towards, it's like how do we deal with, uh, how do we define, uh, ambiguity in agent?
- 10:24
How do we define personality inside of an agent? And, and how does that look like for an organization? And some of the research is showing things like, oh, we can build rubric.
- 10:33
We can do it like how we, how we, you know, evaluate art pictures and, and things like that in, in, in schools. Uh, we can self-curate, uh, suites from traces, as in not me, but the agent can.
- 10:47
You know, once we start tracing these applications, let's just say eighty percent of the time it's the same stuff that's happened to my agent. But now suddenly my customer base has changed.
- 10:56
And because my customers have changed, they're gonna start looking at things, they're gonna start asking questions differently. Things are gonna start changing inside of my agent. But why are we not measuring this?
- 11:06
Like, why are we not taking these traces and feeding them into agents and going, "Something has changed," and then telling that to the user or telling that to the owners of these agents and changing the, the, the suites, the tests.
- 11:19
Uh, we can do online-- always-on evaluation optimization. So like to that point, once we start looking at the traces, once we have agents doing the evals, not static benchmarks, we can have this like as an always-on sort of service.
- 11:32
And then lastly, we can do this sort of like telemetry in the loop. Um, I have a paper that's been written on this, which is essentially when we, when we're writing software applications or MCPs or anything like that, agentic systems, if the harness is aware of the telemetry, it's aware of like what's breaking, it's aware of like
- 11:49
how much it's costing, and you can set some conditions around it, it can kind of self-correct itself. So, uh, we're starting to see this with harnesses where, you know, it's had an error, it's had an issue, and it's gonna fix itself, it's gonna continue on.
- 12:00
So I think this is a kind of a case of like instead of trying to predict what's gone wrong, like how can we be more smart about using that data back into the agent to be able to kind of make it heal itself, uh, to some degree.
- 12:12
So I'm kind of calling this like a calcification problem, like the-- or eval calcification. I'm, I'm still stewing it on my head. Sounds like a really nice paper title.
- 12:21
Um, but this idea that it's just gonna like become harder and harder unless we can get so-kind of smart about it. And I think one of the kind of concepts I want you to kind of stew on and think about, this is like one of the auto research, um, outputs that you can do if you haven't tried
- 12:34
it, like Karpathy's AutoResearch. Like this really basic sort of au-auto optimization using, uh, Python. You set a goal, you set a target, and it kind of tunes itself and tweaks itself.
- 12:45
You could do this with absolutely anything, right? You could do this with like, I don't know, what's the best mix to, to, to... What's the tastiest barbecue or the cheapest barbecue mix that you, you wanna make?
- 12:55
It could be anything, right? You just set a reward signal. But the, the, the, the key here is that your users are gonna have a, a, a point of intent that you wanna sort of optimize towards.
- 13:04
And then how do we sort of get the machine to like correct itself and, and kind of loop towards that, uh, as an eval? So then our evals don't become the dataset or the starting point.
- 13:14
Our evals become like what is the end state that we wanna get to, and then we just let the machines do the work. We have evaluations where it's just the agent and we're just defining the end state.
- 13:23
So just one last thing to sort of bake into your minds before, and I'm gonna finish a little early, uh, so we can do questions or anything else, it's fine.
- 13:30
Is that you can imagine this space where you go like 80% is like the static stuff, like it's been defined in an intentful manner. But that 20% is always gonna keep changing, and it's that 20% that's gonna mess up your business.
- 13:42
It's gonna be someone who's gonna come and ask a weird question or use your agent in a really strange way, and it's gonna be absolute hell for you. So how do you kind of create agents to kind of ma-manage and maintain that 20% and keep an eye on it, and then adapt and change your evals?
- 13:56
So I think people need to start looking at their evals not as this like static dataset thing, but actually as like code, as like software, or as like a, a living agent.
- 14:04
Um, not as a point in time, but as like a self-optimizing growing solution. Um, that's more or less it. Um, I was gonna present more of like an in-depth demonstration of this where we've applied it at Comet, but like the, the end state is not quite finished yet, and it will be over the next coming weeks.
- 14:21
But I wanted to kind of give you guys something that's like not a sales pitch and something that you could kind of conceptually map to, to the problem space in your own worlds as well.
- 14:29
So if you're working with software, you're working with AI agents, I think you need to start realizing that like the agents will start to shift by themselves. The problem space, the datasets will change, and you also need to kind of treat this problem with an agentic mindset as well.
- 14:43
Thank you. [audience applauding] [upbeat music]