AI Engineer World's Fair 2025
Turning Fails into Features: Zapier’s Hard-Won Eval Lessons
Read the talk
Turning Agent Failures into Better Evals
Zapier’s experience with user-built agents shows how production feedback becomes reproducible tests, why narrow evals can mislead model selection, and where real users must make the final call.
From a talk by Rafal Wilinski and Vitor Balocco
Before you start: Familiarity with LLM tool calls and basic software testing will help; no prior experience building an evaluation system is required.
From a working prototype to a feedback loop
How do you make an agent reliable when nontechnical users decide what it should do? In Zapier Agents, a user describes a business process, the product proposes tools and a trigger, and the user enables the resulting automation. Rafal Wilinski and Vitor Balocco describe two years of learning that building this platform is harder than building an individual agent: unpredictable model behavior meets users whose requests and usage patterns cannot all be anticipated.
The deceptively easy version starts with a LangChain tutorial. Take an example, tweak its prompt, attach a few tools, and chat with it until it seems to work. That produces a prototype, but informal conversations cover only a small part of what happens after deployment.
Shipping starts the evaluation work. Usage produces feedback; feedback reveals failures and unfamiliar use cases; those discoveries become evals and product changes. A better product attracts more users, who expose another set of failures. The feedback loop is therefore part of maintaining the product, not a temporary phase before it becomes reliable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Record runs in a form you can replay
Actionable feedback needs a run to attach to. Tracing completion calls through Braintrust or another tracing system is a useful start, but an agent can fail outside the model call. Record tool calls and their errors, along with preprocessing and postprocessing, so the trace contains the steps needed to reconstruct what went wrong.
Preserve data in the same shape it has at runtime. That makes a trace reusable: its recorded inputs and outputs can prepopulate an evaluation case instead of requiring a separate translation step. It also makes side effects easier to control. A recorded tool response can stand in for a tool invocation during an eval, avoiding another real action merely to reproduce the run. Designing traces for replay lowers the cost of turning a failure into a test.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask when the user can judge the result
Once traces are arriving, the next problem is choosing which runs deserve attention. Explicit feedback is valuable, but ordinary thumbs-up and thumbs-down buttons receive few responses. Timing the request helps: ask when the user has a result to assess, rather than interrupting them beforehand.
Zapier added a feedback call to action after an agent finished, including after a test run: “Did this run do what you expected?” The presenters report that this contextual prompt increased feedback submissions, without quantifying the increase. Both satisfaction and frustration can motivate a response at that moment. Even so, explicit feedback remains scarce, and detailed, actionable explanations are scarcer still.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read behavior as feedback
Product interactions provide additional evidence without asking users to fill out a form. In Zapier, enabling an agent after testing it is treated as strong positive feedback: the user has seen its behavior and decided to put it to work. Copying a response is another positive signal the presenters identify, and they cite ChatGPT as another product using that signal.
Conversations contain negative signals too. Users may explicitly express dissatisfaction, tell the agent to stop slacking, repeat an earlier request with different wording, or curse. Rephrasing is particularly useful to notice: a user who tries the same instruction again may be attempting to repair the model’s interpretation rather than asking for something new.
Zapier began using an LLM to detect and group frustrations into a weekly Slack report. Making that useful required substantial iteration on what frustration meant in this particular product. A generic sentiment label is less helpful than a classification that identifies how the agent disappointed the user.
Traditional business metrics provide another way to select traces. Choose the outcomes the business cares about, then inspect the interactions associated with them. The presenters suggest looking at customers who churned in the previous seven days and reading their last interactions before leaving. That connects a business outcome to concrete agent behavior worth investigating.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn trace inspection into a roadmap
A montage repeatedly urges the audience to look at their data; the presenters follow it with a joke that meetings will continue until everyone does. The practical requirement behind the joke is a way to see an entire agent run. One run can include multiple model calls, database interactions, tools, and REST requests. Any one can fail and trigger failures downstream.
Zapier both buys and builds tooling for this inspection. General tracing software captures the execution, while internal tools can present it in the product’s own domain context. The presenters point to Cursor and Claude Code as ways to make such tools easier to build. The critical feature is a low-friction path from an interesting run to an eval—ideally one click—so preserving a discovered failure becomes an instinct.
After individual runs make sense, aggregate them. Cluster feedback, group interactions, and bucket failure modes to discover which tools fail most often and which interactions cause the most trouble. These groups supply a practical roadmap: they show where engineering effort can address recurring customer problems.
The team also experimented with reasoning models as diagnostic assistants. Give the model the trace output, input instructions, and available context, then ask it to explain the failure. Even when it does not identify the root cause, a useful explanation of the run—or a pointer to an unusual step—can narrow the human investigation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with the next decision
A shortlist of failure modes gives evaluation a concrete starting point. Zapier organizes evals into a hierarchy resembling the testing pyramid: unit-like evals at the base, end-to-end or trajectory evals in the middle, and A/B tests with staged rollouts at the top. Each level answers a broader question about whether the agent works.
A unit eval tests the transition from the current state to state N+1. It can check whether the agent selects a particular tool, supplies the right parameters, includes a required keyword, or decides that it has finished. These assertions make a specific observed failure easy to reproduce. A TypeScript assertion for a tool decision can stay small:
typescript
type ToolCall = {
name: string;
arguments: Record<string, unknown>;
};
function assertToolCall(actual: ToolCall, expected: ToolCall): void {
if (actual.name !== expected.name) {
throw new Error(`Expected ${expected.name}, got ${actual.name}`);
}
for (const [key, value] of Object.entries(expected.arguments)) {
if (!Object.is(actual.arguments[key], value)) {
throw new Error(`Unexpected argument: ${key}`);
}
}
}
assertToolCall(
{ name: "lookup_contact", arguments: { email: "alex@example.com" } },
{ name: "lookup_contact", arguments: { email: "alex@example.com" } },
);
Here the check constrains the tool name and specified scalar arguments; it does not assess the rest of the agent’s work.
The presenters recommend starting with these tests because they build a useful habit: inspect data, identify a problem, reproduce it, and fix it. But a positive user response does not automatically deserve its own unit eval. These tests are most useful for improving specific failure modes, rather than preserving every detail of every successful run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When a better model takes a different path
The limitations became visible when models the presenters considered stronger performed worse on Zapier’s internal benchmarks. Much of the suite tested fine-grained decisions. Individual traces were understandable, but aggregating many passes and regressions made it difficult to tell whether a new model was worse or simply behaved differently.
To investigate, the team used Braintrust MCP with a reasoning model to compare experiment runs and explain what changed. The linked documentation describes the current integration; the talk demonstrates the workflow without specifying its historical tool configuration. In the comparison of Gemini Pro and Claude, the reasoning model characterized Claude as a decisive executor, while Gemini was more verbose, asked follow-up questions, sought affirmation, and sometimes hallucinated JSON structures. Those observations concern the team’s experiment; exact model versions and task distribution are not supplied.
The deeper issue was overfitting the eval to the existing model’s path. Data collected from that model encoded how it typically reached a goal. A test that demands the same next step can penalize another model for choosing a different valid route. Such a test may still be valuable for diagnosing a particular decision, but it cannot by itself establish which model completes the overall task better.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Run the whole trajectory without repeating customer actions
A trajectory eval lets the agent continue to its end state. Evaluation then covers the final result, the tool calls along the way, and the artifacts those calls generated. This allows the test to assess a complete attempt rather than a single iteration, and it can be paired with an LLM judge.
The setup is harder because realistic execution includes side effects. Replaying a customer’s run must not send their email again. For trajectory testing, Zapier chose to mirror user environments with synthetic copies rather than mock the environment, seeking results that better reflected real behavior. This is a different choice from substituting recorded tool outputs in a narrow replay: the broader test needs an environment in which the agent can take multiple steps. The talk does not specify the copies’ isolation controls.
The presenters report that trajectory evals can sometimes take up to an hour; they do not specify the workload or timing protocol. They consider the approach valuable despite the setup effort and slower feedback cycle.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tell the judge what matters in each case
An LLM judge can grade an eval result or compare results, but its judgments also need validation. Small, easily overlooked details can introduce bias. Adding a judge therefore creates another component whose behavior must be checked, rather than eliminating the need to define correctness.
Zapier experimented with human-authored, natural-language rubrics attached to individual dataset rows. Each case tells the judge what specifically deserves attention. One criterion asks whether the agent reacted to an unexpected calendar API error and retried. That evaluates recovery across steps without requiring every run to follow an identical sequence. The accompanying slide places rubric examples involving a calendar error, three emails, and a final-answer mention beside branching LLM and tool paths to a correct end state.
The resulting division of labor separates scoring from execution scope:
| Evaluation approach | Main use |
|---|---|
| LLM judges and case-specific rubrics | Capability overviews and model comparisons |
| Trajectory evals | Criteria spanning multiple turns |
| Unit-like evals | Reproducing and fixing specific failures |
These approaches can work together: a rubric can score a trajectory, while unit evals isolate the decisions responsible for a failure. The caution about overfitting remains strongest when a narrow assertion is used to stand in for overall success.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Protect existing behavior and keep harder problems
A metric can lose its usefulness when it becomes the target. The presenters treat a score approaching 100% as a reason to question whether the dataset still poses interesting problems, rather than assume the agent has become exceptionally capable. A suite that only contains problems already solved offers little guidance about what to improve next.
Zapier was experimenting with two dataset pools:
- Regression cases protect existing customer use cases when prompts, models, or other system behavior change.
- Aspirational cases expose capabilities that remain difficult, such as successfully completing 200 tool calls in a row.
The 200-call sequence is an example of an ambitious test, not a reported achievement. Keeping these pools separate distinguishes preserving working behavior from making progress on unsolved tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let production users verify the change
The purpose of evaluation is user satisfaction, not maximizing a laboratory score. Offline results should inform deployment without overriding users’ qualitative experience. That is why the presenters put production A/B testing at the top of their evaluation hierarchy.
Their illustrative rollout sends 5% of traffic to a new model or prompt. Monitor feedback, activation, and retention to see whether the change improves the experience users actually have. That evidence supports a deployment decision in a way that another round of optimizing an offline score cannot: it tests whether the improvement survives contact with the people the agent is meant to serve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current setup and usage documentation for accessing Braintrust experiments and production logs from AI coding tools.
The framework's official repository, with installation instructions and links to model and tool integrations.
Further reading
- Agentic Evals PyramidArticle
The presenters' companion explanation of unit evaluations, complete trajectories, synthetic test environments and production A/B testing.
Vitor Balocco's TypeScript tutorial combines tool-call assertions and LLM scoring in a weather chatbot evaluation.
Read the complete timestamped transcript
- 0:00
[on-hold music] Um, yeah.
- 0:16
In brief introduction to Zapier Agents. Um, I believe many of you know what Zapier is, this automation software. Lot of boxes, arrows, essentially about automation, uh, automating your business processes.
- 0:28
Um, Agents is just, well, more agentic alternative to Zapier. You describe what you want, we propose you a bunch of tools, a trigger, um, you enable that, and hopefully we enable your-- we automate your whole business processes.
- 0:42
Um, and a key lesson that we have after tho-those two years is that, um, building good AI agents is hard, and building good platform to enable non-technical people to build AI agents is even harder.
- 0:56
That's because AI is non-deterministic, but on top of that, your users are even more non-deterministic. They are going to use your products in a way that you cannot imagine upfront.
- 1:07
So, um, if you think that building agents is not that hard, you probably have this kind of picture in mind. Um, you probably stumbled upon this library called LangChain.
- 1:20
Um, you pulled some examples, tutorial, um, you tweaked the prompt, pulled a bunch of tools, you chatted with the solution, and you thought, "Well, it's actually kinda working all right.
- 1:31
So let's deploy it and let's collect some profit." Um, turns out the reality has a surprising amount of detail, and we believe that building probabilistic software is a little bit different than building traditional software.
- 1:44
Um, the initial prototype is only a start, and after you ship something to your users, your responsibility switches to building the data flywheel. So once you start-- Once your user starts using your product, um, you need to collect the feedback.
- 1:58
You're starting to understand the usage patterns, the failures, so they can-- then you can build more evals, build understanding of what's failing, what are the use cases. Um, as you're building more evals and more features, probably your product is getting better, so you're getting more users, and there are more failures, and you have to build more features
- 2:17
and on and on and on. So yeah, it form-- it forms this data flywheel. But starting with the first step.
- 2:25
Okay, yeah. So starting from the beginning, how do you start collecting actionable feedback?
- 2:31
Backing up for just a second, the first step is to make sure you're instrumenting your code, right? Which you probably already are doing. Whether you're using Braintrust or something else, they all offer like an easy way to get started, like just tracing your completion calls.
- 2:42
And this is a good start, but actually, you also wanna make sure that you're recording much more than that in your traces. You wanna record the tool calls, the errors from those tool calls, the pre- and post-processing steps.
- 2:52
That way, it will be much easier to debug what went wrong with the run.
- 2:56
And you also wanna strive to make the run repeatable for eval purposes. So for instance, if you log data in the same shape as it appears in the runtime, it makes it much easier to convert it to an eval run later because you can just pre-populate the inputs and expected outputs directly from your trace for free.
- 3:11
And this is especially useful as well for tool calls because if your tool call produces any side effects, you probably wanna mock those in your evals. So you get all that for free if you're recording them in your trace.
- 3:22
Okay, great. So you've instrumented your code and you started getting, uh, all this raw data from your runs. Now it's time to figure out what runs to actually pay attention to.
- 3:32
Um, explicit user feedboke is-- feedback is really high signal, so that's a good, good place to start. Unfortunately, not many people actually click those classic thumbs up, thumbs up and thumbs down buttons.
- 3:43
So you gotta work a bit harder for that feedback. And in our experience, this works best when you ask for the feedback in the right context. So you can be a little bit more aggressive about asking for the feedback, but you're in the right context.
- 3:53
You're not bothering the user before that. So for us, one example of this is once an agent finish running, even if it was just a test run, we show a feedback call to action at the bottom, right?
- 4:02
"Did this run do what you expected? Give us the feedback now." Um, and this small change actually gave us like a really nice bump in feedback submissions, surprisingly. So thumbs up and thumbs down are a good benchmark, a good base-baseline, but try to find these critical moments in your user's journey where they'll be most likely to provide
- 4:19
you that feedback, either because they're happy and satisfied or because they're angry and they wanna tell you about it. Um, even if you work really hard for the feedback, explicit feedback is still really rare.
- 4:30
Uh, and explicit feedback that's detailed and actionable is even harder because people are just not that interested in providing feedback generally.
- 4:38
So you also wanna mine user interaction for implicit feedback. And the good news is there's actually a lot of low-hanging fruit possibilities here. Here's an example from our, from our app.
- 4:47
Users can test an agent before they turn it on to see if everything's going okay. So if they do turn it on, that's actually really strong positive implicit feedback, right?
- 4:57
Copying a model's response is also good implicit feedback. Even, uh, OpenAI is doing this for ChatGPT.
- 5:06
And you can also look for implicit signals in the conversation. So here the user is clearly letting us know that they're not happy with the results.
- 5:14
Here they're telling the agent to stop slacking around, which is clearly implicit negative feedback, I think.
- 5:21
Sometimes the user sends a follow-up message that is mostly rehashing what they asked the previous time to see if the LLM interprets that phrasing better. That's also, also good implicit negative feedback.
- 5:31
And there's also a surprising amount of [chuckles] of cursing. [audience laughing]
- 5:36
Uh, recently we had a lot of success as well using an LLM to detect and group frustrations, and we have this weekly report that we post in our Slack.
- 5:42
But it took us a lot of tinkering to make sure that the LLM understood what frustration means in the context of our products. So I encourage you to try it out, but expect a lot of tinkering.
- 5:52
You should also not forget to look at more traditional user metrics, right? Uh, there's a lot of stuff in there for you to mine implicit signal too. So find what metrics your business cares about and figure out how to track them.
- 6:02
Then you can distill some signal from that data. You can look for customers, for example, that churned, churned in the last seven days, and go look at their last interaction with your product before they left, and you're likely to find some signal there.
- 6:13
Okay, so I have raw data, but now I'll let the indus- I'll let the industry experts speak.
- 6:22
Why isn't it starting? Oh.
- 6:25
Look at, look at, look at, look at, look at your data. Look at the data. Look at, look at,
- 6:33
look. Look at the data. Look at the data. Look at the data. Look at this data. Always look at your data. Always look at your data.
- 6:40
Yeah, or the meetings will continue until everyone looks at their data. [laughing]
- 6:45
Um, okay, but how actually you're going to do that? So, um, we believe that the first step is to either buy or build LLMOps software. We do both. Um, you're definitely going to need that to understand your agent runs because one agent run is probably multiple LM calls, multiple database interactions, tool calls, REST calls, whatever.
- 7:05
Each one of them can be source of failure, and it's really important to piece together this whole story, understand this, you know, what caused this cascading failure. Um, yeah, I said we are doing both because I believe VIP coding your own internal tooling is really, really easy right now with Cursor and Cloud Code, and it's going to
- 7:23
pay you massive dividends in the future, um, for two reasons. First of all, it gives you an ability to understand your data in your own specific domain context. Uh, and the second of all, it also...
- 7:35
You should be also able to create a functionality to turn every single interacting case or every failure into an eval with the minimal amount of, um, friction. So whenever you see something interesting, there should be like a one click to turn it into an eval.
- 7:51
It should become your instinct. Once you understand what's going on on a singular run basis, you can start understanding things at scale. So now we can do feedback aggregations, clustering.
- 8:02
You can bucket your, uh, your, your failure modes, you can bucket your interactions, and then you're going to starting to see what kind of tools are failing the most, what kind of interactions are the most problematic.
- 8:13
That's going to almost create for you like a automatic roadmap, so you'll know where to apply your time and effort to improve your product the most. Um, doing anything else is going to be a suboptimal strategy.
- 8:25
Something that we are also experimenting with is using reasoning models to explain the failures. Turns out that if you give them the trace output, input instructions and anything you can find, they are pretty good at finding the root cause of a failure.
- 8:40
Um, even if they are not going to do that, they are probably going to explain you the whole run or just direct your attention into something that's really interesting and might help you find the root cause of the problem.
- 8:53
Cool. So now you have a good shortlist of failure modes you wanna work on first. It's time to start building out your evals.
- 9:00
And we realized over time that there are different types of evals, and the types of evals that we wanna build can be placed into this hierarchy that resembles the testing pyramid, for those of you that know that.
- 9:10
Um, so with unit tests like evals at the base, end-to-end evals or trajectory evals, how we like to call them, in the middle, and the ultimate way of evaluating using A/B testing with stage rolls at, uh, rollouts at the top.
- 9:21
So let's talk a bit about those. Starting with unit test evals, we're just trying to predict the N plus one state from the current state, so these work great when you wanna do simple assertions, right?
- 9:31
For instance, you could check whether the next state is a specific tool call, or if the tool call parameters are correct, or if the answer contains a specific keyword, or if the agent determined that it was done.
- 9:41
All that good stuff. So if you're starting out, we recommend focusing on unit test evals first because these are the easiest to add. It helps you build that muscle of looking at your data, spotting problems, creating evals that reproduce them, and then just focusing on fixing them, right?
- 9:55
Beware though of like turning every positive feedback into an eval. We found that unit test evals are best for hill climbing specific failure modes that you spot in your data.
- 10:04
So now unit tests evals are not perfect, and we realized that ourselves. Uh, we realized we had overindexed on unit test evals when the new models were coming out that were objectively stronger models.
- 10:15
But we were-- they were still performing worse in our internal benchmarks, which was weird. Um, and because the majority of our evals were so fine-grained, this made it really hard to see the forest for the trees when benchmarking new mo- new models.
- 10:27
There was always a lot of noise when we would try comparing runs. Like when you're looking at a single trace, it's, it's easy to kind of go through the trace and understand what's happening.
- 10:34
But when you need to kind of look at it from... I don't know how to put it again. Sorry. Uh, when you wanna look at it, uh, through an aggregation of many traces, then it starts getting difficult to understand what's happening.
- 10:44
Why are so many of these, uh, passing and some of these are regressing? Yeah. So we realized that, uh, maybe machine can help us. It turns out in that previe- previous video when I was investigating, uh, one experiment inside Braintrust, there is a lot of looking at that screen trying to figure out what went wrong.
- 11:00
And we were like, "Hey, maybe we can like just give this whole data to, once again, a reasoning LLM and compare the models for us." It turns out that with Braintrust MCP and reasoning model, you can just ask it to, "Hey, look at this run, look at this run, and tell me what's actually different about the new
- 11:18
model that we are going to deploy." In this case, it was Gemini Pro versus Claude. And what the reasoning model found was actually really, really good. It found that Claude is like a deci- decisive executor, whereas Gemini is really yapping a lot.
- 11:31
It's asking follow-up questions. It needs some positive affirmations, and it's sometimes even hallucinating, uh, about JSON structures. So yeah, it helped us a lot. It also surfaces a problem with unit test evals a lot, which is, um, different models have different ways of trying to achieve the same goal.
- 11:50
And unit test evals are penalizing different paths. They are like hard-coded to only follow, follow one path. And yeah, our unit test evals were overfitting to our existing models or actually data collecting using that model.
- 12:03
So what we started experimenting with is trajectory evals. Um- Yeah. Instead of grading just one iteration of an agent, we let the agent run till the whole, uh, to the, to the end state.
- 12:17
And we are not grading just the end state, but we are also grading all the tool calls that were made along the way and, uh, all the artifacts that have been generated along the way.
- 12:26
Um, this can be also paired with LLM-as-a-judge. Vitor is going to speak about that later.
- 12:32
Um, yeah, but they are not free. I think they have really high return on investment, but they are much harder to set up, um, especially if you are evaluating runs that have tools that cause side effects, right?
- 12:46
When you are running an eval, you definitely don't wanna send a email on behalf of the customer once again, right? Um, so we had a fundamental question whether we should mock environment or not, and we decided that we are not going mock, we are not going to mock the environment because otherwise you're going to get data that
- 13:03
is just not reflecting the reality. So what we started doing is just mirroring, uh, user's environment and crafting a synthesis, uh, synthetic copy of that. Uh, also, they are much slower, right?
- 13:15
So they can sometimes take, like, up to an hour, um, so it's not pretty great.
- 13:21
And we're also learning a bit more into LLM as a judge. Uh, this is when you're using LLM to grade or compare results from your evals. And it's tempting to lean into them for everything, but you need to make sure that the judge is judging things correctly, which can be surprisingly hard.
- 13:36
Uh, and you also have to be careful not to introduce subtle biases, right? Because even small things that you might overlook might end up influencing it.
- 13:44
Lately, we've also been experimenting with this concept of rubrics-based scoring. We use an LLM to judge a run, but each row in our dataset has a different set of rubiks, rubrics that were handcrafted by a human and described in natural language what specifically about this run should the LLM be paying attention to for the score.
- 14:02
Uh, so one example of this, did the agent react to an unexpected error from the calendar API and then try it again?
- 14:10
So to sum it up, here's our current mental model of the types of evals that we build for Zapier Agents. We use LLM-as-a-judge or rubrics-based evals to build a high-level overview of your system's capabilities, and these are great for benchmarking new models.
- 14:23
We use trajectory evals to capture multi-turn criteria, and we use unit test-like evals to debug specific failures, uh, hill climb them. Uh, but beware of overfitting with these.
- 14:34
Yeah. And a couple of closing thoughts. Don't obsess over metrics. Uh, remember that when a good metrics become a target, it ceases to be a good target. Um, so when you're close to achieving 100% score on your eval dataset, it's not meaning that you're doing good job.
- 14:49
It actually meaning that your dataset is just not interesting, right? Because we don't have AGI yet, so it's probably not true that your model is that good. Um, something that we're experimenting with lately is dividing the dataset into two pools.
- 15:04
Uh, into the regressions dataset to make sure that if we are making any changes, we are not breaking existing use cases for the customers, and also the aspirational dataset of things that are extremely hard.
- 15:14
For instance, like nailing 200 tool calls, uh, in a row.
- 15:20
And lastly, um, let's take a step back. What's the point of creating evals in the first place? Um, your goal isn't to maximize some imaginary number in a lab-like setting.
- 15:30
Uh, your end goal is user satisfaction, so the ultimate judge, um, are your users. You shouldn't be optimizing for the biggest scores for the evals and completely disregard the vibes.
- 15:40
So that's why we think the ultimate verification method is an AB test. Just take a small proportion of your, portion of your traffic, let's say 5%, and route it to the new model, route it to the new prompt, monitor the feedback, check your metrics like activation, user retention, and so on.
- 15:56
Based on that, probably you can make the most educated guess, uh, instead of being in the lab and optimizing this imaginary number.
- 16:06
That's all. Thank you.
- 16:08
Thank you. [audience applauding] [upbeat music]