# Turning Fails into Features: Zapier’s Hard-Won Eval Lessons

Rafal Wilinski, Zapier · Vitor Balocco

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/blrovBxxN9o-turning-fails-into-features-zapiers-hard-won) · [Video](https://www.youtube.com/watch?v=blrovBxxN9o)

## Turning Fails into Features: Zapier’s Hard-Won Eval Lessons

Zapier’s experience with user-built agents shows how production feedback becomes reproducible tests, why narrow evals can mislead model selection, and where real users must make the final call.

Familiarity with LLM tool calls and basic software testing will help; no prior experience building an evaluation system is required.

### From a working prototype to a feedback loop

[0:16](https://www.youtube.com/watch?v=blrovBxxN9o&t=16s)

How do you make an agent reliable when nontechnical users decide what it should do? In Zapier Agents, a user describes a business process, the product proposes tools and a trigger, and the user enables the resulting automation. Rafal Wilinski and Vitor Balocco describe two years of learning that building this platform is harder than building an individual agent: unpredictable model behavior meets users whose requests and usage patterns cannot all be anticipated.

The deceptively easy version starts with a [LangChain](https://github.com/langchain-ai/langchain) tutorial. Take an example, tweak its prompt, attach a few tools, and chat with it until it seems to work. That produces a prototype, but informal conversations cover only a small part of what happens after deployment.

**Shipping starts the evaluation work.** Usage produces feedback; feedback reveals failures and unfamiliar use cases; those discoveries become evals and product changes. A better product attracts more users, who expose another set of failures. The feedback loop is therefore part of maintaining the product, not a temporary phase before it becomes reliable.

![Flowchart from prototype and initial evals to shipping, collecting feedback, understanding data, and more evals, with an iteration loop.](https://ai.engineer/api/talk-essay-media/9f7009eb4d4eaffe6e00652d16f1dae99e4169bdc625362705986a96add97f0b)

Building good agents requires a feedback and evaluation loop after shipping.

### Record runs in a form you can replay

[2:25](https://www.youtube.com/watch?v=blrovBxxN9o&t=145s)

Actionable feedback needs a run to attach to. Tracing completion calls through Braintrust or another tracing system is a useful start, but an agent can fail outside the model call. Record tool calls and their errors, along with preprocessing and postprocessing, so the trace contains the steps needed to reconstruct what went wrong.

Preserve data in the same shape it has at runtime. That makes a trace reusable: its recorded inputs and outputs can prepopulate an evaluation case instead of requiring a separate translation step. It also makes side effects easier to control. A recorded tool response can stand in for a tool invocation during an eval, avoiding another real action merely to reproduce the run. **Designing traces for replay lowers the cost of turning a failure into a test.**

![Slide titled “Step 1: Instrument your code (properly)” with a code example and three bullets about spans and tags, recording runtime-shaped data, and reproducibility.](https://ai.engineer/api/talk-essay-media/c83dfa524959c8d4863118552c8b85886d040332e3ea2bd411e48d1022f97d96)

Instrument code with named spans, runtime-shaped data, and reproducible runs.

### Ask when the user can judge the result

[3:22](https://www.youtube.com/watch?v=blrovBxxN9o&t=202s)

Once traces are arriving, the next problem is choosing which runs deserve attention. Explicit feedback is valuable, but ordinary thumbs-up and thumbs-down buttons receive few responses. Timing the request helps: ask when the user has a result to assess, rather than interrupting them beforehand.

Zapier added a feedback call to action after an agent finished, including after a test run: “Did this run do what you expected?” The presenters report that this contextual prompt increased feedback submissions, without quantifying the increase. Both satisfaction and frustration can motivate a response at that moment. Even so, explicit feedback remains scarce, and detailed, actionable explanations are scarcer still.

### Read behavior as feedback

[4:38](https://www.youtube.com/watch?v=blrovBxxN9o&t=278s)

Product interactions provide additional evidence without asking users to fill out a form. In Zapier, enabling an agent after testing it is treated as strong positive feedback: the user has seen its behavior and decided to put it to work. Copying a response is another positive signal the presenters identify, and they cite ChatGPT as another product using that signal.

Conversations contain negative signals too. Users may explicitly express dissatisfaction, tell the agent to stop slacking, repeat an earlier request with different wording, or curse. Rephrasing is particularly useful to notice: a user who tries the same instruction again may be attempting to repair the model’s interpretation rather than asking for something new.

Zapier began using an LLM to detect and group frustrations into a weekly Slack report. Making that useful required substantial iteration on what frustration meant in this particular product. A generic sentiment label is less helpful than a classification that identifies how the agent disappointed the user.

Traditional business metrics provide another way to select traces. Choose the outcomes the business cares about, then inspect the interactions associated with them. The presenters suggest looking at customers who churned in the previous seven days and reading their last interactions before leaving. That connects a business outcome to concrete agent behavior worth investigating.

### Turn trace inspection into a roadmap

[6:25](https://www.youtube.com/watch?v=blrovBxxN9o&t=385s)

A montage repeatedly urges the audience to look at their data; the presenters follow it with a joke that meetings will continue until everyone does. The practical requirement behind the joke is a way to see an entire agent run. One run can include multiple model calls, database interactions, tools, and REST requests. Any one can fail and trigger failures downstream.

Zapier both buys and builds tooling for this inspection. General tracing software captures the execution, while internal tools can present it in the product’s own domain context. The presenters point to Cursor and Claude Code as ways to make such tools easier to build. The critical feature is a low-friction path from an interesting run to an eval—ideally one click—so preserving a discovered failure becomes an instinct.

After individual runs make sense, aggregate them. Cluster feedback, group interactions, and bucket failure modes to discover which tools fail most often and which interactions cause the most trouble. These groups supply a practical roadmap: they show where engineering effort can address recurring customer problems.

![Slide titled “Do feedback aggregations / clustering” beside a dashboard showing a purple feedback-by-category bar chart; a presenter inset covers part of the lower bullet.](https://ai.engineer/api/talk-essay-media/28e70b075fa9258e821bd01c1d2f829126fd42ea4659ad2f2ccda90a11bcfc40)

Group feedback into failure categories to prioritize improvements.

The team also experimented with reasoning models as diagnostic assistants. Give the model the trace output, input instructions, and available context, then ask it to explain the failure. Even when it does not identify the root cause, a useful explanation of the run—or a pointer to an unusual step—can narrow the human investigation.

### Start with the next decision

[8:53](https://www.youtube.com/watch?v=blrovBxxN9o&t=533s)

A shortlist of failure modes gives evaluation a concrete starting point. Zapier organizes evals into a hierarchy resembling the testing pyramid: unit-like evals at the base, end-to-end or trajectory evals in the middle, and A/B tests with staged rollouts at the top. Each level answers a broader question about whether the agent works.

A **unit eval** tests the transition from the current state to state N+1. It can check whether the agent selects a particular tool, supplies the right parameters, includes a required keyword, or decides that it has finished. These assertions make a specific observed failure easy to reproduce. A TypeScript assertion for a tool decision can stay small:

```typescript
type ToolCall = {
  name: string;
  arguments: Record<string, unknown>;
};

function assertToolCall(actual: ToolCall, expected: ToolCall): void {
  if (actual.name !== expected.name) {
    throw new Error(`Expected ${expected.name}, got ${actual.name}`);
  }

  for (const [key, value] of Object.entries(expected.arguments)) {
    if (!Object.is(actual.arguments[key], value)) {
      throw new Error(`Unexpected argument: ${key}`);
    }
  }
}

assertToolCall(
  { name: "lookup_contact", arguments: { email: "alex@example.com" } },
  { name: "lookup_contact", arguments: { email: "alex@example.com" } },
);
```

Here the check constrains the tool name and specified scalar arguments; it does not assess the rest of the agent’s work.

The presenters recommend starting with these tests because they build a useful habit: inspect data, identify a problem, reproduce it, and fix it. But a positive user response does not automatically deserve its own unit eval. These tests are most useful for improving specific failure modes, rather than preserving every detail of every successful run.

### When a better model takes a different path

[10:04](https://www.youtube.com/watch?v=blrovBxxN9o&t=604s)

The limitations became visible when models the presenters considered stronger performed worse on Zapier’s internal benchmarks. Much of the suite tested fine-grained decisions. Individual traces were understandable, but aggregating many passes and regressions made it difficult to tell whether a new model was worse or simply behaved differently.

To investigate, the team used [Braintrust MCP](https://www.braintrust.dev/docs/integrations/developer-tools/mcp) with a reasoning model to compare experiment runs and explain what changed. The linked documentation describes the current integration; the talk demonstrates the workflow without specifying its historical tool configuration. In the comparison of **Gemini Pro** and **Claude**, the reasoning model characterized Claude as a decisive executor, while Gemini was more verbose, asked follow-up questions, sought affirmation, and sometimes hallucinated JSON structures. Those observations concern the team’s experiment; exact model versions and task distribution are not supplied.

The deeper issue was **overfitting the eval to the existing model’s path**. Data collected from that model encoded how it typically reached a goal. A test that demands the same next step can penalize another model for choosing a different valid route. Such a test may still be valuable for diagnosing a particular decision, but it cannot by itself establish which model completes the overall task better.

### Run the whole trajectory without repeating customer actions

[12:03](https://www.youtube.com/watch?v=blrovBxxN9o&t=723s)

A **trajectory eval** lets the agent continue to its end state. Evaluation then covers the final result, the tool calls along the way, and the artifacts those calls generated. This allows the test to assess a complete attempt rather than a single iteration, and it can be paired with an LLM judge.

The setup is harder because realistic execution includes side effects. Replaying a customer’s run must not send their email again. For trajectory testing, Zapier chose to mirror user environments with synthetic copies rather than mock the environment, seeking results that better reflected real behavior. This is a different choice from substituting recorded tool outputs in a narrow replay: the broader test needs an environment in which the agent can take multiple steps. The talk does not specify the copies’ isolation controls.

The presenters report that trajectory evals can sometimes take up to an hour; they do not specify the workload or timing protocol. They consider the approach valuable despite the setup effort and slower feedback cycle.

### Tell the judge what matters in each case

[13:21](https://www.youtube.com/watch?v=blrovBxxN9o&t=801s)

An **LLM judge** can grade an eval result or compare results, but its judgments also need validation. Small, easily overlooked details can introduce bias. Adding a judge therefore creates another component whose behavior must be checked, rather than eliminating the need to define correctness.

Zapier experimented with human-authored, natural-language rubrics attached to individual dataset rows. Each case tells the judge what specifically deserves attention. One criterion asks whether the agent reacted to an unexpected calendar API error and retried. That evaluates recovery across steps without requiring every run to follow an identical sequence. The accompanying slide places rubric examples involving a calendar error, three emails, and a final-answer mention beside branching LLM and tool paths to a correct end state.

![Rubrics-based scoring slide lists examples involving a calendar error, three emails, and a final-answer mention, alongside branching LLM and tool calls leading to a correct end state.](https://ai.engineer/api/talk-essay-media/d55e4a7aa1a6bb888506ece5b082dbc55782afca3deb1bd84e2dd6624d7f601f)

Rubric examples accompany alternative agent paths to a correct end state.

The resulting division of labor separates scoring from execution scope:

| Evaluation approach | Main use |
| --- | --- |
| LLM judges and case-specific rubrics | Capability overviews and model comparisons |
| Trajectory evals | Criteria spanning multiple turns |
| Unit-like evals | Reproducing and fixing specific failures |

These approaches can work together: a rubric can score a trajectory, while unit evals isolate the decisions responsible for a failure. The caution about overfitting remains strongest when a narrow assertion is used to stand in for overall success.

### Protect existing behavior and keep harder problems

[14:34](https://www.youtube.com/watch?v=blrovBxxN9o&t=874s)

A metric can lose its usefulness when it becomes the target. The presenters treat a score approaching 100% as a reason to question whether the dataset still poses interesting problems, rather than assume the agent has become exceptionally capable. A suite that only contains problems already solved offers little guidance about what to improve next.

Zapier was experimenting with two dataset pools:

- **Regression cases** protect existing customer use cases when prompts, models, or other system behavior change.
- **Aspirational cases** expose capabilities that remain difficult, such as successfully completing 200 tool calls in a row.

The 200-call sequence is an example of an ambitious test, not a reported achievement. Keeping these pools separate distinguishes preserving working behavior from making progress on unsolved tasks.

### Let production users verify the change

[15:20](https://www.youtube.com/watch?v=blrovBxxN9o&t=920s)

The purpose of evaluation is user satisfaction, not maximizing a laboratory score. Offline results should inform deployment without overriding users’ qualitative experience. That is why the presenters put production A/B testing at the top of their evaluation hierarchy.

Their illustrative rollout sends 5% of traffic to a new model or prompt. Monitor feedback, activation, and retention to see whether the change improves the experience users actually have. That evidence supports a deployment decision in a way that another round of optimizing an offline score cannot: it tests whether the improvement survives contact with the people the agent is meant to serve.

## Resources

- [Agentic Evals Pyramid](https://rwilinski.ai/posts/evals-pyramid/) — The presenters' companion explanation of unit evaluations, complete trajectories, synthetic test environments and production A/B testing.
- [How Zapier uses assertions to evaluate tool usage in chatbots](https://www.braintrust.dev/docs/cookbook/recipes/Assertions) — Vitor Balocco's TypeScript tutorial combines tool-call assertions and LLM scoring in a weather chatbot evaluation.
- [Braintrust MCP](https://www.braintrust.dev/docs/integrations/developer-tools/mcp) — Current setup and usage documentation for accessing Braintrust experiments and production logs from AI coding tools.
- [LangChain](https://github.com/langchain-ai/langchain) — The framework's official repository, with installation instructions and links to model and tool integrations.

## Slides

### Scheduled trigger for an overdue task reminder

The demo shows a weekly overdue-task reminder with the pointer over the weekly trigger. The speaker explains that users describe what they want, and Zapier Agents proposes tools and a trigger.

[0:25](https://www.youtube.com/watch?v=blrovBxxN9o&t=25s) [A scheduled-trigger menu lists daily, hourly, monthly, and weekly options for a task reminder.](https://assets.ai.engineer/sites/aiecode2025/sha256/337411e9c0be5d6e5f3040ce9ed6a907cba15f7abea1d18d202d130824d33dc2.webp)

Weekly Overdue Task Reminder

Trigger
Instructions:
Look at [illegible] Project M[illegible]
and [illegible] Slack: Send [illegible]
who they're assigned t[illegible]
message.

Scheduled
Search thousands of triggers
Every Day
Triggers every day at the time selected.
Every Hour
Triggers every hour.
Every Month
Triggers every month, on the day(s) selected.
Every Week
Triggers every week, on the day(s) selected.

Interface text:
Add from Scheduled
Behavior preview
Test this
Test your behavior and chat [illegible]
running
Test
AIE
Microsoft
smol ai

### Building good AI agents is hard

Building a platform for non-technical people to create agents is harder still: AI is non-deterministic, and users may use the product in ways its builders did not anticipate.

[0:49](https://www.youtube.com/watch?v=blrovBxxN9o&t=49s) [A red slide states that building good AI agents is hard.](https://assets.ai.engineer/sites/aiecode2025/sha256/6767706d054974e2b505330d2aa5ac915c800c845a470b49db305a45328309a8.webp)

Building good AI agents is hard.

AIE
Microsoft
smol ai

### How not to build good agents

The slide sketches a tempting shortcut: prototype, impress with a flashy demo, ship, and expect profit. The speaker warns that building probabilistic software has more detail than this path suggests.

[1:11](https://www.youtube.com/watch?v=blrovBxxN9o&t=71s) [A flowchart progresses from prototype to flashy demo, shipping to users, and profit.](https://assets.ai.engineer/sites/aiecode2025/sha256/272b378034c4f6ef185757c4a44aa6591b517bef164f70b57111312952eee494.webp)

How not to build good agents

Prototype
↓
Vibes-based flashy demo
↓
Ship to users
↓
Profit 💰

Interface text:
AIE
Microsoft
smol ai

### How to build good agents

After shipping, collect feedback to understand usage patterns and failures, then add evals and iterate. The speakers call this a data flywheel.

[1:34](https://www.youtube.com/watch?v=blrovBxxN9o&t=94s) [A workflow from prototyping and initial evaluations to shipping, collecting feedback, understanding data, and adding evaluations in an iterative cycle.](https://assets.ai.engineer/sites/aiecode2025/sha256/16f5838c8dea5401f30d05d9976d63fd689e646f622dc33a27a9e050865c72da.webp)

How to build good agents

Prototype
Vibes-based / static list evals
Ship to users
Collect feedback
Understand the data
More evals
Iterate

AIE
Microsoft
smol ai

### How to collect feedback?

The speakers turn to how to collect actionable feedback.

[2:23](https://www.youtube.com/watch?v=blrovBxxN9o&t=143s) [Section slide asking how to collect feedback.](https://assets.ai.engineer/sites/aiecode2025/sha256/b71b136c40c0a835ebd8709dcef879e37147639124523df6a6a46ca0dda58824.webp)

How to
collect
feedback?

AIE
Microsoft
smol ai

### Step 1: Instrument your code

The example initializes Braintrust logging and wraps an OpenAI model to trace a completion call, a starting point for collecting actionable feedback.

[2:30](https://www.youtube.com/watch?v=blrovBxxN9o&t=150s) [A code example initializes logging and wraps a model call for tracing.](https://assets.ai.engineer/sites/aiecode2025/sha256/34ea2782806bdb3f3add95cdffd97734d6f58fcbdbdba60180b2ef67b1b2993a.webp)

Step 1: Instrument your code

import { initLogger, wrapAISDKModel } from "braintrust";
import { openai } from "@ai-sdk/openai";

initLogger({
  projectName: "My Project",
  apiKey: process.env.BRAINTRUST_API_KEY,
});

const model = wrapAISDKModel(openai.model("gpt-4o"));

async function main() {
  const resp = await model.generateText("What is the capital of France?");
  console.log(resp);
}

main();

Interface text:
AIE
Microsoft
smol ai

### Start gathering feedback

Zapier asks for feedback when an agent run finishes, including test runs. The prompt appears below the result, giving users a chance to say whether the run did what they expected.

[3:21](https://www.youtube.com/watch?v=blrovBxxN9o&t=201s) [Guidance to request feedback when a workflow finishes, alongside an example with thumbs-up and thumbs-down controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/98e66be762ba4e00cd0edd928d08bc8b83edb2a58566fbd35684a871b8c0c504.webp)

Start gathering feedback

• Important to do it in the right context
• For us, this means showing a feedback CTA when we detect a workflow run is finished

Action Complete
Slack: Send Channel Message

All done! The Jira issue has been created and a confirmation has been posted in the Slack thread with the link to the issue. If you need anything else, just let me know!

Did this run do what you expected?
Provide feedback to help us improve Agents

Interface text:
AIE
Microsoft
smol ai

[3:26](https://www.youtube.com/watch?v=blrovBxxN9o&t=206s) [A completed workflow displays a thumbs-up or thumbs-down feedback prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/4597dc0f283264ab615ba5e7873eb4a823c71ce9448884c75a7fbf8df48fd81f.webp)

Start gathering feedback

• Important to do it in the right context
• For us, this means showing a feedback CTA when we detect a workflow run is finished

Action Complete
Slack: Send Channel Message

All done! The Jira issue has been created and a confirmation has been posted in the Slack thread with the link to the issue. If you need anything else, just let me know!

Did this run do what you expected?
Provide feedback to help us improve Agents

Interface text:
AIE
Microsoft
smol ai

### Feedback submissions increased

A feedback prompt shown when an agent run finishes led to 4x as many submissions. The daily charts show the increase after the new feedback UI.

[4:04](https://www.youtube.com/watch?v=blrovBxxN9o&t=244s) [Charts show feedback submissions before and after a new feedback interface; the slide states submissions increased fourfold.](https://assets.ai.engineer/sites/aiecode2025/sha256/d4f78c58be9172b3b1b4305ea593ce8e2a5a6044e212d059b46f5b77ced5c545.webp)

Start gathering feedback

• 4x number of submissions

Feedback submissions by day
new feedback UI
Y-axis ticks: 0, 50, 100, 150
Timeline: Apr 01, 2024; May 01, 2024; Jun 01, 2024; Jul 01, 2024
X-axis: date

Feedback count by score by day
Y-axis: count 👍 / 👎
Y-axis ticks: 0, 50, 100, 150
Timeline: Apr 01, 2024; May 01, 2024; Jun 01, 2024; Jul 01, 2024

AIE
Microsoft
smol ai

### Explicit feedback is rare

Even when teams ask for feedback, detailed, actionable responses are hard to get. Mine user interactions for implicit signals too.

[4:23](https://www.youtube.com/watch?v=blrovBxxN9o&t=263s) [The slide says detailed explicit feedback is rare and recommends looking for implicit signals.](https://assets.ai.engineer/sites/aiecode2025/sha256/27907a000d4f401385e77969e41fb3ea7c49b7922df09cfdf987493f5f2e076c.webp)

Explicit feedback is rare.
Detailed explicit feedback is ultra rare.

You need to look for implicit signals.

AIE
aws

### Enabling a workflow

Users can test an agent before turning it on. The speaker treats enabling it afterward as a strong positive signal from their interaction.

[4:37](https://www.youtube.com/watch?v=blrovBxxN9o&t=277s) [A workflow interface highlights its enable control as a source of implicit feedback.](https://assets.ai.engineer/sites/aiecode2025/sha256/f634856401309be67f2b15a72ec1a67550697b3af834fd35a89637c97d3ae824.webp)

Look for implicit signals in the UI

Enabling a workflow

Implicit feedback!

Interface text:
Off
Enable behavior
Behavior preview
Retest behavior
[illegible] you do.” - Steve Jobs
Does this look good to you? If so, I will send the message. If you need any changes, please let me know!
Vitor Balocco
all good
Personal Assistant
Action Complete
Slack: Send Direct Message
[Partially obscured workflow text: illegible]
AIE
aws

### Copying a response

Copying a model’s response can serve as positive implicit feedback. The slide shows a ChatGPT copy event alongside a post suggesting interaction signals can help when users don’t rate responses.

[4:55](https://www.youtube.com/watch?v=blrovBxxN9o&t=295s) [A social media post and browser screenshot illustrate copying a response as an interaction signal.](https://assets.ai.engineer/sites/aiecode2025/sha256/1b5166950abd0168ddfbeff57bad30dc9cb6cec40219dfb18b47f06585aac3a2.webp)

Look for implicit signals in the UI

Copying a response

Delip Rao e/σ 🇮🇳
@deliprao

Because people don’t thumbs up/down and we need stuff like this to create rewards. This is not new. Search engines/recommenders collect plenty of such interaction signals like ... forever.

kais @kais_rad · May 17
why tf does OpenAI keep track of what you copy from ChatGPT? 💀

Interface text:
Subscribe
ChatGPT
Share
hello
Hey! Good to see you. What’s on your mind today?
[illegible]

### Implicit signals: anger and frustration

After the bot says it needs a timestamp to retrieve a Slack message, the user's frustrated reply signals negative feedback.

[5:12](https://www.youtube.com/watch?v=blrovBxxN9o&t=312s) [A user expresses frustration after a bot cannot search for a message.](https://assets.ai.engineer/sites/aiecode2025/sha256/19b7f33e83bf5cb2002b40c3b04fe5c3d1b4126ecf3b959653df99ad033a3027.webp)

Look for implicit signals in the conversation!

Anger / frustration

Bot
Let me search for the original message about John Doe in the channel.

Action Complete
Slack: Retrieve Slack Message and Thread (Custom Action)

I notice that I can't search for Slack messages by content - the only way to retrieve a message is with its timestamp. Could you either:

1. Provide me with the timestamp of John Doe's original message in the #customer-connection-calls channel, or
2. Clarify how you'd like me to search for messages by content, as this functionality isn't available in the current tools?

User
Stop slacking around and figure it out yourself!

Interface text:
AIE
Microsoft
smol ai

### Implicit signals: repeated messages

A user rephrases the same incident request three times. The speaker treats repeated requests as implicit negative feedback: the user may be trying a different phrasing to get a better response.

[5:20](https://www.youtube.com/watch?v=blrovBxxN9o&t=320s) [A user repeatedly rephrases a request about incident assignments.](https://assets.ai.engineer/sites/aiecode2025/sha256/a95dcf00ce832daa39bcc2fa623cfc96aa3aaa3dcb728eb408bddf4c1c372f4b.webp)

Look for implicit signals in the conversation!

Repeated messages

User
How many incidents belong to Justin Deal?

Assistant
[response]

User
list incidents to Justin Deal?

Assistant
[response]

User
Find Justin Deal’s incidents

Interface text:
AIE
aws

### Implicit signals: hostile responses

After the bot asks for a worksheet name, the user replies “Are you dumb?”—a possible signal of frustration. The speaker also notes a surprising amount of cursing.

[5:29](https://www.youtube.com/watch?v=blrovBxxN9o&t=329s) [A user insults a bot after it asks for help identifying a spreadsheet worksheet.](https://assets.ai.engineer/sites/aiecode2025/sha256/5eee87d127debec1f74f6063d1b2bff91abe60a130c0ce40ee6bf8fc3bd31e82.webp)

Look for implicit signals in the conversation!

A lot of cursing

Bot
I'm having trouble accessing the spreadsheet data. Could you please check the worksheet name in the converted Excel file? The name might be different from what we expected. Could you verify the exact worksheet name so I can proceed with processing the payment information?

User
Are you dumb?

Interface text:
AIE
Microsoft
smol ai

### Detecting frustration from implicit signals

Zapier uses an LLM to detect and group frustration in a weekly report. Getting the judge to recognize frustration in the context of its products took considerable tinkering.

[5:35](https://www.youtube.com/watch?v=blrovBxxN9o&t=335s) [Slide recommends using an LLM to detect frustration and aligning the judge to the product context, alongside a weekly frustration report.](https://assets.ai.engineer/sites/aiecode2025/sha256/59b9b1b4ab17268e4b7b55971839940d7e244f2c36ee2042f90db923e21c7deb.webp)

Look for implicit signals in the conversation!

• Use other LLM to analyze the conversation and detect the frustration
• Beware - you need to align your LLM-as-a-judge properly to teach it what’s “user frustration”

Weekly Frustration Report (April 28, 2025 - May 5, 2025)

Overview
This report analyzes user frustrations collected over the past 7 days, focusing on the total number of issues, the number of unique frustrated users, and the most common themes of problems.

Key Metrics
- Total Number of Issues: 4,343
- Total Number of Unique Frustrated Users: 582

Top 5 Most Pressing Themes of Problems

1. Answer Not Helpful
- Count: 1,145
- Percentage: 26.36%
- Examples:
  - User is expressing frustration with the assistant's explanation of type inference. The user provided code where a number is assigned to a variable named 'string1', and the user is implying that the assistant's explanation is incorrect.
  - User is asking for a synonym for the word "precision" and has already rejected "observations" and "insights". This indicates dissatisfaction with the assistant's suggestions.

2. Not Following Instructions
- Count: 745
- Percentage: 17.15%
- Examples:
  - User is frustrated that the assistant cannot shorten the text to 255 characters without removing any information.

Interface text:
Zapier APP 3:07 PM
AIE
aws

### User metrics as implicit signals

Track metrics your business cares about. For example, inspect the last product interactions of customers who churned in the past seven days for implicit signals.

[5:50](https://www.youtube.com/watch?v=blrovBxxN9o&t=350s) [A slide states that user metrics are the ultimate implicit signal.](https://assets.ai.engineer/sites/aiecode2025/sha256/734df98212b2be94223a25002711726fa83b3a65e406ffcbaff0d4946347131a.webp)

The ultimate implicit signal are your user metrics

AIE
Microsoft
smol ai

### I have raw data. What now?

With raw data in hand, the speaker turns to industry experts for what to do next.

[6:11](https://www.youtube.com/watch?v=blrovBxxN9o&t=371s) [A slide asks what to do after collecting raw data.](https://assets.ai.engineer/sites/aiecode2025/sha256/e2a946ee0cee97869a628d0a61044ccab5367200dbd53748d16f3abfaacb4878.webp)

Ok, I have raw data

What now???

AIE
Microsoft
smol ai

### Evals Pyramid

The pyramid moves from fast, high-volume unit-test-like evals through trajectory evals that score artifacts and tool calls to A/B testing on real-world traffic. Impact, difficulty and confidence increase upward.

[8:59](https://www.youtube.com/watch?v=blrovBxxN9o&t=539s) [A pyramid places unit-test-like evals below trajectory evals and A/B testing, with impact, difficulty and confidence increasing upward.](https://assets.ai.engineer/sites/aiecode2025/sha256/c51188e5a2da02063253af4602792760727675cb4c70f54b8348da094486f434.webp)

Evals Pyramid

A/B Testing
Real world traffic, slow to execute, users vibes matter the most

Trajectory evals
Scoring artifacts/collection of tool calls, better for agentic, side-effects!

Unit-test like evals
Fast to add, fast to run, high volume, Not great for agentic workloads

Impact, difficulty, confidence

### Unit test evals

Unit test evals run the LLM on a current state and check the output with assertions, such as whether the next tool call and its parameters are correct.

[9:22](https://www.youtube.com/watch?v=blrovBxxN9o&t=562s) [Unit test evals use assertions on outputs, tool calls, parameters, recall and precision.](https://assets.ai.engineer/sites/aiecode2025/sha256/9e40927748fced035d60d2cff374de4c313b3c8b21bdcbac83cb57ae0b35a3a3.webp)

Unit test evals

• State + LLM(state) ⇒ Output
• Just assertions!
• Tool call
• Tool call parameters
• Recall/precision

### Starting with unit test evals

Start with unit test evals: they are easy to add and help reproduce specific failure modes. Turn negative feedback with a fixed expectation into an eval, but avoid converting every positive response.

[9:41](https://www.youtube.com/watch?v=blrovBxxN9o&t=581s) [The slide recommends starting with unit test evals and turning negative feedback with fixed expectations into evals.](https://assets.ai.engineer/sites/aiecode2025/sha256/97cc62453c2f88815e027da6374a8d6e46a2483013049c531c211c8e13132fa0.webp)

Unit test evals

• Focus on these first - easiest to add
• Every positive feedback can be turned into eval (but don’t do it)
• Every negative feedback with fixed “expectation” can be turned into eval

### Braintrust MCP to the rescue

A reasoning model uses Braintrust MCP to compare experiment runs, describing Claude as more action-oriented and Gemini as more likely to ask questions, with JSON formatting failures in the Gemini run.

[10:48](https://www.youtube.com/watch?v=blrovBxxN9o&t=648s) [A prompt asks for a comparison of model behavior, alongside a generated summary of differences in action, verbosity and output formatting.](https://assets.ai.engineer/sites/aiecode2025/sha256/6a6f1a8c44e5653d07077aef0f4a46088faf129d0c105a4a05e7f3002e9a952a.webp)

Braintrust MCP to the rescue

You are an expert AI/ML engineer. Can you please look into the following comparison of experiments in Braintrust and tell me (based on the output from "Predict Next Step") how the models are behaving differently?

Project: "Central - Core" (id: [illegible],
- Experiment name: claude-3-5-sonnet-20241022-workflow-agent-v2-predict-next-step-1744899224389-FULL
- Experiment name: gemini-2.5-pro-preview-03-25-workflow-agent-v2-predict-next-step-1744899224389-FULL

ignore latency, costs, tokens consumption, "follow-up question tone" and "final message is not cut off", focus ONLY on the Predict Next Step's output. Give me very thorough, detailed and expert-level report on the differences in behaviors and characteristics of these models. How exactly they're behaving

Executive Summary: Claude vs Gemini "Predict Next Step" Behavior

Performance Gap
• Claude 3.5 Sonnet: 82.3% accuracy vs Gemini 2.5 Pro: 72.2% accuracy (+10pp)
• Cost: Claude 4x cheaper (324 vs 1,252 tokens per response)

Core Behavioral Differences

Claude: Decisive executor
• Takes action when possible, even with uncertainty
• Concise reasoning (few sentences)
• Perfect JSON formatting
• Failure mode: Premature "finished" or wrong tool selection

Gemini: Cautious consultant
• Over-asks clarifying questions instead of acting
• Extremely verbose explanations (4x longer)
• Better follow-up question tone when it does ask
• Failure mode: 5+ cases of unparseable JSON output (null)

Specific Behavior Patterns

When they diverge most:
• Action vs Question: Claude executes tools; Gemini asks for clarification
• JSON Reliability: Claude always valid; Gemini sometimes breaks schema
• Risk Tolerance: Claude acts on incomplete info; Gemini seeks certainty

Interface text:
Restore checkpoint

## Full transcript

[0:00](https://www.youtube.com/watch?v=blrovBxxN9o&t=0s) [on-hold music] Um, yeah.

[0:16](https://www.youtube.com/watch?v=blrovBxxN9o&t=16s) In brief introduction to Zapier Agents. Um, I believe many of you know what Zapier is, this automation software. Lot of boxes, arrows, essentially about automation, uh, automating your business processes.

[0:28](https://www.youtube.com/watch?v=blrovBxxN9o&t=28s) Um, Agents is just, well, more agentic alternative to Zapier. You describe what you want, we propose you a bunch of tools, a trigger, um, you enable that, and hopefully we enable your-- we automate your whole business processes.

[0:42](https://www.youtube.com/watch?v=blrovBxxN9o&t=42s) Um, and a key lesson that we have after tho-those two years is that, um, building good AI agents is hard, and building good platform to enable non-technical people to build AI agents is even harder.

[0:56](https://www.youtube.com/watch?v=blrovBxxN9o&t=56s) That's because AI is non-deterministic, but on top of that, your users are even more non-deterministic. They are going to use your products in a way that you cannot imagine upfront.

[1:07](https://www.youtube.com/watch?v=blrovBxxN9o&t=67s) So, um, if you think that building agents is not that hard, you probably have this kind of picture in mind. Um, you probably stumbled upon this library called LangChain.

[1:20](https://www.youtube.com/watch?v=blrovBxxN9o&t=80s) Um, you pulled some examples, tutorial, um, you tweaked the prompt, pulled a bunch of tools, you chatted with the solution, and you thought, "Well, it's actually kinda working all right.

[1:31](https://www.youtube.com/watch?v=blrovBxxN9o&t=91s) So let's deploy it and let's collect some profit." Um, turns out the reality has a surprising amount of detail, and we believe that building probabilistic software is a little bit different than building traditional software.

[1:44](https://www.youtube.com/watch?v=blrovBxxN9o&t=104s) Um, the initial prototype is only a start, and after you ship something to your users, your responsibility switches to building the data flywheel. So once you start-- Once your user starts using your product, um, you need to collect the feedback.

[1:58](https://www.youtube.com/watch?v=blrovBxxN9o&t=118s) You're starting to understand the usage patterns, the failures, so they can-- then you can build more evals, build understanding of what's failing, what are the use cases. Um, as you're building more evals and more features, probably your product is getting better, so you're getting more users, and there are more failures, and you have to build more features

[2:17](https://www.youtube.com/watch?v=blrovBxxN9o&t=137s) and on and on and on. So yeah, it form-- it forms this data flywheel. But starting with the first step.

[2:25](https://www.youtube.com/watch?v=blrovBxxN9o&t=145s) Okay, yeah. So starting from the beginning, how do you start collecting actionable feedback?

[2:31](https://www.youtube.com/watch?v=blrovBxxN9o&t=151s) Backing up for just a second, the first step is to make sure you're instrumenting your code, right? Which you probably already are doing. Whether you're using Braintrust or something else, they all offer like an easy way to get started, like just tracing your completion calls.

[2:42](https://www.youtube.com/watch?v=blrovBxxN9o&t=162s) And this is a good start, but actually, you also wanna make sure that you're recording much more than that in your traces. You wanna record the tool calls, the errors from those tool calls, the pre- and post-processing steps.

[2:52](https://www.youtube.com/watch?v=blrovBxxN9o&t=172s) That way, it will be much easier to debug what went wrong with the run.

[2:56](https://www.youtube.com/watch?v=blrovBxxN9o&t=176s) And you also wanna strive to make the run repeatable for eval purposes. So for instance, if you log data in the same shape as it appears in the runtime, it makes it much easier to convert it to an eval run later because you can just pre-populate the inputs and expected outputs directly from your trace for free.

[3:11](https://www.youtube.com/watch?v=blrovBxxN9o&t=191s) And this is especially useful as well for tool calls because if your tool call produces any side effects, you probably wanna mock those in your evals. So you get all that for free if you're recording them in your trace.

[3:22](https://www.youtube.com/watch?v=blrovBxxN9o&t=202s) Okay, great. So you've instrumented your code and you started getting, uh, all this raw data from your runs. Now it's time to figure out what runs to actually pay attention to.

[3:32](https://www.youtube.com/watch?v=blrovBxxN9o&t=212s) Um, explicit user feedback is-- feedback is really high signal, so that's a good, good place to start. Unfortunately, not many people actually click those classic thumbs up, thumbs up and thumbs down buttons.

[3:43](https://www.youtube.com/watch?v=blrovBxxN9o&t=223s) So you gotta work a bit harder for that feedback. And in our experience, this works best when you ask for the feedback in the right context. So you can be a little bit more aggressive about asking for the feedback, but you're in the right context.

[3:53](https://www.youtube.com/watch?v=blrovBxxN9o&t=233s) You're not bothering the user before that. So for us, one example of this is once an agent finish running, even if it was just a test run, we show a feedback call to action at the bottom, right?

[4:02](https://www.youtube.com/watch?v=blrovBxxN9o&t=242s) "Did this run do what you expected? Give us the feedback now." Um, and this small change actually gave us like a really nice bump in feedback submissions, surprisingly. So thumbs up and thumbs down are a good benchmark, a good base-baseline, but try to find these critical moments in your user's journey where they'll be most likely to provide

[4:19](https://www.youtube.com/watch?v=blrovBxxN9o&t=259s) you that feedback, either because they're happy and satisfied or because they're angry and they wanna tell you about it. Um, even if you work really hard for the feedback, explicit feedback is still really rare.

[4:30](https://www.youtube.com/watch?v=blrovBxxN9o&t=270s) Uh, and explicit feedback that's detailed and actionable is even harder because people are just not that interested in providing feedback generally.

[4:38](https://www.youtube.com/watch?v=blrovBxxN9o&t=278s) So you also wanna mine user interaction for implicit feedback. And the good news is there's actually a lot of low-hanging fruit possibilities here. Here's an example from our, from our app.

[4:47](https://www.youtube.com/watch?v=blrovBxxN9o&t=287s) Users can test an agent before they turn it on to see if everything's going okay. So if they do turn it on, that's actually really strong positive implicit feedback, right?

[4:57](https://www.youtube.com/watch?v=blrovBxxN9o&t=297s) Copying a model's response is also good implicit feedback. Even, uh, OpenAI is doing this for ChatGPT.

[5:06](https://www.youtube.com/watch?v=blrovBxxN9o&t=306s) And you can also look for implicit signals in the conversation. So here the user is clearly letting us know that they're not happy with the results.

[5:14](https://www.youtube.com/watch?v=blrovBxxN9o&t=314s) Here they're telling the agent to stop slacking around, which is clearly implicit negative feedback, I think.

[5:21](https://www.youtube.com/watch?v=blrovBxxN9o&t=321s) Sometimes the user sends a follow-up message that is mostly rehashing what they asked the previous time to see if the LLM interprets that phrasing better. That's also, also good implicit negative feedback.

[5:31](https://www.youtube.com/watch?v=blrovBxxN9o&t=331s) And there's also a surprising amount of [chuckles] of cursing. [audience laughing]

[5:36](https://www.youtube.com/watch?v=blrovBxxN9o&t=336s) Uh, recently we had a lot of success as well using an LLM to detect and group frustrations, and we have this weekly report that we post in our Slack.

[5:42](https://www.youtube.com/watch?v=blrovBxxN9o&t=342s) But it took us a lot of tinkering to make sure that the LLM understood what frustration means in the context of our products. So I encourage you to try it out, but expect a lot of tinkering.

[5:52](https://www.youtube.com/watch?v=blrovBxxN9o&t=352s) You should also not forget to look at more traditional user metrics, right? Uh, there's a lot of stuff in there for you to mine implicit signal too. So find what metrics your business cares about and figure out how to track them.

[6:02](https://www.youtube.com/watch?v=blrovBxxN9o&t=362s) Then you can distill some signal from that data. You can look for customers, for example, that churned, churned in the last seven days, and go look at their last interaction with your product before they left, and you're likely to find some signal there.

[6:13](https://www.youtube.com/watch?v=blrovBxxN9o&t=373s) Okay, so I have raw data, but now I'll let the indus- I'll let the industry experts speak.

[6:22](https://www.youtube.com/watch?v=blrovBxxN9o&t=382s) Why isn't it starting? Oh.

[6:25](https://www.youtube.com/watch?v=blrovBxxN9o&t=385s) Look at, look at, look at, look at, look at your data. Look at the data. Look at, look at,

[6:33](https://www.youtube.com/watch?v=blrovBxxN9o&t=393s) look. Look at the data. Look at the data. Look at the data. Look at this data. Always look at your data. Always look at your data.

[6:40](https://www.youtube.com/watch?v=blrovBxxN9o&t=400s) Yeah, or the meetings will continue until everyone looks at their data. [laughing]

[6:45](https://www.youtube.com/watch?v=blrovBxxN9o&t=405s) Um, okay, but how actually you're going to do that? So, um, we believe that the first step is to either buy or build LLMOps software. We do both. Um, you're definitely going to need that to understand your agent runs because one agent run is probably multiple LM calls, multiple database interactions, tool calls, REST calls, whatever.

[7:05](https://www.youtube.com/watch?v=blrovBxxN9o&t=425s) Each one of them can be source of failure, and it's really important to piece together this whole story, understand this, you know, what caused this cascading failure. Um, yeah, I said we are doing both because I believe vibe coding your own internal tooling is really, really easy right now with Cursor and Claude Code, and it's going to

[7:23](https://www.youtube.com/watch?v=blrovBxxN9o&t=443s) pay you massive dividends in the future, um, for two reasons. First of all, it gives you an ability to understand your data in your own specific domain context. Uh, and the second of all, it also...

[7:35](https://www.youtube.com/watch?v=blrovBxxN9o&t=455s) You should be also able to create a functionality to turn every single interesting case or every failure into an eval with the minimal amount of, um, friction. So whenever you see something interesting, there should be like a one click to turn it into an eval.

[7:51](https://www.youtube.com/watch?v=blrovBxxN9o&t=471s) It should become your instinct. Once you understand what's going on on a singular run basis, you can start understanding things at scale. So now we can do feedback aggregations, clustering.

[8:02](https://www.youtube.com/watch?v=blrovBxxN9o&t=482s) You can bucket your, uh, your, your failure modes, you can bucket your interactions, and then you're going to starting to see what kind of tools are failing the most, what kind of interactions are the most problematic.

[8:13](https://www.youtube.com/watch?v=blrovBxxN9o&t=493s) That's going to almost create for you like a automatic roadmap, so you'll know where to apply your time and effort to improve your product the most. Um, doing anything else is going to be a suboptimal strategy.

[8:25](https://www.youtube.com/watch?v=blrovBxxN9o&t=505s) Something that we are also experimenting with is using reasoning models to explain the failures. Turns out that if you give them the trace output, input instructions and anything you can find, they are pretty good at finding the root cause of a failure.

[8:40](https://www.youtube.com/watch?v=blrovBxxN9o&t=520s) Um, even if they are not going to do that, they are probably going to explain you the whole run or just direct your attention into something that's really interesting and might help you find the root cause of the problem.

[8:53](https://www.youtube.com/watch?v=blrovBxxN9o&t=533s) Cool. So now you have a good shortlist of failure modes you wanna work on first. It's time to start building out your evals.

[9:00](https://www.youtube.com/watch?v=blrovBxxN9o&t=540s) And we realized over time that there are different types of evals, and the types of evals that we wanna build can be placed into this hierarchy that resembles the testing pyramid, for those of you that know that.

[9:10](https://www.youtube.com/watch?v=blrovBxxN9o&t=550s) Um, so with unit test-like evals at the base, end-to-end evals or trajectory evals, how we like to call them, in the middle, and the ultimate way of evaluating using A/B testing with stage rolls at, uh, rollouts at the top.

[9:21](https://www.youtube.com/watch?v=blrovBxxN9o&t=561s) So let's talk a bit about those. Starting with unit test evals, we're just trying to predict the N plus one state from the current state, so these work great when you wanna do simple assertions, right?

[9:31](https://www.youtube.com/watch?v=blrovBxxN9o&t=571s) For instance, you could check whether the next state is a specific tool call, or if the tool call parameters are correct, or if the answer contains a specific keyword, or if the agent determined that it was done.

[9:41](https://www.youtube.com/watch?v=blrovBxxN9o&t=581s) All that good stuff. So if you're starting out, we recommend focusing on unit test evals first because these are the easiest to add. It helps you build that muscle of looking at your data, spotting problems, creating evals that reproduce them, and then just focusing on fixing them, right?

[9:55](https://www.youtube.com/watch?v=blrovBxxN9o&t=595s) Beware though of like turning every positive feedback into an eval. We found that unit test evals are best for hill climbing specific failure modes that you spot in your data.

[10:04](https://www.youtube.com/watch?v=blrovBxxN9o&t=604s) So now unit test evals are not perfect, and we realized that ourselves. Uh, we realized we had overindexed on unit test evals when the new models were coming out that were objectively stronger models.

[10:15](https://www.youtube.com/watch?v=blrovBxxN9o&t=615s) But we were-- they were still performing worse in our internal benchmarks, which was weird. Um, and because the majority of our evals were so fine-grained, this made it really hard to see the forest for the trees when benchmarking new mo- new models.

[10:27](https://www.youtube.com/watch?v=blrovBxxN9o&t=627s) There was always a lot of noise when we would try comparing runs. Like when you're looking at a single trace, it's, it's easy to kind of go through the trace and understand what's happening.

[10:34](https://www.youtube.com/watch?v=blrovBxxN9o&t=634s) But when you need to kind of look at it from... I don't know how to put it again. Sorry. Uh, when you wanna look at it, uh, through an aggregation of many traces, then it starts getting difficult to understand what's happening.

[10:44](https://www.youtube.com/watch?v=blrovBxxN9o&t=644s) Why are so many of these, uh, passing and some of these are regressing? Yeah. So we realized that, uh, maybe machine can help us. It turns out in that previe- previous video when I was investigating, uh, one experiment inside Braintrust, there is a lot of looking at that screen trying to figure out what went wrong.

[11:00](https://www.youtube.com/watch?v=blrovBxxN9o&t=660s) And we were like, "Hey, maybe we can like just give this whole data to, once again, a reasoning LLM and compare the models for us." It turns out that with Braintrust MCP and reasoning model, you can just ask it to, "Hey, look at this run, look at this run, and tell me what's actually different about the new

[11:18](https://www.youtube.com/watch?v=blrovBxxN9o&t=678s) model that we are going to deploy." In this case, it was Gemini Pro versus Claude. And what the reasoning model found was actually really, really good. It found that Claude is like a deci- decisive executor, whereas Gemini is really yapping a lot.

[11:31](https://www.youtube.com/watch?v=blrovBxxN9o&t=691s) It's asking follow-up questions. It needs some positive affirmations, and it's sometimes even hallucinating, uh, about JSON structures. So yeah, it helped us a lot. It also surfaces a problem with unit test evals a lot, which is, um, different models have different ways of trying to achieve the same goal.

[11:50](https://www.youtube.com/watch?v=blrovBxxN9o&t=710s) And unit test evals are penalizing different paths. They are like hard-coded to only follow, follow one path. And yeah, our unit test evals were overfitting to our existing models or actually data collecting using that model.

[12:03](https://www.youtube.com/watch?v=blrovBxxN9o&t=723s) So what we started experimenting with is trajectory evals. Um- Yeah. Instead of grading just one iteration of an agent, we let the agent run till the whole, uh, to the, to the end state.

[12:17](https://www.youtube.com/watch?v=blrovBxxN9o&t=737s) And we are not grading just the end state, but we are also grading all the tool calls that were made along the way and, uh, all the artifacts that have been generated along the way.

[12:26](https://www.youtube.com/watch?v=blrovBxxN9o&t=746s) Um, this can be also paired with LLM-as-a-judge. Vitor is going to speak about that later.

[12:32](https://www.youtube.com/watch?v=blrovBxxN9o&t=752s) Um, yeah, but they are not free. I think they have really high return on investment, but they are much harder to set up, um, especially if you are evaluating runs that have tools that cause side effects, right?

[12:46](https://www.youtube.com/watch?v=blrovBxxN9o&t=766s) When you are running an eval, you definitely don't wanna send a email on behalf of the customer once again, right? Um, so we had a fundamental question whether we should mock environment or not, and we decided that we are not going mock, we are not going to mock the environment because otherwise you're going to get data that

[13:03](https://www.youtube.com/watch?v=blrovBxxN9o&t=783s) is just not reflecting the reality. So what we started doing is just mirroring, uh, user's environment and crafting a synthesis, uh, synthetic copy of that. Uh, also, they are much slower, right?

[13:15](https://www.youtube.com/watch?v=blrovBxxN9o&t=795s) So they can sometimes take, like, up to an hour, um, so it's not pretty great.

[13:21](https://www.youtube.com/watch?v=blrovBxxN9o&t=801s) And we're also leaning a bit more into LLM as a judge. Uh, this is when you're using LLM to grade or compare results from your evals. And it's tempting to lean into them for everything, but you need to make sure that the judge is judging things correctly, which can be surprisingly hard.

[13:36](https://www.youtube.com/watch?v=blrovBxxN9o&t=816s) Uh, and you also have to be careful not to introduce subtle biases, right? Because even small things that you might overlook might end up influencing it.

[13:44](https://www.youtube.com/watch?v=blrovBxxN9o&t=824s) Lately, we've also been experimenting with this concept of rubrics-based scoring. We use an LLM to judge a run, but each row in our dataset has a different set of rubiks, rubrics that were handcrafted by a human and described in natural language what specifically about this run should the LLM be paying attention to for the score.

[14:02](https://www.youtube.com/watch?v=blrovBxxN9o&t=842s) Uh, so one example of this, did the agent react to an unexpected error from the calendar API and then try it again?

[14:10](https://www.youtube.com/watch?v=blrovBxxN9o&t=850s) So to sum it up, here's our current mental model of the types of evals that we build for Zapier Agents. We use LLM-as-a-judge or rubrics-based evals to build a high-level overview of your system's capabilities, and these are great for benchmarking new models.

[14:23](https://www.youtube.com/watch?v=blrovBxxN9o&t=863s) We use trajectory evals to capture multi-turn criteria, and we use unit test-like evals to debug specific failures, uh, hill climb them. Uh, but beware of overfitting with these.

[14:34](https://www.youtube.com/watch?v=blrovBxxN9o&t=874s) Yeah. And a couple of closing thoughts. Don't obsess over metrics. Uh, remember that when a good metrics become a target, it ceases to be a good target. Um, so when you're close to achieving 100% score on your eval dataset, it's not meaning that you're doing good job.

[14:49](https://www.youtube.com/watch?v=blrovBxxN9o&t=889s) It actually meaning that your dataset is just not interesting, right? Because we don't have AGI yet, so it's probably not true that your model is that good. Um, something that we're experimenting with lately is dividing the dataset into two pools.

[15:04](https://www.youtube.com/watch?v=blrovBxxN9o&t=904s) Uh, into the regressions dataset to make sure that if we are making any changes, we are not breaking existing use cases for the customers, and also the aspirational dataset of things that are extremely hard.

[15:14](https://www.youtube.com/watch?v=blrovBxxN9o&t=914s) For instance, like nailing 200 tool calls, uh, in a row.

[15:20](https://www.youtube.com/watch?v=blrovBxxN9o&t=920s) And lastly, um, let's take a step back. What's the point of creating evals in the first place? Um, your goal isn't to maximize some imaginary number in a lab-like setting.

[15:30](https://www.youtube.com/watch?v=blrovBxxN9o&t=930s) Uh, your end goal is user satisfaction, so the ultimate judge, um, are your users. You shouldn't be optimizing for the biggest scores for the evals and completely disregard the vibes.

[15:40](https://www.youtube.com/watch?v=blrovBxxN9o&t=940s) So that's why we think the ultimate verification method is an A/B test. Just take a small proportion of your, portion of your traffic, let's say 5%, and route it to the new model, route it to the new prompt, monitor the feedback, check your metrics like activation, user retention, and so on.

[15:56](https://www.youtube.com/watch?v=blrovBxxN9o&t=956s) Based on that, probably you can make the most educated guess, uh, instead of being in the lab and optimizing this imaginary number.

[16:06](https://www.youtube.com/watch?v=blrovBxxN9o&t=966s) That's all. Thank you.

[16:08](https://www.youtube.com/watch?v=blrovBxxN9o&t=968s) Thank you. [audience applauding] [upbeat music]
