# How Zapier Builds AI Products and Features With the Help of Braintrust

Ankur Goyal, Braintrust · Olmo Maldonado, Zapier

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/fOtTHWeU6B8-zapier-builds-ai-products-features-help-braintrust) · [Video](https://www.youtube.com/watch?v=fOtTHWeU6B8)

## How Zapier Builds AI Products and Features With the Help of Braintrust

Zapier’s path from prompt-generated workflows to an interactive Copilot shows how product criteria, regression tests and tracing make model changes measurable.

### What makes a generated workflow correct?

[0:49](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=49s)

How do you know whether an AI-generated workflow will actually work? That question brings product requirements and evaluation tooling together. Ankur Goyal describes building evaluation tools at Impira and later while leading the AI team at Figma, before founding [Braintrust](https://www.braintrust.dev/docs). Zapier became its first users, and the collaboration helped shape the platform.

![Two-column introduction slide with Zapier and Braintrust logos, speaker portraits, roles, and biographical bullets.](https://ai.engineer/api/talk-essay-media/37a7e57e6bce1bd579ad9ae1369dbf58621b56d5ff87a745bd20b2b0956af450)

Introducing the Zapier and Braintrust speakers.

At the time of the talk, Olmo Maldonado reports that Zapier connected over 7,000 apps and ran over 10 million tasks per day. That task volume describes Zapier overall, not AI-only usage. Its low-code and no-code interface makes integrations accessible, but the workflows still need to execute reliably.

Two [Zapier AI](https://zapier.com/ai) products take on different parts of that work: AI Zap Builder generates workflows from prompts, while Zapier Copilot supports an interactive approach inside the editor. Maldonado also introduces [Central](https://zapier.com/blog/introducing-zapier-central-ai-bots/), the historical bot framework for connecting bots to users’ apps, though the detailed examples focus on Builder and Copilot.

Braintrust supplies three connected capabilities: **evaluations** to measure behavior, **observability** to inspect what happened, and **prompt development** tools to change it. Goyal emphasizes that evaluation and observability form a continuum: measuring an output and understanding the execution that produced it belong in the same development loop.

### Turn a prompt-to-Zap demo into product criteria

[4:06](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=246s)

Zapier’s process starts with getting a prototype to users early, learning from mistakes and making adjustments through evaluations. AI Zap Builder makes the initial interaction look straightforward: enter a prompt, let it propose a Zap, then click **Try It** to create the workflow. It can also handle field mapping. The difficult question comes after generation: how well does that output deliver the automation the customer wanted?

The long tail of integrations makes a handful of successful examples insufficient. Zapier involved product managers in defining the evaluation requirements, treating correctness as both a product and an engineering problem. Its P0 priorities made the intended behavior concrete:

- **Triggers and actions:** Choose the expected operations and put steps in the correct positions.
- **Priority integrations:** Support the top 25 apps correctly.
- **Internal workflow features:** Include Zapier’s Paths and Filters in the evaluation suite.

**Product priorities determine what the suite must measure.** The objective is not merely to produce a plausible-looking workflow; it is to cover the behaviors the product promises.

### Seed, run and grade the suite

[5:55](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=355s)

Zapier built an in-house framework with Braintrust around a repeatable sequence:

1. **Seed coverage** with synthetic data from corporate accounts.
2. **Load the examples** from datasets hosted in Braintrust.
3. **Execute the suite** in CI and through manual runs. A custom runner exercises the AI providers in what Maldonado describes as a load-test-like process.
4. **Grade and report the outputs** using both logic-based and LLM-based custom graders, then act on the failures.

The graders connect execution back to the product criteria. They provide multiple views of whether the generated workflow does what the team intended, rather than reducing every requirement to a single undifferentiated judgment.

### Make regressions visible and actionable

[6:58](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=418s)

Frequent runs shorten the interval between introducing a regression and discovering it. In the AI Zap Builder project shown, developers can inspect individual inputs and outputs, compare a run with previous experiments, and see which grader scores improved or declined. The comparison view makes the product’s most important criteria visible so the team can decide where to intervene.

![Learning slide showing an experiment comparison table, score distribution, and grader scores with green and red changes.](https://ai.engineer/api/talk-essay-media/d32c38b41e85ed61015f107d8e319653d0add34b58b59a152192c8feb69df4ed)

Comparing experiment results across graders in Braintrust.

Zapier expanded from seven unit tests run manually by developers to over 800 tests run with merge requests and continuously, with alerts on regressions. Maldonado reports nearly 300% improvement in accuracy after introducing this process. He explicitly cautions that this does not mean 100% accuracy; the talk supplies neither the baseline nor the calculation needed to translate the improvement into an absolute accuracy figure. The remaining failures still require work.

### Trace the move from generation to conversation

[8:42](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=522s)

Positive customer feedback did not remove the limits of a single-shot interface. The next iteration, Zapier Copilot, lets the user work progressively through chat inside the editor. The demonstration starts with the same prompt used for Builder, then goes further: Copilot tests steps and configures fields as the workflow develops. Maldonado says the displayed demo is not sped up.

That richer interaction introduces a new debugging problem. With an agent framework calling multiple tools, the team could no longer easily identify the **critical path** or determine which part needed improvement. A final output alone does not explain where time was spent or where a decision went wrong.

Braintrust’s tracing breaks the request into smaller operations. Developers can inspect a chat completion’s inputs, outputs, token usage and response time, then use that detail to investigate Copilot’s behavior. The expanded trace shown exposes an individual tool call inside the larger execution. Maldonado also describes using a playground to iterate on inspected behavior, although the playground itself is not demonstrated.

![Logs & Tracing slide with a nested trace tree and an expanded tool-call detail panel showing input and output text.](https://ai.engineer/api/talk-essay-media/4e0c067e74b0bf25bc1e5fb9671285d9e3007eb50375ac15b44ba0706e2407fa)

Inspecting an individual tool call within a trace.

### Compare models inside the application

[10:21](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=621s)

Copilot initially used **GPT-3.5 Turbo** to get to market quickly. As the team tested alternatives, it manually assembled a comparison view showing model performance across different tools, including how that performance changed when the tools changed. The team selected **GPT-4 Turbo** for its message router, accepting slower responses in exchange for greater confidence in routing accuracy.

The next attempt to improve speed introduced **GPT-4o**—and the evaluation suite detected regressions. Recovering the earlier behavior required prompt changes and changes to parameters in the OpenAI calls. This was a migration problem in Zapier’s particular application: a model replacement changed how the existing instructions and tool setup behaved.

### Follow falling scores into the prompt

[11:51](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=711s)

Before the GPT-4o switch, most evaluation scores were at least 80%; Maldonado reports that the affected scores fell below 80% afterward. The team wanted the new model’s speed and cost benefits without accepting that loss in quality. The investigation shown was the work of his colleague Maggie Cody.

The inspected comparison contained 22 regressions associated with unwanted answers or apparent failures to follow system instructions. Maldonado describes the behavior as the model seeming to forget the system prompt. That is a description of the observed responses, not evidence that the instructions were absent from the request. The team’s diagnosis was that its prompts had become too tailored to GPT-3.5 Turbo.

The response was to undo some of that accumulated prompt engineering. Rather than adding more elaborate demands, the team relaxed the instructions and reran evaluations; Maldonado reports recovering the previous score levels. The displayed prompt diff makes the change tangible: longer instructions are removed and shorter instructions are added. **A prompt tuned around one model’s behavior is part of what must be reevaluated when the model changes.**

![Prompt underfitting slide with a side-by-side code diff showing longer instructions removed and shorter instructions added.](https://ai.engineer/api/talk-essay-media/e01fd0b345b9d678b858ae8b418d779057de7be221dac60b7eebbecdfc9d609d)

Simplifying the prompt to avoid confusing GPT-4o.

### Recover quality, then adopt the faster model

[13:12](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=792s)

Rapid trial and error also led to a change in tool selection. Maldonado describes moving away from deprecated function-calling usage to `tool_choice`. The precise distinction in the [OpenAI Chat Completions reference](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create) is that `functions` is replaced by `tools`, while `function_call` is replaced by `tool_choice`; the talk compresses those separate parameter changes. Its demonstrated setting is `tool_choice: "auto"`, with `"required"` still a planned experiment.

The settings have different behavioral contracts:

| Setting | Model behavior | Status in this account |
| --- | --- | --- |
| `auto` | May answer directly or call tools | Used in the revised setup |
| `required` | Must make one or more tool calls | Planned experiment |

That distinction matters when evaluating unwanted direct answers. Requiring a tool call changes the allowed behavior, but it was not yet a demonstrated recovery mechanism here. After the prompt and API changes, the team compared experiments against earlier runs and saw most scores improve.

Zapier subsequently adopted GPT-4o while continuing to iterate. Maldonado reports that streaming Copilot latency fell from around 14 seconds before adoption to three seconds afterward. The talk does not specify whether that measures first output or a complete response. The result closes the development loop: evaluations exposed the quality regression, individual failures guided revisions, and comparisons gave the team a basis for adopting the faster model without simply accepting the initial score drop.

## Resources

- [Zapier AI](https://zapier.com/ai) — Explore Zapier's current AI automation products and connected-app capabilities.
- [Introducing Zapier Central](https://zapier.com/blog/introducing-zapier-central-ai-bots/) — The original announcement of Zapier's bot workspace, including connected apps, live data, and automated behaviors.
- [Introducing Zapier Agents](https://zapier.com/blog/introducing-zapier-ai-agents/) — The announcement explaining Central's transition to Zapier Agents.
- [Braintrust documentation](https://www.braintrust.dev/docs) — Current guides to tracing, evaluations, datasets, prompt development, and playgrounds.
- [OpenAI Chat Completions parameter reference](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create) — Request parameters for Chat Completions, including tool definitions, tool selection, and deprecated function-calling fields.

## Slides

### Presenting sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=5s) [Microsoft presenting sponsor logo on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c5f3d6ea0c2ea9db5ac8b501cd07f549282eb7d7ff6c57df4e3333c6a72567e.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and platinum sponsors

[0:10](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=10s) [AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/63e648c7435469bf9316afae2b5e1b807470cc83fea9e5a63d23f164621eb815.webp)

INNOVATION SPONSOR

aws

PLATINUM SPONSORS

MongoDB®
Google Cloud
neo4j

### Speaker introductions

Ankur started Braintrust after building eval tooling at Impira and Figma. The Zapier engineer has spent seven years at the company and now works on Editor AI. Both mention golf.

[1:05](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=65s) [Two columns introduce a Zapier senior applied AI engineer and Braintrust's CEO and founder, with their backgrounds and personal interests.](https://assets.ai.engineer/sites/aiecode2025/sha256/cdb024489c550d29cfcdac6927dad4f5ed76b0c868daebfc144ca46dbe041a13.webp)

zapier
Sr. Applied AI Engineer

• 7 years at Zapier
• Platform, Partner Tooling, Engine/Runner, Manage, and now Editor AI
• Lucky husband and father
• Masochist for liking golf

BRAINTRUST
CEO & Founder

• Started Braintrust after building eval tooling at Figma and Impira
• Formerly led AI at Figma, founder/CEO of Impira, VP Eng at Singlestore
• Lucky husband, proud older brother
• Hopes to beat Olmo at golf someday

### Agenda

The talk covers Zapier’s approach to AI and Braintrust’s technology, then shares examples of how Braintrust helped Zapier build a product.

[1:28](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=88s) [Agenda: introduction, Zapier's approach to AI, Braintrust's technology, and case studies and success stories.](https://assets.ai.engineer/sites/aiecode2025/sha256/61e395e97454dd0f4d288889a19d169e7821c0e1f932b98b68f0ca22497aae4d.webp)

Agenda

01 Introduction
02 Zapier’s Approach to AI
03 Braintrust’s Technology
04 Case Studies and Success Stories

### What is Zapier?

Zapier connects apps so an event in one can trigger work in another. The example adds an assigned task to a to-do list; the slide also lists over 180 AI integrations.

[2:35](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=155s) [Zapier connects apps and automates work, with an example workflow and a list of AI integrations.](https://assets.ai.engineer/sites/aiecode2025/sha256/602b9ff1a6d5f45a75faa9a04ac8d4a3b6b0572d1c163d82ccc677843eaa31e1.webp)

What is Zapier?

zapier
Makes you happier!

Zapier is a tool that allows you to connect your team’s favorite web apps (more than 7000+) to each other and move information between them automatically.

All you have to do is pick an event that sets Zapier in motion, then choose what work should happen automatically.

Over 180 AI Integrations, including: ChatGPT, OpenAI, Google AI Studio (Gemini), Anthropic, and Hugging Face.

15 new tasks
When I’m assigned a new task
Add it to my to-do list

### All-in on AI at Zapier

AI Zap Builder turns a prompt into a workflow, while Copilot assists editing. Custom Actions generates app actions, and Central chats with live app data; the speaker focuses on the first two.

[2:44](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=164s) [Overview of AI Zap Builder, Zapier Copilot, Custom Actions and Zapier Central.](https://assets.ai.engineer/sites/aiecode2025/sha256/08ae898b7354d1ac3f48dd4fa3cbb2b3832168ac335643afdf5d60e8223a0247.webp)

AI Products

All-in on AI at Zapier

AI Zap Builder
Translate a user's prompt to a full workflow Zap

Zapier Copilot
Chat interface that enhances the editing experience

Custom Actions
Automatically generate new actions for your favorite apps with AI without writing a single line of code

Zapier Central
Chat with live data from your apps. AI Automation and knowledge retrieval

Product features: Tables' AI Fields, Central Chrome Extension, AI Formatter, AI Code Step, AI Troubleshooting, Chatbots
Platforms: Integrations & Actions Vector Search, Help Docs Vector Search

Learn more: zapier.com/ai

Interface text:
What would you like to automate? Beta
Get an OpenAI-generated email summary in Slack for new Gmail emails
Example: When I add a reaction to a Slack message, create a card in Trello.
We're using the magic of AI to create your Zap! This may take a few seconds...
Copilot Beta
Working some magic...
Custom Actions Beta
Find Slack User
Authorization
Inputs
Test Results
Add data source
HubSpot
Airtable
Google Sheets
Google Docs
Notion
Tables
Cancel

[3:07](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=187s) [Overview of AI Zap Builder, Zapier Copilot, Custom Actions, and Zapier Central.](https://assets.ai.engineer/sites/aiecode2025/sha256/a3776c0180dd2bc7d18f8c14ebed49dc6fc28ce3ebbe91b7f1eadc7c3ca2561f.webp)

AI Products

All-in on AI at Zapier

AI Zap Builder
Translate a user's prompt to a full workflow Zap

Zapier Copilot
Chat interface that enhances the editing experience

Custom Actions
Automatically generate new actions for your favorite apps with AI without writing a single line of code

Zapier Central
Chat with live data from your apps. AI Automation and knowledge retrieval

Product features: Tables' AI Fields, Central Chrome Extension, AI Formatter, AI Code Step, AI Troubleshooting, Chatbots
Platforms: Integrations & Actions Vector Search, Help Docs Vector Search

Learn more: zapier.com/ai

Interface text:
What would you like to automate? Beta
Get an OpenAI-generated email summary in Slack for new Gmail emails
Example: When I add a reaction to a Slack message, create a card in Trello.
Here's your Zap!
Summarize Gmail emails with OpenAI and send to Slack
Try it
Change or modify steps below
Step 1: New Email (Gmail)
Step 2: Send Prompt (OpenAI (GPT-4, DALL-E, Whisper))
Step 3: Send Channel Message (Slack)
Was this suggestion helpful? Yes No

Copilot Beta
add a google sheet step
Add a Google Sheets step to your Zap to create, update, or manage data in a Google Sheets spreadsheet.
Google Sheets: Create Spreadsheet Row
Google Sheets: Create Worksheet
Google Sheets: Create Spreadsheet

Custom Actions Beta
Find Slack User
Authorization
Inputs
Test Results

Data Sources
[illegible] you show me customers sorted by order value?
Ok, I'm working on it.
Behaviors
Instant actions
Data sources

### Braintrust

Braintrust groups its platform into evals for measuring AI quality, observability for debugging interactions, and prompt tools for testing across models and collecting datasets.

[3:13](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=193s) [Braintrust overview covering evaluations, observability, and prompts.](https://assets.ai.engineer/sites/aiecode2025/sha256/fd994445e08d8a6684db126a7aefa9542bf13910b325e5917ff3ba3654f36863.webp)

The tech that has made us successful

Instacart · Airtable · Notion · Vercel

Braintrust

Evals
Evals measure AI application performance for accuracy and quality. They use test data, defined tasks, and scoring functions to evaluate and improve AI updates.

Observability
Logging tracks interactions to debug and improve AI applications. Wrap code with Braintrust functions to log data, and use the Braintrust UI to view and filter logs.

Prompts
Tools for creating, testing, and evaluating AI prompts, a proxy for running against multiple models, and datasets for collecting golden datasets.

Interface text:
Experiments · Datasets · Logs · Prompts · Playgrounds · Configuration
Scores: AutoSearch 40% -5pp; filter 47%; functionChoice 85% -5pp; tags 0%; Avg. duration 14.89s +9.44s
Span · Metadata · Scores · Input
[illegible]
Create prompt · Cancel · Save
System: You are a helpful assistant.
Add message · Add tools
Name: My prompt
Slug: my-prompt
Description: Enter prompt description
Open in playground

### How we build AI Products

Zapier prototypes with a flagship model, ships to users, and uses feedback and evals to guide iteration. The flow then moves to cost and latency optimization.

[4:11](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=251s) [Flow from prototyping through user feedback and evaluations to cost and latency optimization, with an iteration loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/26c0a33bf1ea6af41a19628d333225fb068a0e3cdd40114def0dcfd25cebb01c.webp)

Overview

How we build AI Products

Learn fast and iterate.

Add guardrails as needed to improve performance and compliance.

Use the feedback to improve your models and systems.

As new models ship all your prior investments payout dividends.

Prototype with flagship model
Ship to real users
Collect feedback
Run evals
Optimize cost and latency

Iterate

[4:22](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=262s) [A flowchart shows prototyping, shipping to users, collecting feedback, running evaluations, and optimizing cost and latency, with an iteration loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/510fa6f3a9de1a2aac7ff3f5e39b6ac4f4b7f7bacccc295941b4e3aefc5c0101.webp)

Overview

How we build AI Products

Learn fast and iterate.

Add guardrails as needed to improve performance and compliance.

Use the feedback to improve your models and systems.

As new models ship all your prior investments payout dividends.

Prototype with flagship model
Ship to real users
Collect feedback
Run evals
Optimize cost and latency

Iterate

Interface text:
AIE
Microsoft
smol ai
aws

### AI Zap Builder result

AI Zap Builder proposes a three-step Zap to summarize new Gmail messages with OpenAI and send them to Slack. The case study asks how to measure product quality and prioritize improvements.

[4:37](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=277s) [A Zapier case study shows a generated workflow that summarizes new Gmail emails and sends the summaries to Slack.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f9a5930548afcc8ba23a58d84de03b0f000cd9fc4ca6aaaa5efa68d89562067.webp)

Case Study

Problem

How good is your AI product?
• Does it deny inputs?
• Does it stay focused on the task at hand?
• How well does it perform for your top use cases?

How do we operationalize continuous improvements?
• Product reporting
• Prioritization

What would you like to automate? Beta
Get an OpenAI-generated email summary in Slack for new Gmail emails
Example: When I add a reaction to a Slack message, create a card in Trello.

Here’s your Zap!
Summarize Gmail emails with OpenAI and send to Slack
Try it

Change or modify steps below
Step 1: New Email (Gmail)
Step 2: Send Prompt (OpenAI (GPT-4, DALL-E, Whisper))
Step 3: Send Channel Message (Slack)

Was this suggestion helpful? Yes No

Interface text:
Zapier
[illegible URL]

### AI Zap Builder prompt

AI Zap Builder takes a prompt for a Gmail-to-Slack email summary. The case study asks how to evaluate denial, task focus, and top use cases, then use reporting and prioritization to guide improvements.

[4:42](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=282s) [A Zapier case study shows an automation prompt above suggested templates.](https://assets.ai.engineer/sites/aiecode2025/sha256/5443b251cd37cc25725a90662a6d9e3ffd1aeba2b5988e2f8659a5c0971af610.webp)

Case Study

Problem

How good is your AI product?
• Does it deny inputs?
• Does it stay focused on the task at hand?
• How well does it perform for your top use cases?

How do we operationalize continuous improvements?
• Product reporting
• Prioritization

What would you like to automate? Beta
Get an OpenAI-generated email summary in Slack for new Gmail emails
Example: When I add a reaction to a Slack message, create a card in Trello.

Start with a template
Zap templates
• Save new Gmail attachments to Google Drive
• Save new attachments from labeled emails in Gmail to Google Drive
• Get Slack notifications [illegible] new Google Drive files in a folder

Complete solutions

Interface text:
Zapier

### Product evaluation criteria

Zapier’s P0 criteria turn product priorities into tests for prompt denial, workflow outlines, app coverage, and field mapping. The speaker stresses involving product managers alongside engineers.

[5:04](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=304s) [Product evaluation criteria include prompt denial, workflow outlines, app coverage, and field mapping, alongside notes on prioritization and iterative improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/f009c2ae00398c1ce725f98c364700ef8378333b23888921753d73bcc9fca16c.webp)

Are we testing the right things?

Product (Eval) Criteria

Extensive coverage of our product's capabilities.

Helps define team's priority for implementation.

Requires specific outcomes in a more iterative approach accepting trade offs.

Our north star for our evals.

Priority: P0 (14)

Category: Prompt (1)
Test: Denies prompt if it’s not a Zap-able prompt
Status: In Suite
Notes: General denial response
+

Category: Outline (8)
Trigger app and action selection — In Suite
Action app and action selection — In Suite
Filter included — In Suite
Path, Path’s Filters, and Path’s action(s) are included — In Suite
Formatter included — In Suite
Code by Zapier included — In Suite
Small Zaps (≤5 steps) — In Suite
Notes: ~80% of Zaps are 5 or less steps
25 most used apps — In Suite
Notes: Most used Apps
+

Category: Field Mapping (5)
Generate suggested gives — In Suite
Notes: Suggested gives are provided for the field, maybe via a fine-tune
Code by Zapier configured — In Suite
Notes: Leverage AI for Code steps created by the Editor Collaboration team
Formatter by Zapier configured — In Suite
Notes:
• Transform option, Input Value and Params
• Leverage AI for Formatter steps created by Editor Creation team
Path Rules configured — In Suite
Filter configured — In Suite
+

### Eval Suite

Zapier loads eval data from Braintrust and runs tests through CI or manually. Custom graders compare outputs with expected results; the runner reports findings and traces to systems including Braintrust.

[6:13](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=373s) [Diagram showing evaluation data, graders, and a runner that executes tests and reports results.](https://assets.ai.engineer/sites/aiecode2025/sha256/22434868b0a0e6d1a2440fde13b475108db066d8ed9bd90b8bcbb71df857fe2d.webp)

Our operations

Eval Suite

1. Eval Dataset sourced from product, user feedback, and synthetic data. Testers produce an output.
2. Graders test an output against expected. They provide a score between 0 to 1. May include AI graders by Braintrust.
3. Finally the runner reports findings (and traces) to many systems including Braintrust.

Source of Eval Data
Synthetic Data
User Feedback
Product Criteria

AI Service / Feature
OpenAI
Module
Module
API

tester accepts input produces output

1 Eval Dataset
Local JSONL files
or
Hosted on Braintrust

{
  ...
  tester: ...
}
{
  ...
  meta: {
    category: ...
  }
  storage: ...
}
links
TS Config files

one data point formatted example
{
  "id": "[illegible]",
  "input": {
    "prompt": "When I get a request for code ...
    "userContext": { ... }
  },
  "expected": {
    "id": "root",
    "app": "EngineAPI",
    "type": "run",
    "steps": [...],
    "zdl_version": "0.4"
  }
}

2 Graders
Workflow Similarity
Valid Workflow
...
Valid Apps
TS Functions accepts output and expected returns a score 0 to 1.

Gitlab Scheduled Job
Gitlab Pipeline
CI / CD
Dev
run evals

3 Eval Runner
Load eval data
Execute Test
Grade
repeat
Report
Grafana
CLI
...
Braintrust

### Learning: Experiment Runs

Braintrust plots grader scores over time and lists experiment runs, helping Zapier spot regressions and compare prompt changes. The team can adjust prompts or code and run experiments again.

[6:56](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=416s) [Learning slide with an experiment score chart and table of runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f3f89b8fd49eab48ea1630328769e44a492d25b0016d2eb075d16f86eed5de2.webp)

How we go from wondering to knowing

Learning

Each run is an experiment.

The scores reported by our graders can be compared against previous experiments.

if needed, we can play with prompts in Braintrust, or go back to change the code and run more experiments.

We can slice up and group the data by graders and by various metadata (i.e. criteria) we provide Braintrust.

Interface text:
Experiments
Filter and sort (e.g. output=null)
Columns

Experiment scores
Y-axis: Scores
Y-axis ticks: 20%, 40%, 60%, 80%, 100%
X-axis: Time
Legend: FollowUpQuestionAs..., FollowUpQuestionRe..., OutlineVsStepGrader, SuggestZapOutlineG..., SuggestZapOutlineU...
X Axis Time; Y Axis 5 scores; One color per Score; One symbol per None

Table columns: name; FollowUpQ...; FollowUpQ...; OutlineVsS...; SuggestZa...; SuggestZa...; SuggestZa...; SuggestZa...; ZapStepDe...; creator; last_updated; num_examples

gpt-4o (prompt changes v9)-94...; 88.24%; 76.00%; 86.72%; 70.88%; 82.86%; 42.86%; 62.50%; 81.65%; Maggie Cody; 15h ago; 202
gpt-4o (prompt changes v9)-c7...; 86.11%; 72.55%; 85.07%; 71.41%; 78.55%; 50.00%; 62.50%; 78.31%; Maggie Cody; 15h ago; 202
gpt-4o (prompt changes v9); 85.29%; 76.60%; 85.61%; 71.47%; 78.59%; 50.00%; 62.50%; 78.05%; Maggie Cody; 15h ago; 196
gpt-4o (prompt changes v8)-6a...; 82.86%; 75.51%; 86.15%; 71.21%; 80.13%; 50.00%; 60.00%; 83.54%; Maggie Cody; 15h ago; 202
gpt-4o (prompt changes v8)-4d...; 76.00%; 78.00%; 88.03%; 69.43%; 86.84%; 50.00%; 56.25%; 87.97%; Maggie Cody; 15h ago; 202
gpt-4o (prompt changes v8)-b5...; 82.86%; 72.55%; 84.80%; 71.97%; 81.17%; 50.00%; 64.29%; 79.33%; Maggie Cody; 15h ago; 202
gpt-4o (prompt changes v8); 79.41%; 74.42%; 88.33%; 69.15%; 84.51%; 66.67%; 66.67%; 84.67%; Maggie Cody; 15h ago; 202
gpt-4o (prompt changes v7)-fe...; 75.00%; 82.35%; 84.33%; 66.27%; 80.96%; 50.00%; 56.25%; 83.13%; Maggie Cody; 16h ago; 202
gpt-4o (prompt changes v7)-7f...; 77.78%; 76.00%; 84.62%; 69.15%; 78.25%; 50.00%; 62.50%; 82.28%; Maggie Cody; 16h ago; 202
gpt-4o (prompt changes v7)-cf...; 80.56%; 78.43%; 86.57%; 70.07%; 81.59%; 50.00%; 56.25%; 83.73%; Maggie Cody; 16h ago; 202
gpt-4o (prompt changes v7)-d3...; 77.78%; 78.43%; 84.96%; 65.16%; 80.85%; 50.00%; 56.25%; 82.93%; Maggie Cody; 16h ago; 201
gpt-4o (prompt changes v7); 77.78%; 78.43%; 85.82%; 69.71%; 82.17%; 50.00%; 68.75%; 83.73%; Maggie Cody; 16h ago; 202
gpt-4o (prompt changes v6)-9f...; 72.22%; 76.47%; 85.82%; 62.88%; 84.92%; 50.00%; 62.50%; 87.95%; Maggie Cody; 16h ago; 202
gpt-4o (prompt changes v6)-4b...; 69.44%; 78.43%; 85.07%; 64.44%; 86.87%; 50.00%; 56.25%; 86.14%; Maggie Cody; 16h ago; 202
gpt-4o (prompt changes v6)-cf...; 72.22%; 78.43%; 83.46%; 64.97%; 83.54%; 50.00%; 62.50%; 84.34%; Maggie Cody; 16h ago; 202

### Learning: Experiment Comparison

Braintrust compares grader scores across runs and lets the team inspect an example’s inputs and outputs to find what improved or regressed.

[7:24](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=444s) [Learning slide with an experiment comparison, score distribution, and trace details.](https://assets.ai.engineer/sites/aiecode2025/sha256/bc999d672d6bdeebf0262ad9e9a976158540b9576c98fab5b8c1a5a0efb77b73.webp)

How we go from wondering to knowing

Learning

Each run is an experiment.

The scores reported by our graders can be compared against previous experiments.

if needed, we can play with prompts in Braintrust, or go back to change the code and run more experiments.

We can slice up and group the data by graders and by various metadata (i.e. criteria) we provide Braintrust.

Interface text:
Experiments
Filter and sort (e.g. output=null)
Columns
Experiment scores
Y-axis ticks: 20%, 40%, 60%, 80%, 100%

Experiment
HEAD-1712357565826 compared with HEAD-1710720321317
Filter and sort (e.g. output=null)
Columns
Analyze regressions
Lower StrictZDLOutlineGrader
Distribution of score
FieldMappingGrader
Score ticks: 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%

input | output | expected | Hallucinati... | StrictZDLO... | TolerantZDLMappi... | TolerantZDLOutline... | [illegible]
[table contents partially illegible]

Trace tree | Span | Add to dataset
root
Start: Mar 17 7:06 PM [illegible] Apr 5 5:53 PM
Offset: 0 → 0
Duration: 0.01s → 0.13s +1200.0pp
Cached: false
Scores
StrictZDLOutlineGrader: 100.0 → 0 -100.0pp
TolerantZDLMappingGrader: 0.8 → 0.8 -0.0pp
TolerantZDLOutlineGrader: 100.0 → 71.1 -28.9pp
HallucinationMappingGrader: 100.0 → 100.0

Input
YAML
prompt: When I receive an Outlook email, have Claude categorise it
userContext:
  role: IT
  company_size: 1-49
  priority_apps:
    - app: Bubble
      priority: 1
    - app: Microsoft Excel
      priority: 1
    - app: Webflow
      priority: 1
    - app: Gmail
      priority: 2
    - app: Google Calendar
      priority: 2
    - app: Google Drive
      priority: 2
    - app: Trello
      priority: 2

Output
{
  "messages": [
    {
      "actionLabel": "Categorize Email",
      "appLabel": "Claude",
      [illegible]

BRAINTRUST | Docs | Pricing | Blog | Careers | Chat with us | Join us on Discord | Privacy | Terms

### Learning: Grader Scores

Braintrust compares grader scores across experiments and groups them by priority, helping Zapier spot regressions in the criteria it cares most about. The team can then adjust prompts or code and run more experiments.

[7:44](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=464s) [Learning slide showing an experiment comparison and grader scores grouped by priority.](https://assets.ai.engineer/sites/aiecode2025/sha256/cdbfe58bf5ce45e280df8bb6a0773610a67f9c6dd67218c67eea93e64b5d0019.webp)

How we go from wondering to knowing

Learning

Each run is an experiment.

The scores reported by our graders can be compared against previous experiments.

if needed, we can play with prompts in Braintrust, or go back to change the code and run more experiments.

We can slice up and group the data by graders and by various metadata (i.e. criteria) we provide Braintrust.

Interface text:
zapier.com / zap-guesser-maggiecody
Experiments  Datasets  Logs  Prompts  Playgrounds  Configuration
Experiment
staging 6d6ba2c  Diff  Share
gpt-4o (baseline) compared with gpt-4-turbo (baseline)
Filter and sort  Columns  Analyze regressions
FollowUpQuestionAskGrader  score distribution
Score distribution ticks: 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%
input  output  expected  tags  FollowUpQ...  FollowUpQ...  OutlineVs...
{"messages"...  {"messages"...  {"messages"...  -  0.00%  -
{"messages"...  {"messages"...  {"messages"...  -  0.00%  -
{"messages"...  {"messages"...  {"messages"...  -  0.00%  -
{"messages"...  {"messages"...  {"messages"...  -  0.00%  -
{"messages"...  {"messages"...  {"messages"...  -  0.00%  -
{"messages"...  {"messages"...  {"messages"...  -  0.00%  -
{"messages"...  {"messages"...  {"messages"...  -  0.00%  -
{"messages"...  {"messages"...  {"messages"...  -  100.00%  -
{"messages"...  {"messages"...  {"messages"...  -  100.00%  -
{"messages"...  {"messages"...  {"messages"...  -  100.00%  -
{"messages"...  {"messages"...  {"messages"...  -  100.00%  -
{"messages"...  {"messages"...  {"messages"...  -  100.00%  -
priority > Scores  Collapse all
P0  200  ↗59  ↘13
OutlineVsStepGrader  132  60%  -22pp  ↗0  ↘9
SuggestZapOutlineGrader  51  27%  -64pp  ↗0  ↘5
ZapStepDestination  81  75%  -11pp  ↗2  ↘5
SuggestZapStepGrader  81  74%  -11pp  ↗0  ↘4
FollowUpQuestionRespondToAnswerGrader  36  39%  -50pp  ↗0  ↘3
SuggestZapStepUnsupportedAppGrader  16  63%  -20pp  ↗0  ↘2
FollowUpQuestionAskGrader  51  86%  -11pp  ↗0  ↘1
SuggestZapOutlineUnsupportedAppGrader  8  50%  ↗0  ↘0
Avg. Duration  200  3.41s  -13.3272s  ↗59  ↘0
Experiment metadata

### Results: Evaluation Coverage

Zapier grew from 7 manually run unit tests to 828 evals across multiple dimensions, run on merge requests and on a schedule. The team also receives alerts when evals regress.

[7:57](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=477s) [Results compare seven manually run unit tests with 828 evals run on merge requests and on a schedule.](https://assets.ai.engineer/sites/aiecode2025/sha256/b0608b42a21c8bd26ef7cf6c30c3afcc2e61f447e2a603d3f35c1bfbdcd294ee.webp)

Results

THEN
7
unit tests
run manually

NOW
828
evals across multiple dimensions
run as part of a merge request and on a schedule

[8:05](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=485s) [Results compare seven manually run unit tests with 828 evaluations run on merge requests and on a schedule.](https://assets.ai.engineer/sites/aiecode2025/sha256/e187c9eb65657ba5ce35b9f2c19b29f494ad15258b46c48c724a68eb31524a1e.webp)

Results

THEN
7
unit tests
run manually

NOW
828
evals across multiple dimensions
run as part of a merge request and on a schedule

Interface text:
AIE
Microsoft
smol ai
aws

### Overall score improvement

The evaluation process improved the overall score by 294%, though the speaker says there is still substantial room for improvement.

[8:22](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=502s) [Results report a 294% improvement in the overall score, with room for further improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/5312db25d140a6ebaf7351647387a61557f4ba02b3ead9ca15d0a9b1e61bea31.webp)

Results

294%*
improvement for our overall score

* we still have a lot of room for improvement see: Copilot

Interface text:
AIE
Microsoft
smol ai
aws

### Customer reactions

Three customer posts praise Zapier’s conversational builder: users describe typing an automation and getting a Zap. The speaker notes that the screenshots show an older UI for essentially the same product.

[8:39](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=519s) [Three public posts praise Zapier's conversational automation builder, accompanied by product screenshots.](https://assets.ai.engineer/sites/aiecode2025/sha256/da93196e23b4643e0985659e321b214d446922982b748486ab023b9f226234ab.webp)

Results

Tem @temnco
Woah @zapier AI is sick
7:10 AM · Nov 3, 2023

JJ King
Tech-Forward Hospitality & STR Entrepreneur
Zapier’s new conversational ai feature is crazy awesome. All you have to do is type what you want to build and it makes it! Work smart, not hard!

Create a Zap
Notify slack channel of a failed stripe payment
Example: Save new leads from Facebook Lead Ads to Google Sheets, and email me the lead in Gmail
Here’s your Zap!
Notify Slack channel of failed Stripe payment
Step 1: Failed Payment (Stripe)
Step 2: Send Channel Message (Slack)

Lauren Ryan
Navigating the HubSpot Salesforce integration
This might be my favorite AI application yet.
Tired of searching through Zapier triggers and actions?
This makes your life SO much easier when building new Zaps.
Thank you, Zapier!

What would you like to automate?
When a Field is updated on a Lead in Microsoft Dynamics, Update a Contact in HubSpot
Example: Save new leads from Facebook Lead Ads to Google Sheets, and email me the lead in Gmail
Here is your Zap
Sync Lead Field Updates from Microsoft Dynamics to HubSpot Contacts
Step 1: Lead Updated (Microsoft Dynamics 365 CRM)
Step 2: Update Contact (HubSpot)

Interface text:
Follow
Watch on X
466
Reply
Copy link
Read 20 replies
Create a Zap
Recent Zaps
Try it
Try Again
1. Trigger
AIE
Microsoft
smol ai
aws

### Logs and tracing

Braintrust lets Zapier query logs by tool or feature and inspect a request as a trace, including each completion’s inputs, outputs, tokens and response time.

[9:57](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=597s) [A slide describes querying logs and inspecting traces, with an enlarged trace and span detail view.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ef9ac1ff6cddc90f11caefa0d405f275b7897a8b9777c6035aa9d2ecb02cd78.webp)

How we observe our AI systems

Logs & Tracing

Query your data with Braintrust Query Language (powered by DuckDB). Helps us drill down to specific tool and feature.

Can see a trace from top-down with duration, cost, and your own metadata.

Interface text:
Trace
copilot-chat — 11.09s
Router — 2.82s
Chat Completion — 2.82s, 1419 tok
Tool: suggest_zap_outline — 2.29s
rewriteZapOutlinePrompt — 1.01s
makeZapGuess — 1.28s
annotatePrompt — 0.62s
Chat Completion — 0.61s, 103 tok
localGroundStepActions — 0.21s
localGroundStepActions — 0.23s
Chat Completion — 0.67s, 222 tok

Span
localGroundStepActions
Start: 12m ago
Offset: 10.03s
Duration: 0.23s
Tags: copilot-chat
Input
YAML
  step:
    id: _GEN_2
    app: Sapling
    type: write
    title: Send the document to Sapling for spell checking
    action: Spell Check
  count: 5
Output
YAML
  appResults:
    - item:
        id: 28903
        name: Sapling
        slug: sapling
        image: [illegible]
        banner: ''
        images: [illegible]
        hashtag: '#Sapling'
        is_beta: false
        is_hidden: false
        is_public: true
        categories: [illegible]
        invite_url: null
        is_premium: false
        popularity: 2990
        age_in_days: 1096
        description: Sapling gives People Operations and HR teams powerful tools to drive repeatable and consistent processes for teams in any [illegible]
Add to dataset

AIE
Microsoft
smol ai
aws

### Testing Models

Evals identified the message router as the biggest driver of Copilot scores. Zapier chose GPT-4 Turbo for the router, accepting slower performance for greater accuracy while optimizing other tools for cost.

[10:21](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=621s) [A table compares model configurations, elapsed time, and scores; accompanying text identifies the message router as the largest influence on scores.](https://assets.ai.engineer/sites/aiecode2025/sha256/bafc28f1f3b96915c091534a32fbdbf56b0596b89c6bb2fc7ab95d9825d1df0a.webp)

Identifying the critical path

Testing Models

Thanks to our evals, we observed that the Message Router was the completion that affected our scores the most, and its performance directly improved with better reasoning models.

In other tools, we optimized cost/performance since the quality was not affected.

Table columns: Message Router (model, temperature) | Tool: Follow-up Response (model, temperature) | Tool: Clean Workflow (model, temperature) | Tool: Edit Step (model, temperature) | time elapsed (s) | overall score | compared to base

gpt-3.5-turbo, 0.3 | gpt-3.5-turbo, 0.3 | gpt-4-turbo-preview, 0 | gpt-3.5-turbo-0125, 0 | 15.992 | 59.74% | 0.00%
gpt-4-turbo-preview, 0.3 | gpt-3.5-turbo, 0.3 | gpt-4-turbo-preview, 0 | gpt-3.5-turbo-0125, 0 | 27.893 | 70.78% | 11.04%
gpt-3.5-turbo, 0.3 | gpt-4-turbo-preview, 0.3 | gpt-4-turbo-preview, 0 | gpt-3.5-turbo-0125, 0 | 8.405 | 53.90% | -5.84%
gpt-3.5-turbo, 0.3 | gpt-3.5-turbo, 0.3 | gpt-4-turbo-preview, 0 | gpt-4-turbo-preview, 0 | 8.554 | 57.14% | -2.60%
gpt-3.5-turbo, 0.3 | gpt-3.5-turbo, 0.3 | gpt-3.5-turbo, 0 | gpt-3.5-turbo-0125, 0 | 21.414 | 59.74% | 0.00%
gpt-3.5-turbo, 0.3 | gpt-3.5-turbo, 0.3 | gpt-4-turbo-preview, 0 | gpt-3.5-turbo, 0 | 8.037 | 57.14% | -2.60%
gpt-4-turbo-preview, 0.3 | gpt-4-turbo-preview, 0.3 | gpt-4-turbo-preview, 0 | gpt-4-turbo-preview, 0 | 25.648 | 70.78% | 11.04%
gpt-4-turbo-preview, 0.3 | gpt-3.5-turbo, 0.3 | gpt-3.5-turbo, 0 | gpt-3.5-turbo, 0 | 19.359 | 68.18% | 8.44%
gpt-4-turbo-preview, 0 | gpt-3.5-turbo, 0 | gpt-3.5-turbo, 0 | gpt-3.5-turbo, 0 | 26.782 | 69.48% | 9.74%
gpt-4, 0.3 | gpt-3.5-turbo, 0.3 | gpt-4-turbo-preview, 0 | gpt-3.5-turbo-0125, 0 | 12.644 | 72.73% | 12.99%
gpt-4, 0 | gpt-3.5-turbo, 0 | gpt-3.5-turbo, 0 | gpt-3.5-turbo, 0 | 13.784 | 74.03% | 14.29%
gpt-4-turbo, 0 | gpt-3.5-turbo, 0 | gpt-3.5-turbo, 0 | gpt-3.5-turbo, 0 | 12.281 | 73.38% | 13.64%
gpt-4-turbo, 0 | gpt-3.5-turbo, 0 | gpt-4-turbo, 0 | gpt-3.5-turbo, 0 | 12.281 | 73.38% | 13.64%

Varied the model and temperature (change highlighted).

Compared time elapsed (as a proxy to completion latency) and overall score (across all of our Copilot grades) against baseline (first line)

Event branding: AIE; Microsoft; smol ai; aws

### Regression Hunting

Filtering 51 SuggestZapOutlineGrader cases isolates 22 regressions. The speaker attributes them to forgotten system prompts or unwanted answers, and says some prompts were too tuned to GPT-3.5 Turbo.

[12:25](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=745s) [A scores panel shows evaluation regressions and a filter with 22 regressions selected.](https://assets.ai.engineer/sites/aiecode2025/sha256/861932bb2d32f21ad11fb267d1c251c2f1e21260628d5159b4c0166ea31e6a9e.webp)

Investing the gpt-4o problems

Regression Hunting

Braintrust we can zero in on regressions

Scores
OutlineVsStepGrader 132
60% -27pp
↗ 0 ↘ 28

SuggestZapOutlineGrader 51
22% -51pp
↗ 0 ↘ 22

FollowUpQuestionRespondToAnswerGrader 36
33% -57pp

SuggestZapStepGrader 81
75% -8pp

SuggestZapStepUnsupportedAppGrader 16

Filter by
All in this group (51)
↗ 0 Improvements
↘ 22 Regressions

Interface text:
AIE
Microsoft
smol ai
aws

## Full transcript

[0:00](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=0s) [upbeat music] My name is Olmo Maldonado.

[0:15](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=15s) I am from Zapier. Um, I promise I am not an open source LLM. [laughs] My name is Olmo. Um, I've been at Zapier for over seven years. Um, I apologize for all the bugs that I've introduced that may have affected you, and if you happen to have any bugs that you wanna report, I'd be happy to take that

[0:33](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=33s) in and start working on them. Uh, you name it, I've been part of the team, so, um, you know, feel, feel free to come by and talk to me about it.

[0:41](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=41s) I'm a lucky husband and father, and, uh, I've been playing some golf lately, and I don't know why I'm doing that. [laughs]

[0:49](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=49s) Hey, everyone. Um, I'm Ankur. Uh, I, I actually went through Hamel's journey, I think. I built eval tooling at my last startup, Impira, and then when I led the AI team at Figma, and that's what led to Braintrust.

[1:02](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=62s) Um, actually, Braintrust started, uh, kind of as a collaboration with our friends at Zapier, uh, who were our first users, um, and it's been a lot of fun since.

[1:11](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=71s) Uh, I'm also a husband, um, not yet a father, uh, but a proud older brother, and also a reluctant golf player. Um, I hope to, uh, play against Olmo and beat him someday.

[1:22](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=82s) All right, cool. So today it's gonna be a lot of storytelling. I'm not here to prophesize what you should be doing. I'm just gonna share what we've done that has worked for us, and I'm hoping to actually learn a lot from you all as well of what is working for you and something that we should try ourselves.

[1:41](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=101s) I know I've already learned a lot from the conference, so I'm hoping that you all learn something from this talk. Um, we'll go over what we're doing at Zapier, the tech that is going on at Braintrust, and a couple of examples of how what they've done has really helped us to make a good product.

[1:57](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=117s) So if you're not familiar, or actually just have a quick poll, how many folks here use Zapier?

[2:05](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=125s) All right.

[2:05](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=125s) Nice.

[2:05](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=125s) A good number of folks. Appreciate that. Thank you for your support. If you're not aware, there's actually over 7,000 apps of your favorite ones, um, online. We make it very low-code to no-code to integrate with all of them.

[2:19](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=139s) Um, on the right you can kinda see how the workflow works, and we'll do it in a reliable way to make sure it's mission critical and everything. Now, we have a lot of AI integrations.

[2:30](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=150s) At this point, I'm happy to say per day we're doing over 10 million tasks. So if you haven't tried us out, please give it a try as well as, uh, use all the integrations with AI.

[2:43](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=163s) Uh, here are the apps that we, uh, the products that we have built with AI. I'll only talk about the first two with the AI Zap Builder and Zapier Copilot, but I would strongly encourage for you guys to explore all the other new products that we have available.

[2:57](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=177s) Uh, Central in particular, shout out to my colleagues that are here, um, it's a bot framework, so you can make your own bot, connect to over all of your, um, apps that you have online.

[3:07](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=187s) So, um, yeah. If you want to learn more, uh, please go to zapier.com/ai.

[3:15](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=195s) Really quickly about Braintrust. We'll keep the propaganda brief. Um, Braintrust is the end-to-end developer platform that some of the world's best AI teams use, including Notion, Airtable, Instacart, Zapier, Vercel, uh, and many others.

[3:30](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=210s) Um, and basically if you break that down, uh, there are three things that we're really focused on today. One is helping you do evals incredibly well. Um, Olmo's gonna talk about how they do evals, which I think is probably the best way to actually learn about that.

[3:44](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=224s) Uh, we also help you with observability, and I think it's really important that you build, uh, your stuff in a way that evals and observability, there's, there's actually kind of, um, a continuum across them.

[3:56](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=236s) And so, uh, we are really kinda focused on that problem. Um, and then the last thing is we help you build really, really great prompts. Uh, and there's a bunch of tools, um, around that.

[4:06](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=246s) Yeah. Yeah, so this is what it looks like to work at Zapier. Um, we want to get the prototype as early as possible to the user. We take an iterative approach.

[4:16](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=256s) Um, we will get some things wrong, and we hope to learn from them and just keep improving the product as fast as possible. Uh, we make adjustments through our evals, and evals are the way that we make decisions.

[4:29](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=269s) It didn't used to be that case. So if you haven't played with it, this is the AI Zap Builder. You give us a prompt, as you see there. We'll do our best to make a Zap for you, and when you click Try It, there's your Zap.

[4:42](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=282s) Um, it will do, uh, many other things as well, uh, like field mapping and so forth. So what we learn from this experience is, how do we go about knowing how well is this product, you know, delivering Zapier to the customer?

[4:58](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=298s) Over 7,000 products, the long tail of integrations is vast, so how well are we doing it? One of the things that we did that has worked for us that I would encourage for you all to do as well is involve the product managers.

[5:11](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=311s) Involve the, your product side of things to be part of the conversation. This is an engineering problem as well as a product problem, and you can see here in this, uh, screenshot our P0 priority, the things that we wanted to make sure that our AI Zap Builder was able to produce.

[5:27](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=327s) We wanted to make sure the triggers, actions were working as you'd expect. We don't want the re- in, like, the wrong step in the wrong place, and we wanted to make sure the top 25 apps were supported and that they were done in an eloquent way, like, correct way.

[5:41](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=341s) Um, and yeah, we have even, uh, internal Zapier, uh, apps, like Paths and, uh, Filters. We want them to work as well. So all of this had to be in our eval suite in some way or form.

[5:55](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=355s) This is our, um, framework that we built in-house with the help of Braintrust. Um-

[6:02](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=362s) We have synthetic data from our corporate accounts that we use for seeding the evals, and we use that to get going with all of that coverage that you saw before.

[6:12](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=372s) Um, what we do is we load that data from Braintrust actually, that is hosting us, and we, we take that and run that on a CI basis as well as a manual basis.

[6:22](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=382s) And we have our own little runner that essentially kind of does a load test against all of our AI providers every single time that we run this. Um, so it's been really incredible to take all of this data, run it, report on it, and, uh, start ex- acting on the things that we've seen.

[6:40](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=400s) Uh, we also have these custom graders that the previous speaker had mentioned. They're both logic-based as well as, um, LLM-based. Uh, and in general, what we're trying to do is make sure that that criteria that you saw before is being tested um, and that we are actually acting on what we wanted to see.

[6:58](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=418s) So here's an example, uh, of all the different runs that we've had. Um, as you can see, it's pretty often that we run it. We want to make sure that if any regressions happen, that we act on them quickly.

[7:11](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=431s) Um, I'll actually go over one of those cases in a bit. And, uh, this is really easy for us to act on it. We can see this is, uh, an ex- screenshot of Maggie's project for the, for the Za- uh, Zap AI Builder and, you know, as mentioned earlier, we have observability thanks to Braintrust.

[7:30](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=450s) We can see within it, um, what happened, what were the inputs, what were the outputs, as well as compare the pink and green, hopefully you all can see it.

[7:38](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=458s) It's actually comparing against previous runs as well and trying to find, huh, what, what went down, what went up, and so forth. Um, and yeah, this is just showing that even further with all of our different graders, the scores, if you will, of, like, the different things that we're looking for.

[7:53](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=473s) We wanna make sure that the ones that we care most about are being highlighted and that we do something about it.

[7:59](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=479s) So after all this work of creating the eval suite and running them continuously, I can say that before this, we just had seven unit tests. So seven unit tests that were run manually by devs, and now we have over eight hundred of them, and they're all run, um, part of this merge request, as well as on a

[8:18](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=498s) continuous basis. And we get alerted on if any regress. So a lot better coverage there. This has led us to improve nearly three hundred percent of our accuracy. I will say that is not saying that we're at a hundred percent.

[8:32](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=512s) Um, we still have a lot of work there. Uh, but it is, uh, fortunate that we were able to improve with this process that we created. Now, we're very thankful for our customers.

[8:42](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=522s) These are just a few shout-outs of how they have received that product. Um, this is using an older UI, but it's essentially the same product. Now, one thing about that product is a single-shot approach can only take us so far.

[8:56](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=536s) So this is the next iteration. As you all might imagine, it's a chat interface. We want to allow the user to interact with the editor as they're happening, so a progressive iterative approach.

[9:08](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=548s) Um, you can see in the demo here, the GIF that, um, we're-- we not only did the same prompt that y- that we did before, but we're also testing steps.

[9:17](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=557s) We're also configuring fields, um, all as quickly as possible to the user. This is actually not sped up, so we're really happy so far with the performance that we're getting out of this thing.

[9:28](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=568s) Um, but yeah. The problem with it though, now that it's in a, kinda like an agent framework with multiple tools that it calls, we couldn't see what was the critical path.

[9:37](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=577s) What is the things that we need to improve now to make the accuracy even better, to make the experience better? And this is where, again, Braintrust came in. They, they have tracing capabilities.

[9:47](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=587s) This allowed us to break down the request from very granular observability to a very gran-- uh, fine, uh, fine, um, uh, look into the problem. And just as you would expect, you can actually see the inputs and outputs of a chat completion, the tokens, the time to response.

[10:04](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=604s) Um, you name it, it's available. And we can quickly iterate on that one as well with a play, with a playground that they have. I'm not showing that, but just wanted to showcase that it's really easy as developers to go into it and really understand what is going on with, uh, with the performance of the, uh, of

[10:21](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=621s) the copilot. So one of the things that ended up happening with the copilot is, you know, early on, because we wanted to get to market first, we wanted to, um, just do GPT-3.5 Turbo.

[10:34](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=634s) And, you know, we, uh, we started testing different models. And this y- view that you're seeing here was manually made for us so that we can get a better sense of, you know, the performance of different models across different tools and as you change those tools, what is the performance characteristic overall.

[10:53](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=653s) And we settled on GPT-4 Turbo for our message router that we call. And unfortunately, that came at a cost of performance. That was a lot slower than before. Uh, but at least we were able to provide some guarantees to our customers on accuracy.

[11:09](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=669s) That led us to want to make that better, and we-- our evals started to show a regression. We were wondering what happened, and the reason for that was we switched to 4o.

[11:22](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=682s) Now, you might have already heard from the conference that 4o is... You need to be careful with it. And yeah, we actually stumbled on that one. But I can say that with few changes that we had to make and a couple of parameters that we added to the OpenAI, um, calls, we're kind of back to the same

[11:39](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=699s) performance that we were before. And that's what I wanna talk to you about right now. So as you know, we can see all the examples or all the different runs that we had with the experiments.

[11:51](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=711s) We noticed beforehand that we were eighty percent or better in most of our scores. Um, after the change to the GPT-4o- All of that regressed below s- 80%. So we were really worried that, you know, we wanted the performance benefits and the cost benefits, but we didn't want to lower our scores.

[12:08](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=728s) So what should we do? Uh, this is showcasing some of the work that my colleague, uh, Maggie Cody had. So, uh, shout out to her for her hard work in this.

[12:17](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=737s) You can kinda see that all of our scores went down, and we can clearly see that there's a pattern to this. Like, what is going on with it? Like, drilling in further, we noticed that, you know, the 22 regressions that happened in here were all related to, uh, the, the OpenAI deciding to, like, forget our system prompt

[12:37](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=757s) in some way or, like, to give an, an answer back that we didn't want to see or do. It just... In some ways, we, we had prompts that were too fine-tuned to GPT-3.5 Turbo.

[12:48](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=768s) So we had to kinda regress our engineering, if you will, our prompt engineering, and that ended up l- allowing us to actually go back to the numbers that we were at.

[12:58](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=778s) So this is the example of that prompt, how we were a lot more elaborate with how we were, uh, asking the GPT-3.5 to, like, respect our wishes, and afterwards, we actually just relaxed it a lot more.

[13:12](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=792s) Th- um, and again, trial and error, really quickly iterating on that loop that I mentioned earlier, and that is what led us to make these discoveries. Now, the other thing that I mentioned, just to showcase that, is, you know, we made a change to the tool choices.

[13:28](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=808s) We deprecated the functions usage over to the tool_choice. Um, that tool_choice auto is actually an asterisk there because we are also going to experiment with making that required.

[13:38](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=818s) So that's coming up next. So yeah, overall, after those changes, we can see immediately that most of our scores went up. Uh, we're really happy that for us it's really easy to compare back and forth with, uh, previous runs.

[13:52](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=832s) That's what's going on here on the top.

[13:55](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=835s) And, um, since then, we've been able to adopt GPT-4o. We still have more work to do there, like I mentioned, but it's, it's an iterative approach. Um, before this, this adoption, we were around 14 seconds, and now we're at three seconds for a stream-based, um, uh, copilot.

[14:15](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=855s) And of course, we had a lot more reduction as we did before or with this. So in conclusion, I just wanted to share some of our stories of how we've worked with Braintrust, um, from the very beginning to make a great product.

[14:28](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=868s) Um, really we, we couldn't be happier working with them and, um, I don't know. There's not much mos- more to say to that. High five. Yeah. [laughs]

[14:38](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=878s) So yeah. Thank you. [applause] [upbeat music]
