Turning Fails into Features: Zapier’s Hard-Won Eval Lessons

How do you make an agent reliable when nontechnical users decide what it should do? In Zapier Agents, a user describes a business process, the product proposes tools and a trigger, and the user enables the resulting automation. Rafal Wilinski and Vitor Balocco describe two years of learning that building this platform is harder than building an individual agent: unpredictable model behavior meets users whose requests and usage patterns cannot all be anticipated.

The deceptively easy version starts with a LangChain tutorial. Take an example, tweak its prompt, attach a few tools, and chat with it until it seems to work. That produces a prototype, but informal conversations cover only a small part of what happens after deployment.

Shipping starts the evaluation work. Usage produces feedback; feedback reveals failures and unfamiliar use cases; those discoveries become evals and product changes. A better product attracts more users, who expose another set of failures. The feedback loop is therefore part of maintaining the product, not a temporary phase before it becomes reliable.

Actionable feedback needs a run to attach to. Tracing completion calls through Braintrust or another tracing system is a useful start, but an agent can fail outside the model call. Record tool calls and their errors, along with preprocessing and postprocessing, so the trace contains the steps needed to reconstruct what went wrong.

Preserve data in the same shape it has at runtime. That makes a trace reusable: its recorded inputs and outputs can prepopulate an evaluation case instead of requiring a separate translation step. It also makes side effects easier to control. A recorded tool response can stand in for a tool invocation during an eval, avoiding another real action merely to reproduce the run. Designing traces for replay lowers the cost of turning a failure into a test.

Once traces are arriving, the next problem is choosing which runs deserve attention. Explicit feedback is valuable, but ordinary thumbs-up and thumbs-down buttons receive few responses. Timing the request helps: ask when the user has a result to assess, rather than interrupting them beforehand.

Zapier added a feedback call to action after an agent finished, including after a test run: “Did this run do what you expected?” The presenters report that this contextual prompt increased feedback submissions, without quantifying the increase. Both satisfaction and frustration can motivate a response at that moment. Even so, explicit feedback remains scarce, and detailed, actionable explanations are scarcer still.

Product interactions provide additional evidence without asking users to fill out a form. In Zapier, enabling an agent after testing it is treated as strong positive feedback: the user has seen its behavior and decided to put it to work. Copying a response is another positive signal the presenters identify, and they cite ChatGPT as another product using that signal.

Conversations contain negative signals too. Users may explicitly express dissatisfaction, tell the agent to stop slacking, repeat an earlier request with different wording, or curse. Rephrasing is particularly useful to notice: a user who tries the same instruction again may be attempting to repair the model’s interpretation rather than asking for something new.

Zapier began using an LLM to detect and group frustrations into a weekly Slack report. Making that useful required substantial iteration on what frustration meant in this particular product. A generic sentiment label is less helpful than a classification that identifies how the agent disappointed the user.

Traditional business metrics provide another way to select traces. Choose the outcomes the business cares about, then inspect the interactions associated with them. The presenters suggest looking at customers who churned in the previous seven days and reading their last interactions before leaving. That connects a business outcome to concrete agent behavior worth investigating.

A montage repeatedly urges the audience to look at their data; the presenters follow it with a joke that meetings will continue until everyone does. The practical requirement behind the joke is a way to see an entire agent run. One run can include multiple model calls, database interactions, tools, and REST requests. Any one can fail and trigger failures downstream.

Zapier both buys and builds tooling for this inspection. General tracing software captures the execution, while internal tools can present it in the product’s own domain context. The presenters point to Cursor and Claude Code as ways to make such tools easier to build. The critical feature is a low-friction path from an interesting run to an eval—ideally one click—so preserving a discovered failure becomes an instinct.

After individual runs make sense, aggregate them. Cluster feedback, group interactions, and bucket failure modes to discover which tools fail most often and which interactions cause the most trouble. These groups supply a practical roadmap: they show where engineering effort can address recurring customer problems.

The team also experimented with reasoning models as diagnostic assistants. Give the model the trace output, input instructions, and available context, then ask it to explain the failure. Even when it does not identify the root cause, a useful explanation of the run—or a pointer to an unusual step—can narrow the human investigation.

A shortlist of failure modes gives evaluation a concrete starting point. Zapier organizes evals into a hierarchy resembling the testing pyramid: unit-like evals at the base, end-to-end or trajectory evals in the middle, and A/B tests with staged rollouts at the top. Each level answers a broader question about whether the agent works.

A unit eval tests the transition from the current state to state N+1. It can check whether the agent selects a particular tool, supplies the right parameters, includes a required keyword, or decides that it has finished. These assertions make a specific observed failure easy to reproduce. A TypeScript assertion for a tool decision can stay small:

type ToolCall = {
  name: string;
  arguments: Record<string, unknown>;
};

function assertToolCall(actual: ToolCall, expected: ToolCall): void {
  if (actual.name !== expected.name) {
    throw new Error(`Expected ${expected.name}, got ${actual.name}`);
  }

  for (const [key, value] of Object.entries(expected.arguments)) {
    if (!Object.is(actual.arguments[key], value)) {
      throw new Error(`Unexpected argument: ${key}`);
    }
  }
}

assertToolCall(
  { name: "lookup_contact", arguments: { email: "alex@example.com" } },
  { name: "lookup_contact", arguments: { email: "alex@example.com" } },
);

Here the check constrains the tool name and specified scalar arguments; it does not assess the rest of the agent’s work.

The presenters recommend starting with these tests because they build a useful habit: inspect data, identify a problem, reproduce it, and fix it. But a positive user response does not automatically deserve its own unit eval. These tests are most useful for improving specific failure modes, rather than preserving every detail of every successful run.

The limitations became visible when models the presenters considered stronger performed worse on Zapier’s internal benchmarks. Much of the suite tested fine-grained decisions. Individual traces were understandable, but aggregating many passes and regressions made it difficult to tell whether a new model was worse or simply behaved differently.

To investigate, the team used Braintrust MCP with a reasoning model to compare experiment runs and explain what changed. The linked documentation describes the current integration; the talk demonstrates the workflow without specifying its historical tool configuration. In the comparison of Gemini Pro and Claude, the reasoning model characterized Claude as a decisive executor, while Gemini was more verbose, asked follow-up questions, sought affirmation, and sometimes hallucinated JSON structures. Those observations concern the team’s experiment; exact model versions and task distribution are not supplied.

The deeper issue was overfitting the eval to the existing model’s path. Data collected from that model encoded how it typically reached a goal. A test that demands the same next step can penalize another model for choosing a different valid route. Such a test may still be valuable for diagnosing a particular decision, but it cannot by itself establish which model completes the overall task better.

A trajectory eval lets the agent continue to its end state. Evaluation then covers the final result, the tool calls along the way, and the artifacts those calls generated. This allows the test to assess a complete attempt rather than a single iteration, and it can be paired with an LLM judge.

The setup is harder because realistic execution includes side effects. Replaying a customer’s run must not send their email again. For trajectory testing, Zapier chose to mirror user environments with synthetic copies rather than mock the environment, seeking results that better reflected real behavior. This is a different choice from substituting recorded tool outputs in a narrow replay: the broader test needs an environment in which the agent can take multiple steps. The talk does not specify the copies’ isolation controls.

The presenters report that trajectory evals can sometimes take up to an hour; they do not specify the workload or timing protocol. They consider the approach valuable despite the setup effort and slower feedback cycle.

An LLM judge can grade an eval result or compare results, but its judgments also need validation. Small, easily overlooked details can introduce bias. Adding a judge therefore creates another component whose behavior must be checked, rather than eliminating the need to define correctness.

Zapier experimented with human-authored, natural-language rubrics attached to individual dataset rows. Each case tells the judge what specifically deserves attention. One criterion asks whether the agent reacted to an unexpected calendar API error and retried. That evaluates recovery across steps without requiring every run to follow an identical sequence. The accompanying slide places rubric examples involving a calendar error, three emails, and a final-answer mention beside branching LLM and tool paths to a correct end state.

The resulting division of labor separates scoring from execution scope:

These approaches can work together: a rubric can score a trajectory, while unit evals isolate the decisions responsible for a failure. The caution about overfitting remains strongest when a narrow assertion is used to stand in for overall success.

A metric can lose its usefulness when it becomes the target. The presenters treat a score approaching 100% as a reason to question whether the dataset still poses interesting problems, rather than assume the agent has become exceptionally capable. A suite that only contains problems already solved offers little guidance about what to improve next.

Zapier was experimenting with two dataset pools:

• Regression cases protect existing customer use cases when prompts, models, or other system behavior change.
• Aspirational cases expose capabilities that remain difficult, such as successfully completing 200 tool calls in a row.

The 200-call sequence is an example of an ambitious test, not a reported achievement. Keeping these pools separate distinguishes preserving working behavior from making progress on unsolved tasks.

The purpose of evaluation is user satisfaction, not maximizing a laboratory score. Offline results should inform deployment without overriding users’ qualitative experience. That is why the presenters put production A/B testing at the top of their evaluation hierarchy.

Their illustrative rollout sends 5% of traffic to a new model or prompt. Monitor feedback, activation, and retention to see whether the change improves the experience users actually have. That evidence supports a deployment decision in a way that another round of optimizing an offline score cannot: it tests whether the improvement survives contact with the people the agent is meant to serve.