Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft

A customer-support agent can have a carefully written prompt, useful tools, and a retrieval pipeline—and still be unready to handle a real conversation. How do you establish that it can help customers before those customers become its test data? Nick Ung and Akshay Sharma approach that question through their work on Lyft’s multi-agent support system, connecting offline evaluation, production monitoring, and the harness that makes evaluation repeatable.

Development starts with context management, retrieval-augmented generation, tool definitions, the agent graph, and the system prompt. Before deployment, those pieces need to work together through complete interactions. For a multi-turn support agent, that means a representative scenario dataset and a user LLM that carries conversations through to completion. A grader evaluates the resulting trajectories, and explicit performance criteria determine whether the agent passes a launch gate. The purpose is to find failures before exposing live users to them.

After launch, evaluation continues with production traces recording the agent’s executions and the context behind its responses. Online graders assess those interactions, while people inspect errors, identify failure modes, and feed findings back to development. Offline and online evaluation therefore serve different parts of the same loop: establish readiness, then learn from what actually happens.

An evaluation score needs to change a decision. Three failure modes prevent that:

• No gate: A judge produces numbers, but those numbers do not affect development or deployment.
• No trust: Noisy, generic scores provide little actionable information. Lyft encountered this with its early judges: people did not believe the scores or know how to use them.
• No owner: A production regression has no clear detection mechanism or person responsible for correcting it.

A useful evaluation system needs all three connections: a decision that the score informs, a grader people can trust, and an owner who acts on the result.

Lyft took inspiration from Sierra AI’s τ-bench: let an agent and a simulated user interact under the policies of a particular domain, producing a multi-turn trace. Customer-support policies supply the instructions for handling each issue; the simulation tests how the agent applies them as a conversation unfolds.

In Lyft’s simulator, a LangGraph agent interacts with a user LLM whose instructions define several distinct parts of the scenario:

• Intent: What the user wants to accomplish and what the support issue represents.
• World state: Relevant user data, such as an experienced driver operating in a luxury category.
• Behavior and persona: How the user approaches the interaction—for example, a loyal longtime customer frustrated with Lyft’s earnings system.

These fields determine both the problem the agent must solve and how much cooperation it receives while solving it.

The resulting trajectories go through complementary evaluators. An LLM judge assesses behavior that requires interpretation; deterministic assertions check outcomes that can be specified directly. If a scenario requires the agent to grant a concession, the evaluator compares the agent’s tool calls with that expected action. This tests instruction following at the tool-call level rather than relying only on whether the final response sounds helpful.

Simulation quality starts with the dataset. Asking an LLM to generate 50 test queries is an easy shortcut, but it does not establish that the scenarios resemble production. Lyft supplements its offline dataset with samples of historical support interactions, then mutates scenario criteria to cover both expected paths and edge cases. The accompanying slide extends that workflow with an output schema and an over-generate-then-filter step.

Even a plausible scenario can become too easy when the simulated user behaves unlike a customer. Frontier models are trained to be helpful assistants. When asked to play Lyft users, they patiently described their problems in complete, polite messages. Nick reports a pass or accuracy rate above 90% in Lyft’s first offline evaluation with this overly accommodating user simulator. The talk does not specify the dataset size or a more precise metric definition.

Production interactions looked different. Users were already frustrated, impatient, and less willing to explain their issues. Lyft fine-tuned its user LLM on actual Lyft user language so that the simulated conversations more closely resembled those interactions. Evaluation scores fell after that change; Nick gives no post-tuning rate. The harder simulation created room to improve the agent’s handling of difficult interactions, instead of rewarding its performance with unusually cooperative users.

Personas add another dimension of realism. A bypasser wants immediate escalation to a human without giving the AI a chance; other personas include refund seekers and AI skeptics. These instructions ground the user model in recognizable behaviors. Nick also credits Microsoft research on user LLMs as inspiration, describing a similar decrease in evaluation scores when a fine-tuned user model made conversations more realistic.

Different applications emphasize safety, leaks, cost, latency, or response quality. Some checks can be implemented in code; others require an LLM’s judgment. The problem is not having multiple dimensions—it is defining dimensions that do not tell anyone what to fix.

Lyft initially used prebuilt metrics from DeepEval, including tool-usage appropriateness, response helpfulness, conversation naturalness, and completeness. Akshay’s practical question is what an engineer should do with a helpfulness score of 0.5. Toxicity, bias, fairness, and conciseness scores can present the same problem when detached from a concrete requirement. Prebuilt metrics can supply a baseline, but the core evaluation should connect to product outcomes. This criticism concerns Lyft’s use of generic metrics; current DeepEval also supports custom criteria, rubrics, and binary scoring.

The alternative is to work with domain experts and data scientists to define task success or failure. Binary outcomes make judge calibration more tractable and give error analysis a specific target. Lyft’s education rubric illustrates the difference:

Both excessive persistence and premature escalation can fail. A generic helpfulness score obscures that distinction; the rubric identifies which behavior needs to change.

A judge is itself a model whose decisions need evaluation. Akshay suggests starting with around 100 examples hand-labeled as pass or fail for the relevant task or metric. Split those examples into training, development, and held-out validation data, then measure the judge’s precision and recall against human labels. Agreement with an unvalidated judge cannot establish that an agent is working correctly.

Here, training means developing the judge’s prompt rather than updating its weights. Each partition has a separate job:

1. Use training examples to choose few-shot demonstrations for the prompt.
2. Iterate on the prompt or harness against the development set.
3. Evaluate on the held-out test set to check whether those changes overfit the development examples.

The slide illustrates approximate train/dev/test proportions of 20%/40%/40%. This allocation reflects prompt development: the training portion supplies examples, while the larger development and test portions assess the judge’s behavior.

Validation also has to account for criteria drift. Reviewing more outputs changes a team’s understanding of quality: a previously ambiguous interaction reveals a missing rule, or a new failure shows that an existing rule is incomplete. Evaluation criteria should therefore develop alongside observations of the agent. As the criteria change, refine the judge and reevaluate it against the revised expectations.

A bare alignment rate hides uncertainty. Confidence intervals, calibration, and appropriate sampling make reported scores more useful, especially when those scores determine whether to ship or appear in leadership reporting. Akshay recommends concentrating expensive statistical work at those consequential decision points.

In Akshay’s hypothetical example, two evaluators score 84% and 88% on 50 traces; the four-percentage-point difference alone does not establish a real improvement. The slide shows overlapping confidence intervals to make that uncertainty visible. Determining whether the difference is meaningful requires more information than the two headline rates, including the comparison design. Larger effects and paired designs can require fewer examples; the talk does not supply paired outcomes or an exact sample-size calculation.

The foundation is direct inspection of data. Without looking at examples, a team cannot create meaningful criteria or labels. Without labels, it cannot validate its judges. Without validated judges, it cannot confidently interpret the agent’s scores. LLMs can assist this work, but they do not remove the need to examine what happened.

Run error analysis as a recurring procedure:

1. Inspect raw traces from agent interactions.
2. Identify the specific failure modes.
3. Keep metrics tied to business or functional requirements that actually change a decision; remove the noise.
4. Form a revised hypothesis about what needs improvement, reevaluate, and repeat.

Akshay suggests a weekly or biweekly cadence. The important distinction is that this is continuous work, not a one-time audit.

That procedure depends on useful traces. Tools such as LangSmith and Langfuse let teams log and inspect graph execution: which nodes ran, what the model saw, which tools it called, token usage, and latency for each call. Additional metadata can enrich the record. These details make it possible to move from a failed outcome to the execution that produced it.

Annotation queues make those records accessible to domain experts. Reviewers can label examples and give feedback through a focused interface instead of navigating raw trace JSON. Reviewed traces then serve two purposes: they can become offline evaluation cases, and their human labels can support precision and recall measurements for the judges. Production evidence becomes material for the next round of testing.

Evaluation closes the loop when its findings improve the agent. Nick distinguishes three places to make those improvements:

Lyft’s error analysis has informed agent prompts, knowledge-base updates, and context-management strategies. Identifying the failure mode helps determine which part of the system needs attention.

At the time of the talk, the offline simulator still depended on scripts scattered across notebooks and analysis repositories; Nick described it as insufficiently repeatable. The team was investing in a systematic evaluation harness with predefined primitives and configuration-based contributions. It was also considering how accumulated real-user signals could support task-specific model fine-tuning and reward modeling for reinforcement learning. Those were further directions, distinct from the user-simulator fine-tuning already described.

The harness design standardizes evaluation across agents and sub-agents. Editable YAML configuration lets analysts and data scientists contribute alongside engineers. Nick describes thousands, potentially tens of thousands, of evaluation examples as the motivation for parallelism and throughput. The reusable primitives are task, datasets, personas, LLM adapter, and evaluator.

An illustrative configuration brings those primitives together for the concession scenario. Dataset records supply the world state and expected action; the persona controls user behavior; separate evaluators assess the tool outcome and the education rubric.

task: customer_support_concession
datasets:
  - concession_cases
personas:
  - loyal_customer_frustrated_with_earnings
llm_adapter: support_user_simulator
evaluators:
  - name: expected_concession_tool_call
    type: deterministic
  - name: education_rubric
    type: llm_judge
    outcomes: [pass, fail]

This is the configuration shape the design calls for: reusable scenario and evaluator definitions that contributors can assemble without rewriting the simulation runner.

Once defined, the same configuration can be rerun at different development gates. A local run after a prompt change compares the agent with its previous version. A pre-commit check could catch degradation before a change is pushed to the agent service. Nick also identifies CI/CD regression and acceptance suites as integration points the team is exploring. The intended endpoint is a repeatable engineering workflow in which evaluation accompanies changes closely enough to influence whether they move forward.