AI Engineer World's Fair 2026

AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash

Loading slides…

Read the talk

AI Evals for Cross-Functional Teams

DoorDash’s GenAI platform moved evaluation beyond an engineering harness by giving domain experts stable APIs, task-specific annotation workflows, golden datasets, and a reviewable way to calibrate LLM judges.

From a talk by Nachiket Paranjape and Swaroop Chitlur Haridas

At a glance

Ideas worth remembering

  • Evaluation is cross-functional because engineering can provide traces, datasets, and judges, but domain experts must define and apply the product’s quality criteria.

  • The recurring loop is: trace, sample, annotate, review, create a golden dataset, calibrate, monitor, and repeat.

  • Stable APIs let the platform team maintain shared capabilities while operators use coding agents to build annotation UIs suited to menus, images, manual tests, and other tasks.

  • Self-service judge calibration remains a review process: prompt owners can compare the original and calibrated prompts before promoting a new judge.

  • Prompt ownership can remain flexible while teams learn; DoorDash has seen strategy and operations, product, and engineering each own it in different groups.

  • DoorDash reports lower per-annotation costs and faster iteration for thousands of rows each week, but the talk does not quantify the savings or speedup.

One platform must support different kinds of quality

DoorDash’s GenAI platform team supports product teams with shared infrastructure. Its initial platform choices—an LLM gateway for switching models, an agent gateway that centralizes tool connections, authentication, and agent identity, plus open-weights model hosting—help teams balance three competing forces: accuracy, latency, and cost. The speakers treat those forces as relevant to agents as well as individual models. Evaluation became the fourth pillar because none of the other primitives can tell a product team whether the resulting behavior is actually good enough.

The evaluation target changes by product. A consumer discovery or shopping assistant may need a judgment over an entire session. Personalization work may need human judgment at much greater scale. A multi-agent system may need trajectory-based evaluation, where the sequence of actions matters rather than only the final answer. A common platform therefore cannot assume that every team will grade the same artifact with the same interface.

The first organizational discovery was that engineers were not the only people qualified to define or assess quality. Strategy and operations staff, product managers, and labeling partners carried the domain knowledge needed for the judgments. DoorDash first emphasized UIs so non-engineers could contribute, then added stable APIs so engineers could build without waiting for the platform team. Coding agents pushed the design toward what the speakers call workflow-first: strategy and operations staff and product managers could increasingly run the work themselves, not merely submit input through a fixed screen.

0:130:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:13 · section reference included

Quality becomes a division of work

This changes evaluation from an engineering harness into a cross-functional operating process. Traces, datasets, and scoring mechanisms are technical objects, but the criteria encoded in them come from people who understand the product and its users. The platform’s job is to carry that knowledge through the system without pretending engineering can infer every team’s quality bar.

The speakers divide the work into four parallel responsibilities:

  • Strategy and operations: set priorities and decide what quality bar the product should meet.
  • Product: translate those goals into rubrics and workflows that people can apply.
  • Operations: run the annotation process and produce human judgments.
  • Engineering: provide APIs, telemetry, datasets, and automated judges.

The handoffs matter. A broad business goal must become a rubric before annotators can apply it consistently, and those annotations must become structured data before engineering can use them to calibrate or monitor an automated judge.

DoorDash then turns these responsibilities into a recurring loop: capture and inspect behavior, sample it down to a set humans will really review, annotate it with domain expertise, review the annotations, promote suitable examples into a golden dataset, calibrate against that reference, monitor behavior over time, and repeat. Sampling keeps the human workload manageable; the golden dataset gives later measurement an explicit human reference instead of an assumed definition of quality.

Who contributes what to AI quality

Strategy and operationsQuality definition — Set priorities and the target quality bar.
ProductQuality definition — Translate requirements into rubrics and workflows.
OperationsHuman evaluation — Run annotations and contribute domain-specific judgments.
EngineeringPlatform machinery — Supply APIs, telemetry, datasets, and judges.
The platform combines business priorities, operational judgment, and engineering machinery rather than assigning the whole evaluation process to one function.
Read the diagram as text
  • Strategy and operations. Quality definition — Set priorities and the target quality bar.
  • Product. Quality definition — Translate requirements into rubrics and workflows.
  • Operations. Human evaluation — Run annotations and contribute domain-specific judgments.
  • Engineering. Platform machinery — Supply APIs, telemetry, datasets, and judges.
3:504:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:50 · section reference included

Two platform surfaces support one quality loop

At the platform level, DoorDash separates telemetry from workflow. The telemetry layer holds traces, scores, and observations and exposes them through MCP, an SDK, and APIs. The workflow layer is where strategy, operations, and product teams set annotation tasks, review golden datasets, create judges, and calibrate them. One layer records and exposes what the system did; the other organizes the human and automated work used to judge it.

Stable APIs connect those surfaces. Scores and datasets use platform-owned APIs, and the platform’s own UIs sit on the same foundation. This is the important meaning of API-first here: a team does not need to abandon shared data and evaluation primitives merely because it needs a different interface. SDK users and custom UIs still operate over a common access plane.

The lifecycle begins with evidence from production behavior: capture traces and sessions and calculate available scores. Humans then add judgment, context, and domain knowledge. Those labeled examples support judge calibration, after which monitoring produces the next collection of traces. The talk gives no required sampling method, sample size, annotation-agreement threshold, or rule for admitting an example into the golden dataset. Those remain consequential design choices for each implementation.

DoorDash’s continuous evaluation loop

Trace and inspectTelemetry — Capture sessions, traces, scores, and observations.
SampleTelemetry — Reduce the captured behavior to a set humans can inspect.
Annotate and reviewWorkflow — Apply domain knowledge and examine the resulting judgments.
Create a golden datasetWorkflow — Promote reviewed examples into reference data.
Calibrate judges and workflowsWorkflow — Measure or optimize evaluation behavior against the reference set.
Monitor over timeTelemetry — Observe deployed behavior and begin the next iteration.
Production behavior becomes manageable human review, then reference data for calibration and another round of monitoring.
Read the diagram as text
  • Trace and inspect. Telemetry — Capture sessions, traces, scores, and observations.
  • Sample. Telemetry — Reduce the captured behavior to a set humans can inspect.
  • Annotate and review. Workflow — Apply domain knowledge and examine the resulting judgments.
  • Create a golden dataset. Workflow — Promote reviewed examples into reference data.
  • Calibrate judges and workflows. Workflow — Measure or optimize evaluation behavior against the reference set.
  • Monitor over time. Telemetry — Observe deployed behavior and begin the next iteration.
  • Trace and inspect → Sample: reduce review volume.
  • Sample → Annotate and review: send selected cases to humans.
  • Annotate and review → Create a golden dataset: review and promote.
  • Create a golden dataset → Calibrate judges and workflows: supply a human reference.
  • Calibrate judges and workflows → Monitor over time: deploy and observe.
  • Monitor over time → Trace and inspect: repeat.
6:026:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:02 · section reference included

Operators build the interface the task needs

Annotation is where a large collection of agent sessions becomes an investigation: which cases went well, which failed, and what happened inside each session? The obstacle is interface variety. Image review, manual testing, restaurant-menu grading, and other tasks do not necessarily need the same controls or layout. A central platform team trying to design every task-specific UI would become a queue for the rest of the organization.

DoorDash splits responsibility instead. The platform team maintains the APIs, strategy and operations decide what should be annotated, and annotators perform the labeling. Because the common data operations already exist behind stable APIs, strategy and operations partners can use coding agents to create the task-specific annotation interface themselves. The presenters mention coding tools including Codex and Claude Code, but the architectural point does not depend on a particular agent: generated interfaces call shared platform capabilities rather than recreating the evaluation backend.

The restaurant-menu example is deliberately modest. The interface only needs to present the material and collect the required annotation cleanly. Its value is not technical novelty; it puts workflow design in the hands of operators who understand the task. Image annotation and manual testing can use different UIs while retaining the same underlying platform pattern.

This freedom also creates an unstated governance question. The talk does not explain how generated UI code is reviewed, secured, maintained, or checked for faithful implementation of a rubric. The example establishes that stable APIs make decentralized interface creation possible; it does not establish that every generated annotation tool will be correct merely because it looks clean and completes the task.

A shared backend, with task-specific annotation UIs

Platform APIsCentral platform — Expose shared scores, datasets, and data operations.
Strategy and operationsDomain team — Decide what to annotate and shape the workflow.
Coding agentDomain team — Builds a task-specific interface over the stable APIs.
Annotation UIAnnotation task — Presents menus, images, manual tests, or other task material.
AnnotatorAnnotation task — Reviews the material and records a judgment.
The platform team maintains common APIs while operators choose the data and generate an interface suited to the annotation task.
Read the diagram as text
  • Platform APIs. Central platform — Expose shared scores, datasets, and data operations.
  • Strategy and operations. Domain team — Decide what to annotate and shape the workflow.
  • Coding agent. Domain team — Builds a task-specific interface over the stable APIs.
  • Annotation UI. Annotation task — Presents menus, images, manual tests, or other task material.
  • Annotator. Annotation task — Reviews the material and records a judgment.
  • Strategy and operations → Coding agent: describes the needed workflow.
  • Platform APIs → Coding agent: supplies stable endpoints.
  • Coding agent → Annotation UI: generates.
  • Annotation UI → Annotator: collects judgments from.
  • Annotator → Platform APIs: writes annotation data through.
8:138:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:13 · section reference included

Judge calibration must be self-serve and reviewable

Once human annotations have become a golden dataset, a team can use them to improve an LLM judge. The described process starts with a simple judge prompt stating what should be measured. The team runs that judge over traces to establish baseline scores, applies a prompt-optimization loop, and promotes the resulting prompt when the partner team is satisfied. Calibration here changes the judge’s instructions against reference data; the speakers do not describe training model weights.

DoorDash packages this process behind a self-service UI because judge calibration is still new and evolving, even if the basic idea sounds straightforward. A product manager or operator chooses the exposed configuration and runs the loop without repeated back-and-forth with engineering. The example supports multiple model providers, making model selection part of the workflow rather than a platform-team intervention.

Self-service alone would hide too much. Prompt optimization can otherwise feel like a closed box, so the platform shows the original system prompt beside the calibrated prompt. The prompt owner can inspect what changed before adopting it. The presenters report significant improvement in one example, but provide neither the prompts nor a numerical metric, so the stronger supported lesson is about reviewability: optimization should produce a proposed artifact that a domain owner can examine, not merely a higher unexplained score.

The talk also leaves important calibration details open. It does not specify the optimization objective, training and validation split, acceptance threshold, repeated-run variance, or safeguards against overfitting the golden dataset. A side-by-side prompt view can help people understand the proposed change, but visual inspection alone does not prove that a judge will generalize to new production cases.

Who performs that review varies. Strategy and operations own the judge prompt in some teams; a product manager or engineering owns it elsewhere. DoorDash treats this variation as evidence that teams and organizational design are still learning. The platform therefore supports different owners instead of prematurely standardizing one function as the permanent home of judge prompts.

From a baseline judge to a reviewable calibrated prompt

Golden datasetReference — Human-reviewed reference examples used for calibration.
Initial judge promptBaseline — A simple statement of what the judge should measure.
Baseline scoresBaseline — Run the initial LLM judge over traces.
Optimization loopCalibration — Revise the judge prompt against the reference data.
Original and calibrated promptsReview — Show both prompts side by side for inspection.
Promoted LLM judgeDeployment — Adopt the calibrated prompt when the partner team is satisfied.
Golden data drives prompt optimization, but the prompt owner still inspects the proposed change before promotion.
Read the diagram as text
  • Golden dataset. Reference — Human-reviewed reference examples used for calibration.
  • Initial judge prompt. Baseline — A simple statement of what the judge should measure.
  • Baseline scores. Baseline — Run the initial LLM judge over traces.
  • Optimization loop. Calibration — Revise the judge prompt against the reference data.
  • Original and calibrated prompts. Review — Show both prompts side by side for inspection.
  • Promoted LLM judge. Deployment — Adopt the calibrated prompt when the partner team is satisfied.
  • Initial judge prompt → Baseline scores: run on traces.
  • Golden dataset → Optimization loop: supplies reference judgments.
  • Baseline scores → Optimization loop: establishes a starting point.
  • Optimization loop → Original and calibrated prompts: produces a candidate prompt.
  • Original and calibrated prompts → Promoted LLM judge: owner reviews and accepts.
10:1210:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:43 · section reference included

Lower costs come from shorter dependency loops

The platform itself evolved through this work. DoorDash moved from an initial UI emphasis toward API-first and workflow-first operation while reusing existing internal infrastructure. The self-service goal is organizational as much as technical: teams should not need the central platform group to operate every annotation or calibration cycle.

The speakers report a substantial reduction in per-annotation spending at a workload of thousands of rows each week. They also connect self-service annotation and judge calibration to faster iteration. The talk supplies no before-and-after cost, percentage reduction, or measured cycle time, so these remain qualitative operating results rather than quantified performance claims.

The ending returns to the loop rather than the interface: inspect traces and sessions, sample to a size humans can handle, annotate with domain knowledge, build the golden dataset, and use it to calibrate workflows, agents, and LLM judges. Then repeat. The durable mechanism is not any single UI or judge prompt. It is a system that repeatedly turns observed behavior into human judgment, reviewable automation, and another production measurement cycle.

13:2313:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:23 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…