# AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash

Nachiket Paranjape, DoorDash · Swaroop Chitlur Haridas, DoorDash

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/bMjlRrWjdT0-ai-evals-cross-functional-teams) · [Video](https://www.youtube.com/watch?v=bMjlRrWjdT0)

## AI Evals for Cross-Functional Teams

DoorDash’s GenAI platform moved evaluation beyond an engineering harness by giving domain experts stable APIs, task-specific annotation workflows, golden datasets, and a reviewable way to calibrate LLM judges.

### One platform must support different kinds of quality

[0:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=13s)

DoorDash’s GenAI platform team supports product teams with shared infrastructure. Its initial platform choices—an LLM gateway for switching models, an agent gateway that centralizes tool connections, authentication, and agent identity, plus open-weights model hosting—help teams balance three competing forces: accuracy, latency, and cost. The speakers treat those forces as relevant to agents as well as individual models. Evaluation became the fourth pillar because none of the other primitives can tell a product team whether the resulting behavior is actually good enough.

The evaluation target changes by product. A consumer discovery or shopping assistant may need a judgment over an entire session. Personalization work may need human judgment at much greater scale. A multi-agent system may need trajectory-based evaluation, where the sequence of actions matters rather than only the final answer. A common platform therefore cannot assume that every team will grade the same artifact with the same interface.

The first organizational discovery was that engineers were not the only people qualified to define or assess quality. Strategy and operations staff, product managers, and labeling partners carried the domain knowledge needed for the judgments. DoorDash first emphasized UIs so non-engineers could contribute, then added stable APIs so engineers could build without waiting for the platform team. Coding agents pushed the design toward what the speakers call workflow-first: strategy and operations staff and product managers could increasingly run the work themselves, not merely submit input through a fixed screen.

### Quality becomes a division of work

[3:50](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=230s)

This changes evaluation from an engineering harness into a cross-functional operating process. Traces, datasets, and scoring mechanisms are technical objects, but the criteria encoded in them come from people who understand the product and its users. The platform’s job is to carry that knowledge through the system without pretending engineering can infer every team’s quality bar.

The speakers divide the work into four parallel responsibilities:

- **Strategy and operations:** set priorities and decide what quality bar the product should meet.
- **Product:** translate those goals into rubrics and workflows that people can apply.
- **Operations:** run the annotation process and produce human judgments.
- **Engineering:** provide APIs, telemetry, datasets, and automated judges.

The handoffs matter. A broad business goal must become a rubric before annotators can apply it consistently, and those annotations must become structured data before engineering can use them to calibrate or monitor an automated judge.

DoorDash then turns these responsibilities into a recurring loop: capture and inspect behavior, sample it down to a set humans will really review, annotate it with domain expertise, review the annotations, promote suitable examples into a golden dataset, calibrate against that reference, monitor behavior over time, and repeat. Sampling keeps the human workload manageable; the golden dataset gives later measurement an explicit human reference instead of an assumed definition of quality.

Who contributes what to AI quality
The platform combines business priorities, operational judgment, and engineering machinery rather than assigning the whole evaluation process to one function.
- Strategy and operations — Set priorities and the target quality bar.
- Product — Translate requirements into rubrics and workflows.
- Operations — Run annotations and contribute domain-specific judgments.
- Engineering — Supply APIs, telemetry, datasets, and judges.

### Two platform surfaces support one quality loop

[6:02](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=362s)

At the platform level, DoorDash separates telemetry from workflow. The **telemetry layer** holds traces, scores, and observations and exposes them through MCP, an SDK, and APIs. The **workflow layer** is where strategy, operations, and product teams set annotation tasks, review golden datasets, create judges, and calibrate them. One layer records and exposes what the system did; the other organizes the human and automated work used to judge it.

Stable APIs connect those surfaces. Scores and datasets use platform-owned APIs, and the platform’s own UIs sit on the same foundation. This is the important meaning of API-first here: a team does not need to abandon shared data and evaluation primitives merely because it needs a different interface. SDK users and custom UIs still operate over a common access plane.

The lifecycle begins with evidence from production behavior: capture traces and sessions and calculate available scores. Humans then add judgment, context, and domain knowledge. Those labeled examples support judge calibration, after which monitoring produces the next collection of traces. The talk gives no required sampling method, sample size, annotation-agreement threshold, or rule for admitting an example into the golden dataset. Those remain consequential design choices for each implementation.

DoorDash’s continuous evaluation loop
Production behavior becomes manageable human review, then reference data for calibration and another round of monitoring.
- Trace and inspect — Capture sessions, traces, scores, and observations.
- Sample — Reduce the captured behavior to a set humans can inspect.
- Annotate and review — Apply domain knowledge and examine the resulting judgments.
- Create a golden dataset — Promote reviewed examples into reference data.
- Calibrate judges and workflows — Measure or optimize evaluation behavior against the reference set.
- Monitor over time — Observe deployed behavior and begin the next iteration.
- Trace and inspect → Sample (reduce review volume)
- Sample → Annotate and review (send selected cases to humans)
- Annotate and review → Create a golden dataset (review and promote)
- Create a golden dataset → Calibrate judges and workflows (supply a human reference)
- Calibrate judges and workflows → Monitor over time (deploy and observe)
- Monitor over time → Trace and inspect (repeat)

### Operators build the interface the task needs

[8:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=493s)

Annotation is where a large collection of agent sessions becomes an investigation: which cases went well, which failed, and what happened inside each session? The obstacle is interface variety. Image review, manual testing, restaurant-menu grading, and other tasks do not necessarily need the same controls or layout. A central platform team trying to design every task-specific UI would become a queue for the rest of the organization.

DoorDash splits responsibility instead. The platform team maintains the APIs, strategy and operations decide what should be annotated, and annotators perform the labeling. Because the common data operations already exist behind stable APIs, strategy and operations partners can use coding agents to create the task-specific annotation interface themselves. The presenters mention coding tools including Codex and Claude Code, but the architectural point does not depend on a particular agent: generated interfaces call shared platform capabilities rather than recreating the evaluation backend.

The restaurant-menu example is deliberately modest. The interface only needs to present the material and collect the required annotation cleanly. Its value is not technical novelty; it puts workflow design in the hands of operators who understand the task. Image annotation and manual testing can use different UIs while retaining the same underlying platform pattern.

This freedom also creates an unstated governance question. The talk does not explain how generated UI code is reviewed, secured, maintained, or checked for faithful implementation of a rubric. The example establishes that stable APIs make decentralized interface creation possible; it does not establish that every generated annotation tool will be correct merely because it looks clean and completes the task.

A shared backend, with task-specific annotation UIs
The platform team maintains common APIs while operators choose the data and generate an interface suited to the annotation task.
- Platform APIs — Expose shared scores, datasets, and data operations.
- Strategy and operations — Decide what to annotate and shape the workflow.
- Coding agent — Builds a task-specific interface over the stable APIs.
- Annotation UI — Presents menus, images, manual tests, or other task material.
- Annotator — Reviews the material and records a judgment.
- Strategy and operations → Coding agent (describes the needed workflow)
- Platform APIs → Coding agent (supplies stable endpoints)
- Coding agent → Annotation UI (generates)
- Annotation UI → Annotator (collects judgments from)
- Annotator → Platform APIs (writes annotation data through)

### Judge calibration must be self-serve and reviewable

[10:43](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=643s)

Once human annotations have become a golden dataset, a team can use them to improve an LLM judge. The described process starts with a simple judge prompt stating what should be measured. The team runs that judge over traces to establish baseline scores, applies a prompt-optimization loop, and promotes the resulting prompt when the partner team is satisfied. Calibration here changes the judge’s instructions against reference data; the speakers do not describe training model weights.

DoorDash packages this process behind a self-service UI because judge calibration is still new and evolving, even if the basic idea sounds straightforward. A product manager or operator chooses the exposed configuration and runs the loop without repeated back-and-forth with engineering. The example supports multiple model providers, making model selection part of the workflow rather than a platform-team intervention.

Self-service alone would hide too much. Prompt optimization can otherwise feel like a closed box, so the platform shows the original system prompt beside the calibrated prompt. The prompt owner can inspect what changed before adopting it. The presenters report significant improvement in one example, but provide neither the prompts nor a numerical metric, so the stronger supported lesson is about reviewability: optimization should produce a proposed artifact that a domain owner can examine, not merely a higher unexplained score.

The talk also leaves important calibration details open. It does not specify the optimization objective, training and validation split, acceptance threshold, repeated-run variance, or safeguards against overfitting the golden dataset. A side-by-side prompt view can help people understand the proposed change, but visual inspection alone does not prove that a judge will generalize to new production cases.

Who performs that review varies. Strategy and operations own the judge prompt in some teams; a product manager or engineering owns it elsewhere. DoorDash treats this variation as evidence that teams and organizational design are still learning. The platform therefore supports different owners instead of prematurely standardizing one function as the permanent home of judge prompts.

From a baseline judge to a reviewable calibrated prompt
Golden data drives prompt optimization, but the prompt owner still inspects the proposed change before promotion.
- Golden dataset — Human-reviewed reference examples used for calibration.
- Initial judge prompt — A simple statement of what the judge should measure.
- Baseline scores — Run the initial LLM judge over traces.
- Optimization loop — Revise the judge prompt against the reference data.
- Original and calibrated prompts — Show both prompts side by side for inspection.
- Promoted LLM judge — Adopt the calibrated prompt when the partner team is satisfied.
- Initial judge prompt → Baseline scores (run on traces)
- Golden dataset → Optimization loop (supplies reference judgments)
- Baseline scores → Optimization loop (establishes a starting point)
- Optimization loop → Original and calibrated prompts (produces a candidate prompt)
- Original and calibrated prompts → Promoted LLM judge (owner reviews and accepts)

### Lower costs come from shorter dependency loops

[13:23](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=803s)

The platform itself evolved through this work. DoorDash moved from an initial UI emphasis toward API-first and workflow-first operation while reusing existing internal infrastructure. The self-service goal is organizational as much as technical: teams should not need the central platform group to operate every annotation or calibration cycle.

The speakers report a substantial reduction in per-annotation spending at a workload of thousands of rows each week. They also connect self-service annotation and judge calibration to faster iteration. The talk supplies no before-and-after cost, percentage reduction, or measured cycle time, so these remain qualitative operating results rather than quantified performance claims.

The ending returns to the loop rather than the interface: inspect traces and sessions, sample to a size humans can handle, annotate with domain knowledge, build the golden dataset, and use it to calibrate workflows, agents, and LLM judges. Then repeat. The durable mechanism is not any single UI or judge prompt. It is a system that repeatedly turns observed behavior into human judgment, reviewable automation, and another production measurement cycle.

## Key takeaways

- Evaluation is cross-functional because engineering can provide traces, datasets, and judges, but domain experts must define and apply the product’s quality criteria.
- The recurring loop is: trace, sample, annotate, review, create a golden dataset, calibrate, monitor, and repeat.
- Stable APIs let the platform team maintain shared capabilities while operators use coding agents to build annotation UIs suited to menus, images, manual tests, and other tasks.
- Self-service judge calibration remains a review process: prompt owners can compare the original and calibrated prompts before promoting a new judge.
- Prompt ownership can remain flexible while teams learn; DoorDash has seen strategy and operations, product, and engineering each own it in different groups.
- DoorDash reports lower per-annotation costs and faster iteration for thousands of rows each week, but the talk does not quantify the savings or speedup.

## Resources

- [AI Evals for Cross-Functional Teams — official AI Engineer talk page](https://ai.engineer/talks/bMjlRrWjdT0-youtube-de0e357f07f3800aaed9) — The official recording page includes the video, timestamped transcript, chapter navigation, and a reading version of the talk.
- [Nachiket Paranjape on X](https://x.com/nmparanjape) — Speaker profile supplied with the recording, including posts about DoorDash’s evaluation platform and cross-functional AI quality work.

## Related talks

- [The maturity phases of running evals](https://ai.engineer/talks/FB-MLPhL9Ms-maturity-phases-running-evals) — Extends the same progression from expert human judgment to validated automated judges and production-informed evaluation systems.
- [Shipping AI That Works: An Evaluation Framework for PMs](https://ai.engineer/talks/2HNSG990Ew8-shipping-ai-that-works-evaluation-framework-pms) — Develops the product-manager side of shared prompt and evaluation ownership through repeatable experiments and human-labeled data.
- [Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft](https://ai.engineer/talks/3z2uT5aDx_Y-build-evals-that-actually-matter) — Provides a complementary production example involving representative simulations, LLM judges, launch gates, and continual evaluation of multi-turn agents.

## Slides

### Building an AI Evals Platform for Cross Functional Teams

DoorDash’s GenAI platform team shares how its eval work grew from an engineering task into a cross-functional effort, with API-first evals, annotation, and calibration as a shared loop.

[0:19](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=19s) [Title slide introducing API-first evaluations, annotation, and calibration as a shared loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/7d7fc03da7f29e0916b0d19b19ce3d8b5b7b5a63afffb18bf81f5e0f77b9d57e.webp)

Building an AI Evals Platform for Cross Functional Teams

Building an AI Evals Platform for Cross Functional Teams

API-first evals, annotation, and calibration as a shared loop

• Nachiket Paranjape  & Swaroop Chitlur  X  • DoorDash GenAI Platform  • AI Engineer World's Fair 2026

AI Evals Platform
1 / 27

### Help every team get the trade-offs right

DoorDash’s GenAI platform helps product teams balance accuracy, latency, and cost. The triangle places quality at the center; the speaker says the trade-off applies to both models and agents.

[0:46](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=46s) [Triangle showing accuracy, latency, and cost around quality.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b65497eb592ab11255652349a9c59f64b0a5b0571d9490accd536a9a33adb1e.webp)

Building an AI Evals Platform for Cross Functional Teams

OUR PURPOSE
Help every team get the trade-offs right

Accuracy
Quality
Latency
Cost

GenAI Platform
2 / 27

### The GenAI Platform: four systems

DoorDash pairs gateways for models and agents with open-weight model hosting and evals. The gateways handle 600M+ LLM API calls per week and 300K+ tool calls daily.

[1:21](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=81s) [Four cards summarize the LLM Gateway, Agent Gateway, Slingshot, and evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/b43be6d99a7ae4efa9db9c69daa33a94a73d4ceff858fbaf523665605500d67c.webp)

Building an AI Evals Platform for Cross Functional Teams

WHAT WE BUILD

The GenAI Platform: four systems, every AI product at DoorDash

600M+
LLM API calls/week — LLM Gateway

300K+
daily tool calls, 50+ MCP servers — Agent Gateway

Millions
in savings by using open-weight models — Slingshot

Evals
tracing, annotation, calibration — today's focus

For a deeper dive: QCon Boston 2026 · boston.qcon.ai/presentation/boston2026/building-genai-platform-doordash

3 / 27

### Every AI product needed evals

DoorDash teams needed different evals: session-level judgments for shopping conversations, human-aligned automated judges for personalization, and trajectory-level evals and annotation at scale for multi-agent systems.

[2:09](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=129s) [Three cards describe evaluation needs for conversational AI, personalization ML, and agentic operational systems.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c7adab6bcefc1cfada571945413db525843884e6f889b7915776c233455431d.webp)

Building an AI Evals Platform for Cross Functional Teams

WHO USES EVALS

Every AI product needed evals. Each one looked different.

Multi-turn conversational AI
• Team Consumer discovery & shopping assistant, millions of users
• Need Session-level quality judgment across multi-turn transcripts

Personalization ML
• Team Memory generation, catalog tagging, navigation AI
• Need Production-grade automated judges that align with humans

Agentic operational systems
• Team Multi-agent systems, millions of automated decisions/day
• Need Trajectory-level evals and S&O annotation at scale

DoorDash Assistant deep dive: careersatdoordash.com/blog/building-doordash-assistant-an-engineering-overview

4 / 27

### Quality judgment belongs to the people who know the domain

DoorDash started with annotation UIs for domain experts, added APIs so engineers could build independently, then used coding agents to help PMs and S&O teams run workflows.

[2:48](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=168s) [The platform evolved through UI-first, API-first, and workflow-first approaches to support domain experts.](https://assets.ai.engineer/sites/aiecode2025/sha256/3c30a2cfc9958f97772346a8e72f9545b1ddad6d5101d8f8c0ab240c906d7e2b.webp)

Building an AI Evals Platform for Cross Functional Teams

HOW THE PLATFORM EVOLVED

Quality judgment belongs to the people who know the domain

Domain expertise drives quality — so evals had to reach PMs, S&O teams, and external labeling partners, not just engineers.

UI-first
Annotation interfaces non-engineers could use; moved teams off spreadsheets

API-first
Stable platform contracts let engineering teams build custom workflows independently

Workflow-first
Coding agents for ad hoc workflow execution and custom annotation UIs built on platform APIs

GenAI Platform
5 / 27

### Evals as cross-functional quality infrastructure

Traces, datasets, and automated scoring provide a foundation for strategy, operations, product, and engineering to apply domain judgment together and improve AI quality.

[3:44](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=224s) [The thesis states that traces, datasets, and automated scoring support quality judgment shared across functions.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c575a0426a706b63f7e5224f3769a15356610ee2180a80b71cc295753f4b6d5.webp)

Building an AI Evals Platform for Cross Functional Teams

THE THESIS

Evals is not just an engineering harness. It is cross-functional quality infrastructure.

Traces, datasets, and automated scoring are the foundation. Quality only becomes real when strategy, ops, product, and engineering can operationalize judgment together.

The thesis
6 / 27

### AI quality is a cross-functional operating loop

Strategy sets quality bars, product builds rubrics, operations runs review, and engineering supplies the tooling. Shared evidence connects human judgment to calibrated automation.

[4:31](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=271s) [Slide lists the roles of strategy, product, operations, engineering, and shared evidence in AI quality.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca2956256141fdeaf1e83a955b5253c210b87991240a409007e61e4dd08cb199.webp)

Building an AI Evals Platform for Cross Functional Teams

WHY THIS MATTERS NOW

AI quality is a cross-functional operating loop

Strategy, operations, product, and engineering each own a different part of the same quality system.

• Strategy sets priorities and quality bars
• Product translates experience into rubrics and workflows
• Operations runs annotation and review at scale
• Engineering provides APIs, telemetry, datasets, and judges
• Shared evidence turns human judgment into calibrated automation

Why this matters now
7 / 27

### Eight steps, one continuous loop

Production traces are sampled for expert annotation and review. Locked examples become ground truth for calibrating judges; monitoring tracks drift and feeds new examples back into the loop.

[5:19](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=319s) [Four cards show trace and sample, annotate and review, create golden sets and calibrate, and monitor and repeat.](https://assets.ai.engineer/sites/aiecode2025/sha256/334c7d7d5a3fd15974e065214ff9f115af3ea380426abed93ecceee634d157b3.webp)

Building an AI Evals Platform for Cross Functional Teams

THE EVALS WORKFLOW

Eight steps, one continuous loop

The whole evals workflow in one picture — we will walk each phase in the chapters that follow.

Trace → Sample
Capture production behavior; select spans worth judging

Annotate → Review
Human experts label examples; review surfaces disagreement

Create golden sets → Calibrate
Locked rows become ground truth; judges calibrated against them

Monitor → Repeat
Track drift; feed new examples back into the loop

The evals loop
8 / 27

### One platform surface, two connected capabilities

The telemetry layer stores traces and scores and provides access to them. The workflow layer supports annotation, review, golden datasets, and judge calibration for product and operations teams.

[6:10](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=370s) [Two cards describe the telemetry layer and the workflow layer.](https://assets.ai.engineer/sites/aiecode2025/sha256/f038e5bb90321cffc1544931f7a20ef8b9dfbd4c74c6a25eb179286370d5e06d.webp)

Building an AI Evals Platform for Cross Functional Teams

THE PLATFORM MODEL

One platform surface, two connected capabilities

Telemetry layer
Traces, scores, datasets, SQL access, SDK tracing, evaluator APIs

Workflow layer
Annotation tasks, review, golden datasets, judge evaluation, GEPA calibration

Two complementary planes
9 / 27

### Chapter 1: Trace and sample

Tracing captures production behavior and sampling selects spans worth judging. The loop then moves through human review, golden-set calibration, and monitoring, which feeds new examples back in.

[6:46](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=406s) [The evaluation workflow highlights trace and sample as the capture foundation; the remaining stages are faded.](https://assets.ai.engineer/sites/aiecode2025/sha256/85d1a8493559a122e16ff2361aa69873cfdd0e58a288a1b9271d6f14f9e8a4b9.webp)

Building an AI Evals Platform for Cross Functional Teams

CHAPTER 1 · WHERE WE ARE

Eight steps, one continuous loop

Chapter 1 — the capture foundation: production traces, sampling, and the API that makes them addressable.

Trace → Sample
Capture production behavior; select spans worth judging

Annotate → Review
Human experts label examples; review surfaces disagreement

Create golden sets → Calibrate
Locked rows become ground truth; judges calibrated against them

Monitor → Repeat
Track drift; feed new examples back into the loop

The evals loop
10 / 27

### API-first platform foundation

Stable APIs power the platform’s shared model, with UIs built on top. SQL access and SDK tracing let teams bring their own analysis.

[7:08](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=428s) [Chapter 1 slide describes APIs as a stable contract, a shared platform model, and SQL and SDK access.](https://assets.ai.engineer/sites/aiecode2025/sha256/4b02ce054a82fecffcb7aa098bb71a0cbd78895be09aed28d6437c75a67e5817.webp)

Building an AI Evals Platform for Cross Functional Teams

CHAPTER 1
1
API-first platform foundation

• APIs are the stable contract; UIs are clients on top
• Traces, scores, datasets, tasks, reviews, and judges share one platform model
• SQL access and SDK tracing support teams that bring their own analysis

Chapter 1
11 / 27

### From production signal to calibrated judgment

The platform moves from captured traces and scores through human judgment to calibrated judges. One API and shared object model support the workflow.

[7:42](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=462s) [Diagram connects capture to human judgment and calibrated judges, supported by one API and a shared object model.](https://assets.ai.engineer/sites/aiecode2025/sha256/faaefd743bffa0dfc99356202dbffddbaaa449d166c56fd8e6529d7052205f11.webp)

Building an AI Evals Platform for Cross Functional Teams

CHAPTER 1 · ARCHITECTURE

One platform, from production signal to calibrated judgment

Capture
traces · observations · scores
→
Human judgment
annotation · review · golden datasets
→
Calibrated judges
LLM · agent · code · GEPA

One API & shared object model
shared datasets · SDK + SQL access · async ingestion

Evals platform
12 / 27

### Eight steps, one continuous loop

After traces are sampled, human experts label examples and review disagreements. Locked rows become ground truth for calibrating judges, while monitoring feeds new examples back into the loop.

[8:11](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=491s) [Four stages of the evaluation loop, with annotate and review highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/9913de834f9d5ddcfdd8a93fcaf6796ead91e71bff548ce1c9db35238cf648cb.webp)

Building an AI Evals Platform for Cross Functional Teams

CHAPTER 2 · WHERE WE ARE

Eight steps, one continuous loop

Chapter 2 — human judgment: annotation and review turn captured signal into trusted labels.

Trace → Sample
Capture production behavior; select spans worth judging

Annotate → Review
Human experts label examples; review surfaces disagreement

Create golden sets → Calibrate
Locked rows become ground truth; judges calibrated against them

Monitor → Repeat
Track drift; feed new examples back into the loop

The evals loop
13 / 27

### Human judgment at scale

Teams need different context to judge quality. The platform team took an API-first approach, enabling teams to use coding agents to build their own annotation interfaces.

[8:37](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=517s) [Chapter 2 introduces annotation workflows, team-specific context, and specialized interfaces.](https://assets.ai.engineer/sites/aiecode2025/sha256/35d7971fa0a85e97e1bd8f6c8edf2a561d96afa433d8e61b0bf55799789ae518.webp)

Human judgment at scale

Building an AI Evals Platform for Cross Functional Teams

CHAPTER 2
2

• Annotation is a workflow, not a spreadsheet
• Different teams need different context to judge quality
• Shared contracts, specialized interfaces

Chapter 2
14 / 27

### Three teams, three interfaces, one platform

Shared APIs support specialized annotation interfaces: memory-block review exposes disagreements, image annotation uses labeling tools, and manual testing gives reviewers journey context and rubrics.

[9:36](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=576s) [Three cards describe memory-block annotation, image annotation, and manual testing workflows.](https://assets.ai.engineer/sites/aiecode2025/sha256/3b6e2bc4f2aeeb766007616cc6136e344ab38094debeb09ad77169eaa5176247.webp)

Building an AI Evals Platform for Cross Functional Teams

CHAPTER 2 · EXAMPLES

Three teams, three interfaces, one platform

Memory-block annotation
• Structured multi-step review
• Context windows with labeled memory blocks
• Disagreement surfaced per block

Image annotation
• Bounding boxes and polygon tools
• Label hierarchies and attribute forms
• Shared dataset versioning and export

JETS / manual testing
• Ground-truth manual-testing surfaces
• Full journey context for reviewers
• Golden row creation with structured rubrics

Specialized workflows
15 / 27

### What annotators actually see

Operators can build annotation UIs on platform APIs. This restaurant-menu example lets experts assign a preference tag to each dish and add optional notes.

[9:59](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=599s) [A dish annotation table shows preference tags and optional notes for each row.](https://assets.ai.engineer/sites/aiecode2025/sha256/314ca64f0050413609c09625dfb10b304dd21f853b17317cc481226da727a661.webp)

Building an AI Evals Platform for Cross Functional Teams
CHAPTER 2 · ANNOTATION UI
What annotators actually see

Dishes
Assign a preference tag to each dish
# | Dish | Tag | Orders | Notes

1. McDonald's - Hash Browns Hash Browns and Sides - Hash Browns and Sides
Deliciously tasty and perfectly crispy. Features shredded potato hash brown patties that are prepare... customizations: Add Ketchup Packet
breakfast
Orders: 1

2. Starbucks - Coffee Traveler - Pike Place® Roast Hot Coffee - Hot Coffee
A convenient carrier filled with 96 fl oz of our featured brewed medium roast coffee (equivalent of ...
breakfast
Orders: 1

3. McDonald's - Sausage & Egg McMuffin® Most Popular - Most Popular
Start your day off with a McDonald's Sausage McMuffin with Egg breakfast sandwich. Our Sausage Egg M...
breakfast
Orders: 1

4. DX Braised Chicken Rice - Yellow Braised Pork Ribs [illegible]
Byitself [illegible] - Byitself [illegible]
Sizzling Pot + Rice [illegible] customizations: No Spicy [illegible]
dinner
Orders: 1

5. Pokeworks - Poke Bowl - Large (3 Proteins) Poke Your Way - Poke Your Way
Mix up to 3 proteins with your choice of mix-ins, toppings, flavor, and base. customizations:
Cabbage, Cilantro, Cucumber, Edamame, [illegible], Garlic Crisps, Green Onion, Half & Half, Half Brown Rice, Half Ramen Noodles, Hijiki Seaweed, Mandarin Orange, Mango, Medium Flavor, Onion Crisps, Pickled Ginger, Pineapple, Pokeworks Classic, Salmon, Sesame Seeds, Shredded Nori, Shrimp, Soft Tofu, Spicy Furikake, Surimi Salad, Sweet Corn, Sweet Shoyu (V), Wasabi, Wonton Crisps
dinner
Orders: 1

Tag menu:
— select tag —
N/A
like
dislike
neutral

Domain experts assign a preference tag to each row — like, dislike, neutral, or N/A — with optional notes. A task-specific UI built on platform APIs turns judgment into structured, reviewable labels.

GenAI Platform
16 / 27

### Eight steps, one continuous loop

Locked rows become ground truth for calibrating judges with GEPA. This stage follows expert annotation and review; monitoring feeds new examples back into the loop.

[10:27](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=627s) [Four stages of the evaluation loop, with golden sets and calibration highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/f05cc5c857cb867333d7eba723670ec01aa957c15b1352210fead7740d09ce51.webp)

Building an AI Evals Platform for Cross Functional Teams

CHAPTER 3 · WHERE WE ARE

Eight steps, one continuous loop

Chapter 3 — golden sets and calibration: locked ground truth, and judges calibrated against it with GEPA.

Trace → Sample
Capture production behavior; select spans worth judging

Annotate → Review
Human experts label examples; review surfaces disagreement

Create golden sets → Calibrate
Locked rows become ground truth; judges calibrated against them

Monitor → Repeat
Track drift; feed new examples back into the loop

The evals loop
17 / 27

### Calibrating automation

Golden datasets help improve judge prompts. Calibration makes alignment with expert judgment reviewable, while GEPA makes prompt optimization auditable.

[10:41](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=641s) [Chapter 3 introduces expert-aligned judges, reviewable calibration, and auditable prompt optimization.](https://assets.ai.engineer/sites/aiecode2025/sha256/85067b41877f6383db502ef7833bb39c76dd4f3ee8e62240b5ef8dd635b98d3b.webp)

Calibrating automation

Building an AI Evals Platform for Cross Functional Teams

CHAPTER 3
3

• Judges are only useful if they reproduce expert judgment
• Calibration makes alignment visible and reviewable
• GEPA turns prompt optimization into an auditable process

Chapter 3
18 / 27

### Calibration as an iterative, trackable process

GEPA explores candidate prompts after an initial baseline evaluation, tracking iterations and snapshots. The owner reviews before/after evidence and saves the optimized prompt.

[10:44](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=644s) [Four stages of GEPA calibration: start, baseline, optimize, and accept.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab9d7d82dc3c47e8a5b6cb3e30ff26b7610d9d3739bb5846c6f59ecdb6844349.webp)

Calibration as an iterative, trackable process

Building an AI Evals Platform for Cross Functional Teams
CHAPTER 3 · GEPA

GEPA explores optimized prompt candidates and makes progress visible at every step.

Start
Judge prompt + annotated examples as the baseline

Baseline
Run initial evaluation; record before score

Optimize
GEPA explores candidate prompts; track iterations and snapshots

Accept
Owner reviews before/after evidence; saves the optimized prompt

GEPA calibration workflow
19 / 27

### Configuring a calibration run

A self-service screen lets product managers and operators select a judge and golden dataset, adjust calibration settings, and run the loop themselves. The example uses Gemini 2.5 Flash.

[11:22](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=682s) [A calibration setup screen with judge, dataset, model, budget, and strategy settings.](https://assets.ai.engineer/sites/aiecode2025/sha256/0682311e94cc9bd105c88b7af4826b37f259660237df133ccf267cb8c28b9ded.webp)

Building an AI Evals Platform for Cross Functional Teams

GEPA IN THE PRODUCT
Configuring a calibration run

Auto-Calibrate Judge
Automatically optimize the judge’s system prompt

Select Judge
Select a judge...
Human Annotation/Golden Dataset
Select a dataset...

Advanced Settings

Model Settings
Task Model: Gemini 2.5 Flash
Reflection Model: Gemini 2.5 Flash

Reward Metric
Metric: Alignment (accuracy) — default

Budget Settings
Max Metric Calls: 60
Max Full Evals: Auto
Batch Size: 3

Train / Validation Split
Train: 70% of examples · Validation: 30% of examples
70% / 30%

Strategy Settings
Selection Strategy: Pareto
Use Merge: Yes
Max Merge Invocations: 5

Advanced Parameters
Temperature: Default
Top-K: Default
Candidate Pool: Default
Seed: 0

Pick the judge and a human-annotated golden dataset, then tune models, budget, and search strategy — sensible defaults mean most teams just press go.

GenAI Platform
20 / 27

### Eight steps, one continuous loop: monitor and repeat

After teams sample production behavior, review labels, and calibrate judges against golden sets, they monitor drift and feed new examples back into the loop.

[12:58](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=778s) [Four stages of the evaluation loop, with monitor and repeat highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/649fed00cf66d0c1e9e61859b0ee645caf1796a42ca9859e6720a816d5b7e978.webp)

Building an AI Evals Platform for Cross Functional Teams

CHAPTER 4 · WHERE WE ARE

Eight steps, one continuous loop
Chapter 4 — monitor and repeat: track drift, feed new examples back, and keep the loop running.

Trace → Sample
Capture production behavior; select spans worth judging

Annotate → Review
Human experts label examples; review surfaces disagreement

Create golden sets → Calibrate
Locked rows become ground truth; judges calibrated against them

Monitor → Repeat
Track drift; feed new examples back into the loop

The evals loop
22 / 27

### The quality loop in practice

DoorDash started with UIs, then shifted toward API-first workflows and sought to reuse existing infrastructure as it built the quality loop.

[13:34](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=814s) [Chapter 4 introduces end-to-end workflows, API-first velocity, and reusable infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/e62dc63cffaf71873c7298c3757facbb78da15c3c8c5052df35bf2ef0ca354fd.webp)

The quality loop in practice

Building an AI Evals Platform for Cross Functional Teams

CHAPTER 4
4

• The platform wins when the loop runs end to end
• API-first workflows increase team velocity
• Reusable infrastructure compounds into long-run cost savings

Chapter 4
23 / 27

### The loop compounded into measurable savings

Moving from spreadsheets to the Evals UI reduced observed per-task annotation cost by 40%. Teams reached annotated data with fewer handoffs, while reusable workflows replaced one-off scripts over time.

[13:58](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=838s) [Impact cards describe a 40% reduction in per-task annotation cost, faster loops, and compounding savings.](https://assets.ai.engineer/sites/aiecode2025/sha256/e6b9942b30ba525c8110b5ba4cc4be91f62ccd0bff122ec722fccfbb6c48bc02.webp)

The loop compounded into measurable savings

Building an AI Evals Platform for Cross Functional Teams
CHAPTER 4 · IMPACT

40%
observed reduction in per-task annotation cost after moving from spreadsheets to the Evals UI

Faster loops
teams move from idea to annotated data with fewer handoffs

Compounding savings
reusable workflows replace one-off scripts over time

Impact
24 / 27

### Eight steps, one continuous loop

Teams sample production traces, review expert annotations, and calibrate judges against golden sets. Monitoring feeds new examples back into the loop.

[14:52](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=892s) [Four cards describe tracing and sampling, annotation and review, golden sets and calibration, and monitoring and repetition.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e27f9e8b8929ab911f430cafcf42608dc7eff26aaf648a2c93e1bccc1894d1d.webp)

Eight steps, one continuous loop

Building an AI Evals Platform for Cross Functional Teams
CHAPTER 4 · THE LOOP

Trace → Sample
Capture production behavior; select spans worth judging

Annotate → Review
Human experts label examples; review surfaces disagreement

Create golden sets → Calibrate
Locked rows become ground truth; judges calibrated against them

Monitor → Repeat
Track drift; feed new examples back into the loop

The quality loop
25 / 27

### Q&A

The speakers invite questions on Evals Service architecture, annotation workflows, GEPA calibration, custom annotation UIs and cross-functional collaboration, now or after the talk.

[15:39](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=939s) [Q&A slide lists topics for further discussion, including architecture, annotation workflows, calibration and collaboration.](https://assets.ai.engineer/sites/aiecode2025/sha256/70c65c65680bb814d06d600580d5041fa276050f090dcde92189e6d194bc561e.webp)

Q&A

Building an AI Evals Platform for Cross Functional Teams

Happy to go deeper on Evals Service architecture · annotation workflows · GEPA calibration · custom annotation UIs · cross-functional collaboration patterns.

• Nachiket Paranjape & Swaroop Chitlur
• AI Engineer World's Fair 2026

Q&A
26 / 27

### AI Engineer World's Fair

[16:04](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=964s) [AI Engineer World's Fair title card in white on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/1bc272163caa5afc69272594ebd951b340765f51cd80a58ec268957bba668e3e.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:01](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=1s) [music]

[0:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=13s) Good afternoon everyone. Thanks for uh

[0:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=16s) coming for a post lunch uh talk. Always

[0:19](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=19s) appreciate that. Um my name is Farup and

[0:22](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=22s) here's my teammate Nachiket. Uh we are

[0:25](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=25s) uh here behalf of the Door Dash Genai

[0:28](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=28s) platform team. Um and we kind of wanted

[0:31](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=31s) to share our eval journey. Uh it started

[0:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=35s) as uh uh you know eval is another

[0:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=38s) engineering thing but then it slowly we

[0:40](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=40s) realized it evolved into a cross

[0:42](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=42s) functional effort and we kind of want to

[0:45](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=45s) share our story here. So what is this

[0:48](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=48s) team? This team is a gen platform team.

[0:50](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=50s) Uh we are a horizontal team that helps

[0:52](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=52s) all other product teams. So product

[0:54](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=54s) teams at Door Dash build on top of the

[0:57](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=57s) infrastructure and the primitives that

[0:58](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=58s) we provide. Um and we see our uh USP and

[1:03](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=63s) the value that we provide is that we

[1:05](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=65s) help product teams balance these three

[1:07](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=67s) forces which is accuracy, latency and

[1:10](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=70s) cost. Um initially we applied this in

[1:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=73s) terms of models but if you think about

[1:14](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=74s) it it also applies to agents. Um and the

[1:18](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=78s) way we achieve this is we have uh

[1:20](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=80s) primitives and building blocks. Um so

[1:23](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=83s) for example we have an LLM gateway where

[1:25](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=85s) you can easily switch between different

[1:27](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=87s) models uh and try the latest and

[1:29](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=89s) greatest. Uh we have an agent gateway

[1:32](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=92s) where you can connect to tools uh and

[1:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=95s) other agents uh and we help solve

[1:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=98s) authentication uh agent identity and

[1:40](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=100s) other things in a central place uh which

[1:43](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=103s) our security team can bless. Um

[1:45](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=105s) similarly we pair the LLM gateway with

[1:48](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=108s) open weights models hosting. Uh, of

[1:50](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=110s) course cost is a number one concern

[1:52](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=112s) these days. Uh, and we uh kind of

[1:55](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=115s) invested in open weights models uh and

[1:57](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=117s) have seen significant impact uh already.

[2:00](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=120s) Um, and maybe we'll talk about that in a

[2:03](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=123s) future conference. Uh, the fourth pillar

[2:05](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=125s) is eval and that's the part that we

[2:07](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=127s) would want to share today. Um

[2:11](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=131s) when we started talking to product teams

[2:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=133s) internally at Door Dash uh there were

[2:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=136s) varying uh distinct needs across teams.

[2:20](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=140s) We had a consumer discovery and shopping

[2:22](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=142s) assistant team. Uh for those who

[2:24](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=144s) attended Ragago talk earlier today uh

[2:26](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=146s) you will uh see the need for session

[2:29](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=149s) level quality judgments. um uh then the

[2:32](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=152s) personalization ML then you needed uh a

[2:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=155s) way to scale up human judgment and with

[2:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=158s) multi- aent systems we needed trajectory

[2:40](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=160s) based evals now the question is how do

[2:43](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=163s) you cater to all these different needs

[2:47](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=167s) under a common platform

[2:50](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=170s) and uh as we spoke to these teams we

[2:52](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=172s) realized like um we needed to empower

[2:56](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=176s) the people who are the domain experts

[2:58](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=178s) and in our case that was strategy and

[3:00](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=180s) operations folks it as product managers

[3:02](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=182s) uh it was even labeling partners uh and

[3:05](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=185s) not only engineers so we kind of started

[3:07](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=187s) with like okay we have to be UI first

[3:09](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=189s) and this was the guidance we had from

[3:11](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=191s) Andy Fang our co-founder as well um so

[3:14](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=194s) we had UIs for non-engineers to

[3:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=196s) contribute uh then we kind of evolved to

[3:19](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=199s) also being API first so that engineers

[3:22](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=202s) can also build and not be blocked on the

[3:24](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=204s) central platform and they can build

[3:26](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=206s) their own uh uh uh systems uh and Then

[3:30](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=210s) of course with the coding agents now we

[3:33](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=213s) have become workflow first where we kind

[3:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=215s) of empower SNO and PMs to also being

[3:37](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=217s) able to uh navigate uh the platform and

[3:41](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=221s) uh run operations as well. Um so with

[3:45](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=225s) that context I'll hand it off to Nachig

[3:47](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=227s) to talk about uh how we went about

[3:49](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=229s) delivering this. Cool. Thanks Harup. Um

[3:52](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=232s) and thanks everyone for joining us. I

[3:54](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=234s) know France is playing right now and I

[3:56](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=236s) promise you this will be better than

[3:58](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=238s) that. I'm kidding. Um so as Faroo was

[4:01](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=241s) saying uh Evals is not just an

[4:03](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=243s) engineering harness it is a cross

[4:05](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=245s) functional effort across different

[4:07](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=247s) pillars across different uh teams uh

[4:10](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=250s) that actually helps us add all the

[4:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=253s) domain specific knowledge into our uh

[4:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=256s) into the quality of the AI itself. So

[4:18](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=258s) from your traces to your data sets uh

[4:21](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=261s) from you know scoring mechanisms uh this

[4:24](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=264s) is all basically a team sport. we all

[4:27](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=267s) have to play uh and help improve the

[4:30](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=270s) quality of AI.

[4:33](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=273s) So going a little bit deeper into the

[4:36](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=276s) same aspect uh we have different uh

[4:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=278s) teams uh at Door Dash who help us

[4:40](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=280s) actually improve the quality of AI. So

[4:42](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=282s) you're going to have your strategy and

[4:43](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=283s) operations folks who are going to set

[4:45](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=285s) priorities, set the quality bar that you

[4:47](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=287s) want to aim for. You're going to have

[4:48](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=288s) your product people who are going to

[4:50](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=290s) translate uh these requirements into

[4:53](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=293s) rubrics workflows. You're going to have

[4:55](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=295s) your operations teams running uh

[4:57](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=297s) annotations. You're going to have your

[4:58](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=298s) engineering teams like us uh providing

[5:01](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=301s) APIs, telemetry, data sets, judges, all

[5:04](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=304s) you know the the cool things. Um and

[5:08](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=308s) combining all these together is is what

[5:11](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=311s) a recipe is for actually making sure

[5:14](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=314s) that you are shipping quality AI

[5:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=316s) products through an eval platform.

[5:19](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=319s) So we've tried to boil this down uh into

[5:22](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=322s) sort of you know like a a continuous

[5:24](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=324s) iteration loop. Uh so right from tracing

[5:28](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=328s) uh you know having a tracing solution

[5:30](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=330s) viewing your sessions your traces to

[5:33](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=333s) sampling them down you uh you know to a

[5:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=335s) very small uh set that you actually want

[5:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=338s) to look at uh annotating these with the

[5:40](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=340s) domain specific expertise that you bring

[5:42](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=342s) in with the different teams I mentioned

[5:45](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=345s) reviewing those uh then creating those

[5:47](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=347s) golden data sets which are going to be

[5:50](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=350s) you you know your uh golden data sets

[5:53](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=353s) and that that you want to measure or

[5:54](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=354s) calibrate against uh and then of course

[5:56](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=356s) like you know monitoring this over a

[5:58](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=358s) period of time and then you know rinse

[6:00](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=360s) and repeat uh go through the whole loop

[6:02](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=362s) again. So this is in our experience has

[6:05](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=365s) been you know like a good sort of

[6:06](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=366s) continuous loop uh for you know shipping

[6:09](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=369s) quality AI

[6:12](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=372s) at the plat on on the platform level uh

[6:14](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=374s) we we have two surfaces uh so we have

[6:17](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=377s) the telemetry layer uh where we have all

[6:19](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=379s) our traces our scores uh observations

[6:22](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=382s) that is also sort of the plane where

[6:25](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=385s) users are able to access these traces

[6:27](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=387s) using an MCP using an SDK uh using our

[6:30](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=390s) APIs and then we have the workflow This

[6:33](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=393s) is where a lot of our strat ops, our

[6:34](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=394s) product teams operate on the platform.

[6:37](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=397s) So this is where all the annotation

[6:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=398s) tasks are set. Uh you know this is where

[6:41](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=401s) they review their golden data sets, uh

[6:43](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=403s) create their judges, calibrate their

[6:45](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=405s) judges and so on.

[6:47](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=407s) So maybe today we'll go through you know

[6:50](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=410s) these sort of four different uh modules

[6:53](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=413s) or pillars of our platform uh step by

[6:55](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=415s) step. Uh so again first one uh tracing

[6:58](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=418s) and sampling uh which is actually

[7:00](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=420s) capturing what your agents what your

[7:03](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=423s) LLMs are actually uh you know outputting

[7:06](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=426s) for the lack of better words uh and

[7:08](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=428s) actually viewing those.

[7:10](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=430s) Now in order to also power this uh whole

[7:14](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=434s) platform we have I think as far

[7:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=436s) mentioned we have gone in an API first

[7:18](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=438s) uh approach. Uh what that has allowed us

[7:20](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=440s) to do is have these table APIs that

[7:23](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=443s) actually uh you know and then you know

[7:25](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=445s) build UIs uh on top of that. Uh so all

[7:29](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=449s) our scores our data sets uh these are

[7:31](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=451s) all powered by very stable APIs uh that

[7:34](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=454s) our team owns. Uh so all your API access

[7:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=458s) uh including you know like an SDK access

[7:40](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=460s) is basically powered by this single uh

[7:42](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=462s) plane.

[7:44](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=464s) Um again

[7:47](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=467s) going back uh and you know like just

[7:49](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=469s) refreshing your memory. Uh step one

[7:51](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=471s) capture your traces uh capture your

[7:54](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=474s) sessions uh measure your scores. Uh then

[7:57](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=477s) you want to start uh almost you know

[7:59](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=479s) like adding all your judgment your

[8:02](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=482s) context your domain knowledge uh and

[8:05](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=485s) then calibrating your judges is what we

[8:07](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=487s) have seen as the whole uh life cycle.

[8:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=493s) Step two is on the annotation side. Uh

[8:15](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=495s) so you obviously are capturing a lot of

[8:17](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=497s) your uh agentic behavior, your sessions,

[8:19](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=499s) your traces, but you actually want to

[8:21](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=501s) see what are some places where things

[8:24](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=504s) went well and what are some places where

[8:26](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=506s) things did not go well. This is where

[8:28](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=508s) you can actually titrate your your you

[8:31](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=511s) know and actually look in inside what's

[8:33](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=513s) actually happening uh at the session

[8:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=515s) level and annotate these data sets. Um

[8:39](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=519s) and as Surup mentioned, we have a lot of

[8:41](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=521s) use cases. we have we we talked to

[8:43](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=523s) multiple different teams who have uh

[8:46](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=526s) various uh ways of annotating uh their

[8:49](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=529s) data sets uh and it's it's almost hard

[8:52](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=532s) for a platform team to you know build

[8:55](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=535s) like a UI specific uh for each use case

[8:58](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=538s) uh and and you know to give you an

[8:59](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=539s) example uh it's usually going to be an

[9:02](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=542s) annotator who's going to annotate these

[9:05](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=545s) data sets so the platform team is you

[9:07](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=547s) know in charge of the APIs we have a

[9:09](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=549s) strategy and of person who's actually

[9:12](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=552s) deciding what to annotate and then you

[9:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=553s) have an annotator who's actually going

[9:15](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=555s) to annotate uh your data set. So we took

[9:18](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=558s) this approach uh everybody uh has uh you

[9:21](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=561s) know access to coding agents uh and we

[9:23](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=563s) actually doubled down on that API first

[9:25](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=565s) approach. So because we had these APIs

[9:27](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=567s) we were actually uh able to enable our

[9:30](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=570s) statops teams to use something like a

[9:32](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=572s) codeex or a claw code and v code their

[9:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=575s) own annotation UIs. Uh so we had

[9:39](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=579s) different use cases. Uh I think we had a

[9:41](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=581s) talk from Ragav before. Uh we had image

[9:44](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=584s) annotation use cases. We had some uh you

[9:46](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=586s) know manual testing use cases. What

[9:49](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=589s) stood out to us was the underlying

[9:51](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=591s) patterns were similar. So if we are are

[9:54](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=594s) API first uh we can actually enable our

[9:57](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=597s) our our partners to simply v code these

[9:59](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=599s) UIs for annotation. So it's it's like a

[10:02](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=602s) very simple example then you know of of

[10:05](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=605s) a vibe coded UI looks pretty clean does

[10:07](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=607s) the job uh and you get you know the

[10:10](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=610s) annotation that you eat this is

[10:11](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=611s) basically like a menu from a restaurant

[10:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=613s) uh it's it's you know nothing crazy uh

[10:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=616s) but the point I want to make here is

[10:18](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=618s) that what helped us was to give this

[10:22](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=622s) workflow in the hands of the operators

[10:24](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=624s) so that they can actually build their

[10:26](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=626s) own vcoded annotation UIs. Uh so moving

[10:29](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=629s) on once you have these annotation UIs

[10:31](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=631s) you obviously want to you know calibrate

[10:33](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=633s) your your your judge prompts you

[10:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=635s) obviously have some LM as a judge uh

[10:37](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=637s) metric that you're tracking you want to

[10:39](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=639s) now start improving that with these

[10:41](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=641s) golden data sets

[10:43](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=643s) u in order to do that uh you know we

[10:46](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=646s) have a pretty simple process uh you

[10:48](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=648s) you're going to start with you know some

[10:49](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=649s) judge prompt take a look at you know

[10:51](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=651s) what exactly do you want to measure from

[10:53](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=653s) the output uh g you know have have us

[10:56](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=656s) have something simple you're going to

[10:58](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=658s) have your baseline scores uh where

[11:00](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=660s) you're going to simply run those LLM

[11:02](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=662s) judges on your traces and then you're

[11:04](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=664s) going to have that optimization loop. Uh

[11:06](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=666s) so we use uh the JPEA library which is a

[11:09](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=669s) pretty commonly used library out there

[11:11](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=671s) for prompt optimization. Uh and once you

[11:14](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=674s) know the the iteration loop is complete

[11:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=676s) uh our partner teams are happy they're

[11:18](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=678s) going to then elevate that judge prompt

[11:21](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=681s) as their LLM as a judge. Now even while

[11:24](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=684s) doing that uh LLM as a judge as a

[11:26](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=686s) concept the whole prompt calibration

[11:28](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=688s) concept might be uh straightforward to a

[11:31](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=691s) lot of folks but it is still like a

[11:32](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=692s) pretty new and evolving field. Uh and

[11:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=695s) what we wanted to do was really reduce

[11:37](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=697s) the friction of back and forth with an

[11:39](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=699s) engineering team. So we tried to really

[11:42](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=702s) remove all the complicated logic and

[11:45](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=705s) make this into a self-s serve UI. So the

[11:47](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=707s) screenshot that you actually see is what

[11:48](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=708s) actually exists. uh so uh you know like

[11:51](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=711s) a product manager or an operator is

[11:53](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=713s) going to come to our UI. They're going

[11:55](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=715s) to set some of these configs uh on the

[11:57](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=717s) platform and then actually run the

[11:59](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=719s) calibration loop themselves. So they

[12:01](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=721s) don't have to worry about the different

[12:03](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=723s) settings that they need to worry about

[12:05](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=725s) what are the different uh you know

[12:07](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=727s) tweaks that they need to do and they can

[12:08](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=728s) actually like you know run a calibration

[12:10](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=730s) loop using any model of their choice. I

[12:12](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=732s) think in this example I have Gemini they

[12:14](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=734s) can use run it using uh you know any of

[12:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=736s) the claude or the openi models too. The

[12:19](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=739s) other important piece was actually uh

[12:21](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=741s) making this reviewable. Uh you know

[12:23](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=743s) again a lot of this uh is a closed box

[12:26](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=746s) where you can't really it's hard to see

[12:28](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=748s) what's actually happening. Uh so the

[12:30](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=750s) second piece that we built was actually

[12:32](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=752s) giving them vis visualization and

[12:34](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=754s) visibility into what's actually

[12:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=755s) happening. So on the left you can see we

[12:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=758s) and this is like one of the good

[12:39](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=759s) examples where we saw like a significant

[12:41](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=761s) amount of improvement in the judge

[12:44](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=764s) prompt. uh and we actually show the you

[12:47](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=767s) know the the previous the original

[12:49](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=769s) system prompt and the calibrated prompt

[12:51](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=771s) to our partners so that they are also

[12:53](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=773s) able to gain that trust uh why as as as

[12:57](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=777s) we build this

[12:59](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=779s) [clears throat]

[12:59](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=779s) >> yeah just want to add to that is this

[13:02](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=782s) enables different configurations in

[13:04](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=784s) different teams in some teams you have

[13:06](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=786s) seen the strategy and operations folks

[13:08](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=788s) own the prompt uh you have seen some

[13:10](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=790s) teams where the product manager owns the

[13:11](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=791s) prompt you have seen some teams where

[13:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=793s) engineering owns the prompt so this

[13:14](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=794s) gives gives the flexibility for teams to

[13:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=796s) design and evolve because we are all

[13:18](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=798s) learning. So the even the org uh design

[13:20](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=800s) is improving and we are enabling that.

[13:23](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=803s) >> Yeah, that that's a good point. I think

[13:25](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=805s) the overall idea was to you know build

[13:27](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=807s) something which is as self-s served as

[13:28](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=808s) possible so that uh you know people

[13:31](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=811s) aren't always necessarily blocked by our

[13:33](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=813s) team helping them out. Um and then

[13:36](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=816s) finally you know uh the quality loop in

[13:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=818s) practice. you know as we've been going

[13:39](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=819s) through this exercise we've seen a lot

[13:41](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=821s) of improvements happening to our product

[13:43](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=823s) as well. So you know for example we sort

[13:46](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=826s) mentioned we started with the UIs we are

[13:48](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=828s) you know now API and workflow first uh

[13:51](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=831s) we're trying to reuse a lot of the

[13:53](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=833s) existing infrastructure that already

[13:54](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=834s) existed at Door Dash uh and that's

[13:57](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=837s) helped us uh get a long way. Now some of

[14:00](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=840s) uh we we we've seen obviously like you

[14:02](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=842s) know really good results. I think a very

[14:04](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=844s) good result that we we do like to call

[14:06](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=846s) out is we actually did see a lot of

[14:08](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=848s) reduction in the spend uh at per

[14:11](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=851s) annotation cost as you as you all can

[14:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=853s) imagine we do have you know thousands of

[14:16](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=856s) rows that need to get annotated every

[14:18](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=858s) week uh and it can get pretty expensive

[14:20](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=860s) at doash scale uh and having this

[14:23](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=863s) selfserve uh annotation platform really

[14:26](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=866s) helped us reduce increase the velocity

[14:29](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=869s) and reduce the cost that we were

[14:30](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=870s) actually spending with these annotators.

[14:33](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=873s) to to annotate the data for us. Uh

[14:35](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=875s) obviously uh this resulted in faster

[14:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=878s) loops. Uh teams were able to iterate

[14:40](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=880s) faster. They were able to uh you know

[14:43](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=883s) calibrate their own judges in a

[14:45](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=885s) completely self-s served way. Uh and

[14:47](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=887s) thus it has resulted us in in in moving

[14:50](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=890s) with a very very high velocity.

[14:54](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=894s) So uh finally I just wanted to you know

[14:56](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=896s) quickly touch on this slide again uh the

[14:59](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=899s) eight steps you know continuous loop uh

[15:01](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=901s) which is you know you you have your

[15:03](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=903s) traces you want to look at your traces

[15:05](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=905s) your sessions you want to sample it down

[15:07](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=907s) to a size which is which you are

[15:09](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=909s) comfortable with uh you want to start

[15:11](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=911s) annotating your data sets you really

[15:13](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=913s) want to start uh making the data better

[15:17](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=917s) with the human knowledge that exists and

[15:19](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=919s) the domain knowledge that exists and

[15:21](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=921s) then calibrate your workflows was

[15:23](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=923s) calibrate your agents, calibrate your

[15:24](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=924s) LLM judges with this golden data set and

[15:28](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=928s) then repeat this whole cycle uh over a

[15:30](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=930s) period of time to you know to ship

[15:32](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=932s) reliably and ship with high quality. Um

[15:36](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=936s) yeah, we have 4 minutes left. Thank you

[15:38](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=938s) once again. I think that was the last

[15:40](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=940s) slide. Uh thanks for attending and if

[15:42](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=942s) there's any questions, we'd be happy to

[15:43](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=943s) hang out after the talk or even happy to

[15:46](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=946s) answer them now.

[15:51](https://www.youtube.com/watch?v=bMjlRrWjdT0&t=951s) >> [applause]
