AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash

[music]

Good afternoon everyone. Thanks for uh

coming for a post lunch uh talk. Always

appreciate that. Um my name is Farup and

here's my teammate Nachiket. Uh we are

uh here behalf of the Door Dash Genai

platform team. Um and we kind of wanted

to share our eval journey. Uh it started

as uh uh you know eval is another

engineering thing but then it slowly we

realized it evolved into a cross

functional effort and we kind of want to

share our story here. So what is this

team? This team is a gen platform team.

Uh we are a horizontal team that helps

all other product teams. So product

teams at Door Dash build on top of the

infrastructure and the primitives that

we provide. Um and we see our uh USP and

the value that we provide is that we

help product teams balance these three

forces which is accuracy, latency and

cost. Um initially we applied this in

terms of models but if you think about

it it also applies to agents. Um and the

way we achieve this is we have uh

primitives and building blocks. Um so

for example we have an LLM gateway where

you can easily switch between different

models uh and try the latest and

greatest. Uh we have an agent gateway

where you can connect to tools uh and

other agents uh and we help solve

authentication uh agent identity and

other things in a central place uh which

our security team can bless. Um

similarly we pair the LLM gateway with

open weights models hosting. Uh, of

course cost is a number one concern

these days. Uh, and we uh kind of

invested in open weights models uh and

have seen significant impact uh already.

Um, and maybe we'll talk about that in a

future conference. Uh, the fourth pillar

is eval and that's the part that we

would want to share today. Um

when we started talking to product teams

internally at Door Dash uh there were

varying uh distinct needs across teams.

We had a consumer discovery and shopping

assistant team. Uh for those who

attended Ragago talk earlier today uh

you will uh see the need for session

level quality judgments. um uh then the

personalization ML then you needed uh a

way to scale up human judgment and with

multi- aent systems we needed trajectory

based evals now the question is how do

you cater to all these different needs

under a common platform

and uh as we spoke to these teams we

realized like um we needed to empower

the people who are the domain experts

and in our case that was strategy and

operations folks it as product managers

uh it was even labeling partners uh and

not only engineers so we kind of started

with like okay we have to be UI first

and this was the guidance we had from

Andy Fang our co-founder as well um so

we had UIs for non-engineers to

contribute uh then we kind of evolved to

also being API first so that engineers

can also build and not be blocked on the

central platform and they can build

their own uh uh uh systems uh and Then

of course with the coding agents now we

have become workflow first where we kind

of empower SNO and PMs to also being

able to uh navigate uh the platform and

uh run operations as well. Um so with

that context I'll hand it off to Nachig

to talk about uh how we went about

delivering this. Cool. Thanks Harup. Um

and thanks everyone for joining us. I

know France is playing right now and I

promise you this will be better than

that. I'm kidding. Um so as Faroo was

saying uh Evals is not just an

engineering harness it is a cross

functional effort across different

pillars across different uh teams uh

that actually helps us add all the

domain specific knowledge into our uh

into the quality of the AI itself. So

from your traces to your data sets uh

from you know scoring mechanisms uh this

is all basically a team sport. we all

have to play uh and help improve the

quality of AI.

So going a little bit deeper into the

same aspect uh we have different uh

teams uh at Door Dash who help us

actually improve the quality of AI. So

you're going to have your strategy and

operations folks who are going to set

priorities, set the quality bar that you

want to aim for. You're going to have

your product people who are going to

translate uh these requirements into

rubrics workflows. You're going to have

your operations teams running uh

annotations. You're going to have your

engineering teams like us uh providing

APIs, telemetry, data sets, judges, all

you know the the cool things. Um and

combining all these together is is what

a recipe is for actually making sure

that you are shipping quality AI

products through an eval platform.

So we've tried to boil this down uh into

sort of you know like a a continuous

iteration loop. Uh so right from tracing

uh you know having a tracing solution

viewing your sessions your traces to

sampling them down you uh you know to a

very small uh set that you actually want

to look at uh annotating these with the

domain specific expertise that you bring

in with the different teams I mentioned

reviewing those uh then creating those

golden data sets which are going to be

you you know your uh golden data sets

and that that you want to measure or

calibrate against uh and then of course

like you know monitoring this over a

period of time and then you know rinse

and repeat uh go through the whole loop

again. So this is in our experience has

been you know like a good sort of

continuous loop uh for you know shipping

quality AI

at the plat on on the platform level uh

we we have two surfaces uh so we have

the telemetry layer uh where we have all

our traces our scores uh observations

that is also sort of the plane where

users are able to access these traces

using an MCP using an SDK uh using our

APIs and then we have the workflow This

is where a lot of our strat ops, our

product teams operate on the platform.

So this is where all the annotation

tasks are set. Uh you know this is where

they review their golden data sets, uh

create their judges, calibrate their

judges and so on.

So maybe today we'll go through you know

these sort of four different uh modules

or pillars of our platform uh step by

step. Uh so again first one uh tracing

and sampling uh which is actually

capturing what your agents what your

LLMs are actually uh you know outputting

for the lack of better words uh and

actually viewing those.

Now in order to also power this uh whole

platform we have I think as far

mentioned we have gone in an API first

uh approach. Uh what that has allowed us

to do is have these table APIs that

actually uh you know and then you know

build UIs uh on top of that. Uh so all

our scores our data sets uh these are

all powered by very stable APIs uh that

our team owns. Uh so all your API access

uh including you know like an SDK access

is basically powered by this single uh

plane.

Um again

going back uh and you know like just

refreshing your memory. Uh step one

capture your traces uh capture your

sessions uh measure your scores. Uh then

you want to start uh almost you know

like adding all your judgment your

context your domain knowledge uh and

then calibrating your judges is what we

have seen as the whole uh life cycle.

Step two is on the annotation side. Uh

so you obviously are capturing a lot of

your uh agentic behavior, your sessions,

your traces, but you actually want to

see what are some places where things

went well and what are some places where

things did not go well. This is where

you can actually titrate your your you

know and actually look in inside what's

actually happening uh at the session

level and annotate these data sets. Um

and as Surup mentioned, we have a lot of

use cases. we have we we talked to

multiple different teams who have uh

various uh ways of annotating uh their

data sets uh and it's it's almost hard

for a platform team to you know build

like a UI specific uh for each use case

uh and and you know to give you an

example uh it's usually going to be an

annotator who's going to annotate these

data sets so the platform team is you

know in charge of the APIs we have a

strategy and of person who's actually

deciding what to annotate and then you

have an annotator who's actually going

to annotate uh your data set. So we took

this approach uh everybody uh has uh you

know access to coding agents uh and we

actually doubled down on that API first

approach. So because we had these APIs

we were actually uh able to enable our

statops teams to use something like a

codeex or a claw code and v code their

own annotation UIs. Uh so we had

different use cases. Uh I think we had a

talk from Ragav before. Uh we had image

annotation use cases. We had some uh you

know manual testing use cases. What

stood out to us was the underlying

patterns were similar. So if we are are

API first uh we can actually enable our

our our partners to simply v code these

UIs for annotation. So it's it's like a

very simple example then you know of of

a vibe coded UI looks pretty clean does

the job uh and you get you know the

annotation that you eat this is

basically like a menu from a restaurant

uh it's it's you know nothing crazy uh

but the point I want to make here is

that what helped us was to give this

workflow in the hands of the operators

so that they can actually build their

own vcoded annotation UIs. Uh so moving

on once you have these annotation UIs

you obviously want to you know calibrate

your your your judge prompts you

obviously have some LM as a judge uh

metric that you're tracking you want to

now start improving that with these

golden data sets

u in order to do that uh you know we

have a pretty simple process uh you

you're going to start with you know some

judge prompt take a look at you know

what exactly do you want to measure from

the output uh g you know have have us

have something simple you're going to

have your baseline scores uh where

you're going to simply run those LLM

judges on your traces and then you're

going to have that optimization loop. Uh

so we use uh the JPEA library which is a

pretty commonly used library out there

for prompt optimization. Uh and once you

know the the iteration loop is complete

uh our partner teams are happy they're

going to then elevate that judge prompt

as their LLM as a judge. Now even while

doing that uh LLM as a judge as a

concept the whole prompt calibration

concept might be uh straightforward to a

lot of folks but it is still like a

pretty new and evolving field. Uh and

what we wanted to do was really reduce

the friction of back and forth with an

engineering team. So we tried to really

remove all the complicated logic and

make this into a self-s serve UI. So the

screenshot that you actually see is what

actually exists. uh so uh you know like

a product manager or an operator is

going to come to our UI. They're going

to set some of these configs uh on the

platform and then actually run the

calibration loop themselves. So they

don't have to worry about the different

settings that they need to worry about

what are the different uh you know

tweaks that they need to do and they can

actually like you know run a calibration

loop using any model of their choice. I

think in this example I have Gemini they

can use run it using uh you know any of

the claude or the openi models too. The

other important piece was actually uh

making this reviewable. Uh you know

again a lot of this uh is a closed box

where you can't really it's hard to see

what's actually happening. Uh so the

second piece that we built was actually

giving them vis visualization and

visibility into what's actually

happening. So on the left you can see we

and this is like one of the good

examples where we saw like a significant

amount of improvement in the judge

prompt. uh and we actually show the you

know the the previous the original

system prompt and the calibrated prompt

to our partners so that they are also

able to gain that trust uh why as as as

we build this

[clears throat]

>> yeah just want to add to that is this

enables different configurations in

different teams in some teams you have

seen the strategy and operations folks

own the prompt uh you have seen some

teams where the product manager owns the

prompt you have seen some teams where

engineering owns the prompt so this

gives gives the flexibility for teams to

design and evolve because we are all

learning. So the even the org uh design

is improving and we are enabling that.

>> Yeah, that that's a good point. I think

the overall idea was to you know build

something which is as self-s served as

possible so that uh you know people

aren't always necessarily blocked by our

team helping them out. Um and then

finally you know uh the quality loop in

practice. you know as we've been going

through this exercise we've seen a lot

of improvements happening to our product

as well. So you know for example we sort

mentioned we started with the UIs we are

you know now API and workflow first uh

we're trying to reuse a lot of the

existing infrastructure that already

existed at Door Dash uh and that's

helped us uh get a long way. Now some of

uh we we we've seen obviously like you

know really good results. I think a very

good result that we we do like to call

out is we actually did see a lot of

reduction in the spend uh at per

annotation cost as you as you all can

imagine we do have you know thousands of

rows that need to get annotated every

week uh and it can get pretty expensive

at doash scale uh and having this

selfserve uh annotation platform really

helped us reduce increase the velocity

and reduce the cost that we were

actually spending with these annotators.

to to annotate the data for us. Uh

obviously uh this resulted in faster

loops. Uh teams were able to iterate

faster. They were able to uh you know

calibrate their own judges in a

completely self-s served way. Uh and

thus it has resulted us in in in moving

with a very very high velocity.

So uh finally I just wanted to you know

quickly touch on this slide again uh the

eight steps you know continuous loop uh

which is you know you you have your

traces you want to look at your traces

your sessions you want to sample it down

to a size which is which you are

comfortable with uh you want to start

annotating your data sets you really

want to start uh making the data better

with the human knowledge that exists and

the domain knowledge that exists and

then calibrate your workflows was

calibrate your agents, calibrate your

LLM judges with this golden data set and

then repeat this whole cycle uh over a

period of time to you know to ship

reliably and ship with high quality. Um

yeah, we have 4 minutes left. Thank you

once again. I think that was the last

slide. Uh thanks for attending and if

there's any questions, we'd be happy to

hang out after the talk or even happy to

answer them now.

>> [applause]