[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 13440,
    "text": "Good afternoon everyone. Thanks for uh"
  },
  {
    "startMs": 16160,
    "text": "coming for a post lunch uh talk. Always"
  },
  {
    "startMs": 19119,
    "text": "appreciate that. Um my name is Farup and"
  },
  {
    "startMs": 22720,
    "text": "here's my teammate Nachiket. Uh we are"
  },
  {
    "startMs": 25359,
    "text": "uh here behalf of the Door Dash Genai"
  },
  {
    "startMs": 28560,
    "text": "platform team. Um and we kind of wanted"
  },
  {
    "startMs": 31679,
    "text": "to share our eval journey. Uh it started"
  },
  {
    "startMs": 35040,
    "text": "as uh uh you know eval is another"
  },
  {
    "startMs": 38160,
    "text": "engineering thing but then it slowly we"
  },
  {
    "startMs": 40079,
    "text": "realized it evolved into a cross"
  },
  {
    "startMs": 42239,
    "text": "functional effort and we kind of want to"
  },
  {
    "startMs": 45120,
    "text": "share our story here. So what is this"
  },
  {
    "startMs": 48160,
    "text": "team? This team is a gen platform team."
  },
  {
    "startMs": 50879,
    "text": "Uh we are a horizontal team that helps"
  },
  {
    "startMs": 52879,
    "text": "all other product teams. So product"
  },
  {
    "startMs": 54879,
    "text": "teams at Door Dash build on top of the"
  },
  {
    "startMs": 57120,
    "text": "infrastructure and the primitives that"
  },
  {
    "startMs": 58559,
    "text": "we provide. Um and we see our uh USP and"
  },
  {
    "startMs": 63760,
    "text": "the value that we provide is that we"
  },
  {
    "startMs": 65760,
    "text": "help product teams balance these three"
  },
  {
    "startMs": 67840,
    "text": "forces which is accuracy, latency and"
  },
  {
    "startMs": 70080,
    "text": "cost. Um initially we applied this in"
  },
  {
    "startMs": 73360,
    "text": "terms of models but if you think about"
  },
  {
    "startMs": 74880,
    "text": "it it also applies to agents. Um and the"
  },
  {
    "startMs": 78640,
    "text": "way we achieve this is we have uh"
  },
  {
    "startMs": 80799,
    "text": "primitives and building blocks. Um so"
  },
  {
    "startMs": 83360,
    "text": "for example we have an LLM gateway where"
  },
  {
    "startMs": 85759,
    "text": "you can easily switch between different"
  },
  {
    "startMs": 87439,
    "text": "models uh and try the latest and"
  },
  {
    "startMs": 89920,
    "text": "greatest. Uh we have an agent gateway"
  },
  {
    "startMs": 92560,
    "text": "where you can connect to tools uh and"
  },
  {
    "startMs": 95360,
    "text": "other agents uh and we help solve"
  },
  {
    "startMs": 98400,
    "text": "authentication uh agent identity and"
  },
  {
    "startMs": 100799,
    "text": "other things in a central place uh which"
  },
  {
    "startMs": 103040,
    "text": "our security team can bless. Um"
  },
  {
    "startMs": 105759,
    "text": "similarly we pair the LLM gateway with"
  },
  {
    "startMs": 108479,
    "text": "open weights models hosting. Uh, of"
  },
  {
    "startMs": 110880,
    "text": "course cost is a number one concern"
  },
  {
    "startMs": 112799,
    "text": "these days. Uh, and we uh kind of"
  },
  {
    "startMs": 115680,
    "text": "invested in open weights models uh and"
  },
  {
    "startMs": 117920,
    "text": "have seen significant impact uh already."
  },
  {
    "startMs": 120880,
    "text": "Um, and maybe we'll talk about that in a"
  },
  {
    "startMs": 123040,
    "text": "future conference. Uh, the fourth pillar"
  },
  {
    "startMs": 125920,
    "text": "is eval and that's the part that we"
  },
  {
    "startMs": 127920,
    "text": "would want to share today. Um"
  },
  {
    "startMs": 131599,
    "text": "when we started talking to product teams"
  },
  {
    "startMs": 133360,
    "text": "internally at Door Dash uh there were"
  },
  {
    "startMs": 136800,
    "text": "varying uh distinct needs across teams."
  },
  {
    "startMs": 140319,
    "text": "We had a consumer discovery and shopping"
  },
  {
    "startMs": 142959,
    "text": "assistant team. Uh for those who"
  },
  {
    "startMs": 144959,
    "text": "attended Ragago talk earlier today uh"
  },
  {
    "startMs": 146959,
    "text": "you will uh see the need for session"
  },
  {
    "startMs": 149120,
    "text": "level quality judgments. um uh then the"
  },
  {
    "startMs": 152800,
    "text": "personalization ML then you needed uh a"
  },
  {
    "startMs": 155760,
    "text": "way to scale up human judgment and with"
  },
  {
    "startMs": 158640,
    "text": "multi- aent systems we needed trajectory"
  },
  {
    "startMs": 160879,
    "text": "based evals now the question is how do"
  },
  {
    "startMs": 163920,
    "text": "you cater to all these different needs"
  },
  {
    "startMs": 167120,
    "text": "under a common platform"
  },
  {
    "startMs": 170239,
    "text": "and uh as we spoke to these teams we"
  },
  {
    "startMs": 172560,
    "text": "realized like um we needed to empower"
  },
  {
    "startMs": 176000,
    "text": "the people who are the domain experts"
  },
  {
    "startMs": 178239,
    "text": "and in our case that was strategy and"
  },
  {
    "startMs": 180000,
    "text": "operations folks it as product managers"
  },
  {
    "startMs": 182640,
    "text": "uh it was even labeling partners uh and"
  },
  {
    "startMs": 185040,
    "text": "not only engineers so we kind of started"
  },
  {
    "startMs": 187760,
    "text": "with like okay we have to be UI first"
  },
  {
    "startMs": 189599,
    "text": "and this was the guidance we had from"
  },
  {
    "startMs": 191200,
    "text": "Andy Fang our co-founder as well um so"
  },
  {
    "startMs": 194159,
    "text": "we had UIs for non-engineers to"
  },
  {
    "startMs": 196319,
    "text": "contribute uh then we kind of evolved to"
  },
  {
    "startMs": 199680,
    "text": "also being API first so that engineers"
  },
  {
    "startMs": 202080,
    "text": "can also build and not be blocked on the"
  },
  {
    "startMs": 204720,
    "text": "central platform and they can build"
  },
  {
    "startMs": 206480,
    "text": "their own uh uh uh systems uh and Then"
  },
  {
    "startMs": 210799,
    "text": "of course with the coding agents now we"
  },
  {
    "startMs": 213040,
    "text": "have become workflow first where we kind"
  },
  {
    "startMs": 215280,
    "text": "of empower SNO and PMs to also being"
  },
  {
    "startMs": 217680,
    "text": "able to uh navigate uh the platform and"
  },
  {
    "startMs": 221599,
    "text": "uh run operations as well. Um so with"
  },
  {
    "startMs": 225360,
    "text": "that context I'll hand it off to Nachig"
  },
  {
    "startMs": 227440,
    "text": "to talk about uh how we went about"
  },
  {
    "startMs": 229280,
    "text": "delivering this. Cool. Thanks Harup. Um"
  },
  {
    "startMs": 232879,
    "text": "and thanks everyone for joining us. I"
  },
  {
    "startMs": 234480,
    "text": "know France is playing right now and I"
  },
  {
    "startMs": 236879,
    "text": "promise you this will be better than"
  },
  {
    "startMs": 238080,
    "text": "that. I'm kidding. Um so as Faroo was"
  },
  {
    "startMs": 241760,
    "text": "saying uh Evals is not just an"
  },
  {
    "startMs": 243760,
    "text": "engineering harness it is a cross"
  },
  {
    "startMs": 245680,
    "text": "functional effort across different"
  },
  {
    "startMs": 247599,
    "text": "pillars across different uh teams uh"
  },
  {
    "startMs": 250400,
    "text": "that actually helps us add all the"
  },
  {
    "startMs": 253439,
    "text": "domain specific knowledge into our uh"
  },
  {
    "startMs": 256400,
    "text": "into the quality of the AI itself. So"
  },
  {
    "startMs": 258799,
    "text": "from your traces to your data sets uh"
  },
  {
    "startMs": 261359,
    "text": "from you know scoring mechanisms uh this"
  },
  {
    "startMs": 264800,
    "text": "is all basically a team sport. we all"
  },
  {
    "startMs": 267520,
    "text": "have to play uh and help improve the"
  },
  {
    "startMs": 270240,
    "text": "quality of AI."
  },
  {
    "startMs": 273520,
    "text": "So going a little bit deeper into the"
  },
  {
    "startMs": 276160,
    "text": "same aspect uh we have different uh"
  },
  {
    "startMs": 278320,
    "text": "teams uh at Door Dash who help us"
  },
  {
    "startMs": 280160,
    "text": "actually improve the quality of AI. So"
  },
  {
    "startMs": 282240,
    "text": "you're going to have your strategy and"
  },
  {
    "startMs": 283680,
    "text": "operations folks who are going to set"
  },
  {
    "startMs": 285199,
    "text": "priorities, set the quality bar that you"
  },
  {
    "startMs": 287040,
    "text": "want to aim for. You're going to have"
  },
  {
    "startMs": 288960,
    "text": "your product people who are going to"
  },
  {
    "startMs": 290479,
    "text": "translate uh these requirements into"
  },
  {
    "startMs": 293199,
    "text": "rubrics workflows. You're going to have"
  },
  {
    "startMs": 295040,
    "text": "your operations teams running uh"
  },
  {
    "startMs": 297360,
    "text": "annotations. You're going to have your"
  },
  {
    "startMs": 298960,
    "text": "engineering teams like us uh providing"
  },
  {
    "startMs": 301360,
    "text": "APIs, telemetry, data sets, judges, all"
  },
  {
    "startMs": 304800,
    "text": "you know the the cool things. Um and"
  },
  {
    "startMs": 308880,
    "text": "combining all these together is is what"
  },
  {
    "startMs": 311919,
    "text": "a recipe is for actually making sure"
  },
  {
    "startMs": 314720,
    "text": "that you are shipping quality AI"
  },
  {
    "startMs": 316720,
    "text": "products through an eval platform."
  },
  {
    "startMs": 319600,
    "text": "So we've tried to boil this down uh into"
  },
  {
    "startMs": 322960,
    "text": "sort of you know like a a continuous"
  },
  {
    "startMs": 324479,
    "text": "iteration loop. Uh so right from tracing"
  },
  {
    "startMs": 328639,
    "text": "uh you know having a tracing solution"
  },
  {
    "startMs": 330639,
    "text": "viewing your sessions your traces to"
  },
  {
    "startMs": 333360,
    "text": "sampling them down you uh you know to a"
  },
  {
    "startMs": 335919,
    "text": "very small uh set that you actually want"
  },
  {
    "startMs": 338240,
    "text": "to look at uh annotating these with the"
  },
  {
    "startMs": 340960,
    "text": "domain specific expertise that you bring"
  },
  {
    "startMs": 342960,
    "text": "in with the different teams I mentioned"
  },
  {
    "startMs": 345199,
    "text": "reviewing those uh then creating those"
  },
  {
    "startMs": 347759,
    "text": "golden data sets which are going to be"
  },
  {
    "startMs": 350240,
    "text": "you you know your uh golden data sets"
  },
  {
    "startMs": 353120,
    "text": "and that that you want to measure or"
  },
  {
    "startMs": 354720,
    "text": "calibrate against uh and then of course"
  },
  {
    "startMs": 356960,
    "text": "like you know monitoring this over a"
  },
  {
    "startMs": 358639,
    "text": "period of time and then you know rinse"
  },
  {
    "startMs": 360639,
    "text": "and repeat uh go through the whole loop"
  },
  {
    "startMs": 362240,
    "text": "again. So this is in our experience has"
  },
  {
    "startMs": 365280,
    "text": "been you know like a good sort of"
  },
  {
    "startMs": 366800,
    "text": "continuous loop uh for you know shipping"
  },
  {
    "startMs": 369919,
    "text": "quality AI"
  },
  {
    "startMs": 372319,
    "text": "at the plat on on the platform level uh"
  },
  {
    "startMs": 374639,
    "text": "we we have two surfaces uh so we have"
  },
  {
    "startMs": 377120,
    "text": "the telemetry layer uh where we have all"
  },
  {
    "startMs": 379440,
    "text": "our traces our scores uh observations"
  },
  {
    "startMs": 382720,
    "text": "that is also sort of the plane where"
  },
  {
    "startMs": 385600,
    "text": "users are able to access these traces"
  },
  {
    "startMs": 387600,
    "text": "using an MCP using an SDK uh using our"
  },
  {
    "startMs": 390639,
    "text": "APIs and then we have the workflow This"
  },
  {
    "startMs": 393039,
    "text": "is where a lot of our strat ops, our"
  },
  {
    "startMs": 394880,
    "text": "product teams operate on the platform."
  },
  {
    "startMs": 397360,
    "text": "So this is where all the annotation"
  },
  {
    "startMs": 398800,
    "text": "tasks are set. Uh you know this is where"
  },
  {
    "startMs": 401199,
    "text": "they review their golden data sets, uh"
  },
  {
    "startMs": 403280,
    "text": "create their judges, calibrate their"
  },
  {
    "startMs": 405199,
    "text": "judges and so on."
  },
  {
    "startMs": 407600,
    "text": "So maybe today we'll go through you know"
  },
  {
    "startMs": 410880,
    "text": "these sort of four different uh modules"
  },
  {
    "startMs": 413039,
    "text": "or pillars of our platform uh step by"
  },
  {
    "startMs": 415280,
    "text": "step. Uh so again first one uh tracing"
  },
  {
    "startMs": 418319,
    "text": "and sampling uh which is actually"
  },
  {
    "startMs": 420880,
    "text": "capturing what your agents what your"
  },
  {
    "startMs": 423360,
    "text": "LLMs are actually uh you know outputting"
  },
  {
    "startMs": 426080,
    "text": "for the lack of better words uh and"
  },
  {
    "startMs": 428160,
    "text": "actually viewing those."
  },
  {
    "startMs": 430720,
    "text": "Now in order to also power this uh whole"
  },
  {
    "startMs": 434560,
    "text": "platform we have I think as far"
  },
  {
    "startMs": 436160,
    "text": "mentioned we have gone in an API first"
  },
  {
    "startMs": 438560,
    "text": "uh approach. Uh what that has allowed us"
  },
  {
    "startMs": 440880,
    "text": "to do is have these table APIs that"
  },
  {
    "startMs": 443360,
    "text": "actually uh you know and then you know"
  },
  {
    "startMs": 445919,
    "text": "build UIs uh on top of that. Uh so all"
  },
  {
    "startMs": 449039,
    "text": "our scores our data sets uh these are"
  },
  {
    "startMs": 451840,
    "text": "all powered by very stable APIs uh that"
  },
  {
    "startMs": 454319,
    "text": "our team owns. Uh so all your API access"
  },
  {
    "startMs": 458319,
    "text": "uh including you know like an SDK access"
  },
  {
    "startMs": 460400,
    "text": "is basically powered by this single uh"
  },
  {
    "startMs": 462560,
    "text": "plane."
  },
  {
    "startMs": 464080,
    "text": "Um again"
  },
  {
    "startMs": 467280,
    "text": "going back uh and you know like just"
  },
  {
    "startMs": 469680,
    "text": "refreshing your memory. Uh step one"
  },
  {
    "startMs": 471840,
    "text": "capture your traces uh capture your"
  },
  {
    "startMs": 474319,
    "text": "sessions uh measure your scores. Uh then"
  },
  {
    "startMs": 477599,
    "text": "you want to start uh almost you know"
  },
  {
    "startMs": 479840,
    "text": "like adding all your judgment your"
  },
  {
    "startMs": 482080,
    "text": "context your domain knowledge uh and"
  },
  {
    "startMs": 485440,
    "text": "then calibrating your judges is what we"
  },
  {
    "startMs": 487759,
    "text": "have seen as the whole uh life cycle."
  },
  {
    "startMs": 493120,
    "text": "Step two is on the annotation side. Uh"
  },
  {
    "startMs": 495440,
    "text": "so you obviously are capturing a lot of"
  },
  {
    "startMs": 497280,
    "text": "your uh agentic behavior, your sessions,"
  },
  {
    "startMs": 499840,
    "text": "your traces, but you actually want to"
  },
  {
    "startMs": 501840,
    "text": "see what are some places where things"
  },
  {
    "startMs": 504319,
    "text": "went well and what are some places where"
  },
  {
    "startMs": 506639,
    "text": "things did not go well. This is where"
  },
  {
    "startMs": 508400,
    "text": "you can actually titrate your your you"
  },
  {
    "startMs": 511039,
    "text": "know and actually look in inside what's"
  },
  {
    "startMs": 513279,
    "text": "actually happening uh at the session"
  },
  {
    "startMs": 515360,
    "text": "level and annotate these data sets. Um"
  },
  {
    "startMs": 519519,
    "text": "and as Surup mentioned, we have a lot of"
  },
  {
    "startMs": 521919,
    "text": "use cases. we have we we talked to"
  },
  {
    "startMs": 523839,
    "text": "multiple different teams who have uh"
  },
  {
    "startMs": 526320,
    "text": "various uh ways of annotating uh their"
  },
  {
    "startMs": 529279,
    "text": "data sets uh and it's it's almost hard"
  },
  {
    "startMs": 532720,
    "text": "for a platform team to you know build"
  },
  {
    "startMs": 535120,
    "text": "like a UI specific uh for each use case"
  },
  {
    "startMs": 538320,
    "text": "uh and and you know to give you an"
  },
  {
    "startMs": 539760,
    "text": "example uh it's usually going to be an"
  },
  {
    "startMs": 542800,
    "text": "annotator who's going to annotate these"
  },
  {
    "startMs": 545200,
    "text": "data sets so the platform team is you"
  },
  {
    "startMs": 547519,
    "text": "know in charge of the APIs we have a"
  },
  {
    "startMs": 549839,
    "text": "strategy and of person who's actually"
  },
  {
    "startMs": 552080,
    "text": "deciding what to annotate and then you"
  },
  {
    "startMs": 553920,
    "text": "have an annotator who's actually going"
  },
  {
    "startMs": 555200,
    "text": "to annotate uh your data set. So we took"
  },
  {
    "startMs": 558080,
    "text": "this approach uh everybody uh has uh you"
  },
  {
    "startMs": 561120,
    "text": "know access to coding agents uh and we"
  },
  {
    "startMs": 563440,
    "text": "actually doubled down on that API first"
  },
  {
    "startMs": 565040,
    "text": "approach. So because we had these APIs"
  },
  {
    "startMs": 567760,
    "text": "we were actually uh able to enable our"
  },
  {
    "startMs": 570640,
    "text": "statops teams to use something like a"
  },
  {
    "startMs": 572800,
    "text": "codeex or a claw code and v code their"
  },
  {
    "startMs": 575519,
    "text": "own annotation UIs. Uh so we had"
  },
  {
    "startMs": 579279,
    "text": "different use cases. Uh I think we had a"
  },
  {
    "startMs": 581760,
    "text": "talk from Ragav before. Uh we had image"
  },
  {
    "startMs": 584640,
    "text": "annotation use cases. We had some uh you"
  },
  {
    "startMs": 586640,
    "text": "know manual testing use cases. What"
  },
  {
    "startMs": 589040,
    "text": "stood out to us was the underlying"
  },
  {
    "startMs": 591600,
    "text": "patterns were similar. So if we are are"
  },
  {
    "startMs": 594080,
    "text": "API first uh we can actually enable our"
  },
  {
    "startMs": 597120,
    "text": "our our partners to simply v code these"
  },
  {
    "startMs": 599519,
    "text": "UIs for annotation. So it's it's like a"
  },
  {
    "startMs": 602399,
    "text": "very simple example then you know of of"
  },
  {
    "startMs": 605200,
    "text": "a vibe coded UI looks pretty clean does"
  },
  {
    "startMs": 607760,
    "text": "the job uh and you get you know the"
  },
  {
    "startMs": 610000,
    "text": "annotation that you eat this is"
  },
  {
    "startMs": 611519,
    "text": "basically like a menu from a restaurant"
  },
  {
    "startMs": 613279,
    "text": "uh it's it's you know nothing crazy uh"
  },
  {
    "startMs": 616560,
    "text": "but the point I want to make here is"
  },
  {
    "startMs": 618399,
    "text": "that what helped us was to give this"
  },
  {
    "startMs": 622399,
    "text": "workflow in the hands of the operators"
  },
  {
    "startMs": 624399,
    "text": "so that they can actually build their"
  },
  {
    "startMs": 626079,
    "text": "own vcoded annotation UIs. Uh so moving"
  },
  {
    "startMs": 629600,
    "text": "on once you have these annotation UIs"
  },
  {
    "startMs": 631600,
    "text": "you obviously want to you know calibrate"
  },
  {
    "startMs": 633600,
    "text": "your your your judge prompts you"
  },
  {
    "startMs": 635360,
    "text": "obviously have some LM as a judge uh"
  },
  {
    "startMs": 637519,
    "text": "metric that you're tracking you want to"
  },
  {
    "startMs": 639040,
    "text": "now start improving that with these"
  },
  {
    "startMs": 641440,
    "text": "golden data sets"
  },
  {
    "startMs": 643760,
    "text": "u in order to do that uh you know we"
  },
  {
    "startMs": 646640,
    "text": "have a pretty simple process uh you"
  },
  {
    "startMs": 648480,
    "text": "you're going to start with you know some"
  },
  {
    "startMs": 649920,
    "text": "judge prompt take a look at you know"
  },
  {
    "startMs": 651920,
    "text": "what exactly do you want to measure from"
  },
  {
    "startMs": 653600,
    "text": "the output uh g you know have have us"
  },
  {
    "startMs": 656399,
    "text": "have something simple you're going to"
  },
  {
    "startMs": 658240,
    "text": "have your baseline scores uh where"
  },
  {
    "startMs": 660079,
    "text": "you're going to simply run those LLM"
  },
  {
    "startMs": 662079,
    "text": "judges on your traces and then you're"
  },
  {
    "startMs": 664480,
    "text": "going to have that optimization loop. Uh"
  },
  {
    "startMs": 666800,
    "text": "so we use uh the JPEA library which is a"
  },
  {
    "startMs": 669120,
    "text": "pretty commonly used library out there"
  },
  {
    "startMs": 671440,
    "text": "for prompt optimization. Uh and once you"
  },
  {
    "startMs": 674000,
    "text": "know the the iteration loop is complete"
  },
  {
    "startMs": 676640,
    "text": "uh our partner teams are happy they're"
  },
  {
    "startMs": 678720,
    "text": "going to then elevate that judge prompt"
  },
  {
    "startMs": 681360,
    "text": "as their LLM as a judge. Now even while"
  },
  {
    "startMs": 684480,
    "text": "doing that uh LLM as a judge as a"
  },
  {
    "startMs": 686880,
    "text": "concept the whole prompt calibration"
  },
  {
    "startMs": 688560,
    "text": "concept might be uh straightforward to a"
  },
  {
    "startMs": 691200,
    "text": "lot of folks but it is still like a"
  },
  {
    "startMs": 692720,
    "text": "pretty new and evolving field. Uh and"
  },
  {
    "startMs": 695920,
    "text": "what we wanted to do was really reduce"
  },
  {
    "startMs": 697839,
    "text": "the friction of back and forth with an"
  },
  {
    "startMs": 699519,
    "text": "engineering team. So we tried to really"
  },
  {
    "startMs": 702880,
    "text": "remove all the complicated logic and"
  },
  {
    "startMs": 705279,
    "text": "make this into a self-s serve UI. So the"
  },
  {
    "startMs": 707519,
    "text": "screenshot that you actually see is what"
  },
  {
    "startMs": 708959,
    "text": "actually exists. uh so uh you know like"
  },
  {
    "startMs": 711279,
    "text": "a product manager or an operator is"
  },
  {
    "startMs": 713760,
    "text": "going to come to our UI. They're going"
  },
  {
    "startMs": 715120,
    "text": "to set some of these configs uh on the"
  },
  {
    "startMs": 717920,
    "text": "platform and then actually run the"
  },
  {
    "startMs": 719600,
    "text": "calibration loop themselves. So they"
  },
  {
    "startMs": 721200,
    "text": "don't have to worry about the different"
  },
  {
    "startMs": 723120,
    "text": "settings that they need to worry about"
  },
  {
    "startMs": 725279,
    "text": "what are the different uh you know"
  },
  {
    "startMs": 727440,
    "text": "tweaks that they need to do and they can"
  },
  {
    "startMs": 728959,
    "text": "actually like you know run a calibration"
  },
  {
    "startMs": 730800,
    "text": "loop using any model of their choice. I"
  },
  {
    "startMs": 732480,
    "text": "think in this example I have Gemini they"
  },
  {
    "startMs": 734720,
    "text": "can use run it using uh you know any of"
  },
  {
    "startMs": 736480,
    "text": "the claude or the openi models too. The"
  },
  {
    "startMs": 739279,
    "text": "other important piece was actually uh"
  },
  {
    "startMs": 741360,
    "text": "making this reviewable. Uh you know"
  },
  {
    "startMs": 743200,
    "text": "again a lot of this uh is a closed box"
  },
  {
    "startMs": 746320,
    "text": "where you can't really it's hard to see"
  },
  {
    "startMs": 748560,
    "text": "what's actually happening. Uh so the"
  },
  {
    "startMs": 750480,
    "text": "second piece that we built was actually"
  },
  {
    "startMs": 752079,
    "text": "giving them vis visualization and"
  },
  {
    "startMs": 754000,
    "text": "visibility into what's actually"
  },
  {
    "startMs": 755920,
    "text": "happening. So on the left you can see we"
  },
  {
    "startMs": 758399,
    "text": "and this is like one of the good"
  },
  {
    "startMs": 759680,
    "text": "examples where we saw like a significant"
  },
  {
    "startMs": 761440,
    "text": "amount of improvement in the judge"
  },
  {
    "startMs": 764079,
    "text": "prompt. uh and we actually show the you"
  },
  {
    "startMs": 767279,
    "text": "know the the previous the original"
  },
  {
    "startMs": 769200,
    "text": "system prompt and the calibrated prompt"
  },
  {
    "startMs": 771680,
    "text": "to our partners so that they are also"
  },
  {
    "startMs": 773920,
    "text": "able to gain that trust uh why as as as"
  },
  {
    "startMs": 777440,
    "text": "we build this"
  },
  {
    "startMs": 779458,
    "text": "[clears throat]"
  },
  {
    "startMs": 779920,
    "text": ">> yeah just want to add to that is this"
  },
  {
    "startMs": 782639,
    "text": "enables different configurations in"
  },
  {
    "startMs": 784560,
    "text": "different teams in some teams you have"
  },
  {
    "startMs": 786320,
    "text": "seen the strategy and operations folks"
  },
  {
    "startMs": 788000,
    "text": "own the prompt uh you have seen some"
  },
  {
    "startMs": 790240,
    "text": "teams where the product manager owns the"
  },
  {
    "startMs": 791760,
    "text": "prompt you have seen some teams where"
  },
  {
    "startMs": 793120,
    "text": "engineering owns the prompt so this"
  },
  {
    "startMs": 794399,
    "text": "gives gives the flexibility for teams to"
  },
  {
    "startMs": 796560,
    "text": "design and evolve because we are all"
  },
  {
    "startMs": 798320,
    "text": "learning. So the even the org uh design"
  },
  {
    "startMs": 800959,
    "text": "is improving and we are enabling that."
  },
  {
    "startMs": 803760,
    "text": ">> Yeah, that that's a good point. I think"
  },
  {
    "startMs": 805519,
    "text": "the overall idea was to you know build"
  },
  {
    "startMs": 807200,
    "text": "something which is as self-s served as"
  },
  {
    "startMs": 808959,
    "text": "possible so that uh you know people"
  },
  {
    "startMs": 811519,
    "text": "aren't always necessarily blocked by our"
  },
  {
    "startMs": 813680,
    "text": "team helping them out. Um and then"
  },
  {
    "startMs": 816560,
    "text": "finally you know uh the quality loop in"
  },
  {
    "startMs": 818560,
    "text": "practice. you know as we've been going"
  },
  {
    "startMs": 819920,
    "text": "through this exercise we've seen a lot"
  },
  {
    "startMs": 821600,
    "text": "of improvements happening to our product"
  },
  {
    "startMs": 823760,
    "text": "as well. So you know for example we sort"
  },
  {
    "startMs": 826639,
    "text": "mentioned we started with the UIs we are"
  },
  {
    "startMs": 828480,
    "text": "you know now API and workflow first uh"
  },
  {
    "startMs": 831200,
    "text": "we're trying to reuse a lot of the"
  },
  {
    "startMs": 833279,
    "text": "existing infrastructure that already"
  },
  {
    "startMs": 834720,
    "text": "existed at Door Dash uh and that's"
  },
  {
    "startMs": 837440,
    "text": "helped us uh get a long way. Now some of"
  },
  {
    "startMs": 840560,
    "text": "uh we we we've seen obviously like you"
  },
  {
    "startMs": 842320,
    "text": "know really good results. I think a very"
  },
  {
    "startMs": 844639,
    "text": "good result that we we do like to call"
  },
  {
    "startMs": 846800,
    "text": "out is we actually did see a lot of"
  },
  {
    "startMs": 848720,
    "text": "reduction in the spend uh at per"
  },
  {
    "startMs": 851839,
    "text": "annotation cost as you as you all can"
  },
  {
    "startMs": 853839,
    "text": "imagine we do have you know thousands of"
  },
  {
    "startMs": 856320,
    "text": "rows that need to get annotated every"
  },
  {
    "startMs": 858560,
    "text": "week uh and it can get pretty expensive"
  },
  {
    "startMs": 860720,
    "text": "at doash scale uh and having this"
  },
  {
    "startMs": 863760,
    "text": "selfserve uh annotation platform really"
  },
  {
    "startMs": 866880,
    "text": "helped us reduce increase the velocity"
  },
  {
    "startMs": 869440,
    "text": "and reduce the cost that we were"
  },
  {
    "startMs": 870880,
    "text": "actually spending with these annotators."
  },
  {
    "startMs": 873120,
    "text": "to to annotate the data for us. Uh"
  },
  {
    "startMs": 875839,
    "text": "obviously uh this resulted in faster"
  },
  {
    "startMs": 878320,
    "text": "loops. Uh teams were able to iterate"
  },
  {
    "startMs": 880399,
    "text": "faster. They were able to uh you know"
  },
  {
    "startMs": 883760,
    "text": "calibrate their own judges in a"
  },
  {
    "startMs": 885680,
    "text": "completely self-s served way. Uh and"
  },
  {
    "startMs": 887839,
    "text": "thus it has resulted us in in in moving"
  },
  {
    "startMs": 890720,
    "text": "with a very very high velocity."
  },
  {
    "startMs": 894240,
    "text": "So uh finally I just wanted to you know"
  },
  {
    "startMs": 896560,
    "text": "quickly touch on this slide again uh the"
  },
  {
    "startMs": 899040,
    "text": "eight steps you know continuous loop uh"
  },
  {
    "startMs": 901680,
    "text": "which is you know you you have your"
  },
  {
    "startMs": 903199,
    "text": "traces you want to look at your traces"
  },
  {
    "startMs": 905440,
    "text": "your sessions you want to sample it down"
  },
  {
    "startMs": 907600,
    "text": "to a size which is which you are"
  },
  {
    "startMs": 909120,
    "text": "comfortable with uh you want to start"
  },
  {
    "startMs": 911760,
    "text": "annotating your data sets you really"
  },
  {
    "startMs": 913680,
    "text": "want to start uh making the data better"
  },
  {
    "startMs": 917120,
    "text": "with the human knowledge that exists and"
  },
  {
    "startMs": 919519,
    "text": "the domain knowledge that exists and"
  },
  {
    "startMs": 921279,
    "text": "then calibrate your workflows was"
  },
  {
    "startMs": 923040,
    "text": "calibrate your agents, calibrate your"
  },
  {
    "startMs": 924959,
    "text": "LLM judges with this golden data set and"
  },
  {
    "startMs": 928160,
    "text": "then repeat this whole cycle uh over a"
  },
  {
    "startMs": 930880,
    "text": "period of time to you know to ship"
  },
  {
    "startMs": 932720,
    "text": "reliably and ship with high quality. Um"
  },
  {
    "startMs": 936079,
    "text": "yeah, we have 4 minutes left. Thank you"
  },
  {
    "startMs": 938160,
    "text": "once again. I think that was the last"
  },
  {
    "startMs": 940560,
    "text": "slide. Uh thanks for attending and if"
  },
  {
    "startMs": 942160,
    "text": "there's any questions, we'd be happy to"
  },
  {
    "startMs": 943680,
    "text": "hang out after the talk or even happy to"
  },
  {
    "startMs": 946480,
    "text": "answer them now."
  },
  {
    "startMs": 951070,
    "text": ">> [applause]"
  }
]
