[
  {
    "startMs": 16640,
    "text": "Okay. Hello. Thank you for showing up."
  },
  {
    "startMs": 19920,
    "text": "Uh my name is Zuben Isola. Uh I work at"
  },
  {
    "startMs": 22160,
    "text": "Weights and Biases. I'm building the"
  },
  {
    "startMs": 24000,
    "text": "Arya agent which is something that we"
  },
  {
    "startMs": 25920,
    "text": "released to general availability on"
  },
  {
    "startMs": 28000,
    "text": "Monday. You might have seen Tim's talk"
  },
  {
    "startMs": 30000,
    "text": "uh one of the pe my peers at Weights and"
  },
  {
    "startMs": 31599,
    "text": "Biases who I work with uh on the main"
  },
  {
    "startMs": 33680,
    "text": "stage yesterday. So I'm going to give"
  },
  {
    "startMs": 35840,
    "text": "you a little bit of a deep dive into how"
  },
  {
    "startMs": 37280,
    "text": "we evaluate and build the Arya agent uh"
  },
  {
    "startMs": 39520,
    "text": "and in particular uh the evaluation"
  },
  {
    "startMs": 41760,
    "text": "framework that we use and how we use"
  },
  {
    "startMs": 42960,
    "text": "Arya to sort of self-reinforce itself in"
  },
  {
    "startMs": 44960,
    "text": "the loop of doing research for itself uh"
  },
  {
    "startMs": 47520,
    "text": "and talk about some weights and biases"
  },
  {
    "startMs": 49039,
    "text": "things. So yeah, this is a rip from some"
  },
  {
    "startMs": 51680,
    "text": "of the slides that I presented at"
  },
  {
    "startMs": 52960,
    "text": "Nurups, but uh I think there's some"
  },
  {
    "startMs": 54800,
    "text": "interesting things about like using uh"
  },
  {
    "startMs": 56559,
    "text": "agentic harnesses. I I imagine most"
  },
  {
    "startMs": 58480,
    "text": "people in the audience or at this"
  },
  {
    "startMs": 59920,
    "text": "conference are all excited to build"
  },
  {
    "startMs": 61440,
    "text": "agent harnesses or maybe use cloud code"
  },
  {
    "startMs": 63120,
    "text": "with skills in a variety of different"
  },
  {
    "startMs": 64559,
    "text": "packages, but there's, you know, pretty"
  },
  {
    "startMs": 66320,
    "text": "strong differences in the way that you"
  },
  {
    "startMs": 67439,
    "text": "package these AI agents if you use them"
  },
  {
    "startMs": 69360,
    "text": "with different tool calls, etc. on a"
  },
  {
    "startMs": 71040,
    "text": "variety of different benchmarks. And so"
  },
  {
    "startMs": 72479,
    "text": "inspired by that, we're doing some work"
  },
  {
    "startMs": 74240,
    "text": "at Weights and Biases to build our own"
  },
  {
    "startMs": 75600,
    "text": "agent harness to do research for you in"
  },
  {
    "startMs": 77280,
    "text": "the weights and biases platform. Um, you"
  },
  {
    "startMs": 79360,
    "text": "know, and then there's probably some"
  },
  {
    "startMs": 80799,
    "text": "other things that I don't need to"
  },
  {
    "startMs": 81680,
    "text": "convince people of as to like why we"
  },
  {
    "startMs": 83520,
    "text": "want software agents and why a software"
  },
  {
    "startMs": 85119,
    "text": "harness as opposed to like just letting"
  },
  {
    "startMs": 86560,
    "text": "the, you know, LM run free and call"
  },
  {
    "startMs": 88159,
    "text": "tools or whatever else. Uh, but that's"
  },
  {
    "startMs": 89759,
    "text": "not for here. Uh, but one big problem"
  },
  {
    "startMs": 92960,
    "text": "presents itself, which is basically the"
  },
  {
    "startMs": 94560,
    "text": "thing that I go to bed thinking about"
  },
  {
    "startMs": 96240,
    "text": "every day, which is, you know,"
  },
  {
    "startMs": 98079,
    "text": "benchmarks, evaluations, the agents, and"
  },
  {
    "startMs": 100240,
    "text": "how you configure them are all"
  },
  {
    "startMs": 101360,
    "text": "covariant. And so like if you're trying"
  },
  {
    "startMs": 103280,
    "text": "to apply principled evaluations to any"
  },
  {
    "startMs": 105280,
    "text": "sort of system that dynamically changes,"
  },
  {
    "startMs": 107200,
    "text": "you need to have good measurement to see"
  },
  {
    "startMs": 109360,
    "text": "how the system actually performs. And"
  },
  {
    "startMs": 111439,
    "text": "you really want to see how the system"
  },
  {
    "startMs": 112560,
    "text": "performs in both your production"
  },
  {
    "startMs": 113840,
    "text": "environment and your offline"
  },
  {
    "startMs": 115360,
    "text": "environment. Uh you know if you're RLP"
  },
  {
    "startMs": 117680,
    "text": "build from a long time ago, you might"
  },
  {
    "startMs": 119040,
    "text": "think about like the sim tore gap uh"
  },
  {
    "startMs": 121040,
    "text": "which is like basically how do we"
  },
  {
    "startMs": 122079,
    "text": "translate simulation environments into"
  },
  {
    "startMs": 123439,
    "text": "the real world environments etc. Uh and"
  },
  {
    "startMs": 126399,
    "text": "so there's sort of two things that I'll"
  },
  {
    "startMs": 127920,
    "text": "talk about uh and then give a live demo"
  },
  {
    "startMs": 129679,
    "text": "of which is weights and biases weave for"
  },
  {
    "startMs": 132000,
    "text": "agents which is basically in my opinion"
  },
  {
    "startMs": 133840,
    "text": "at least the best observability platform"
  },
  {
    "startMs": 135840,
    "text": "for both production and offline tracing"
  },
  {
    "startMs": 137760,
    "text": "of agents. So I use it predominantly"
  },
  {
    "startMs": 140319,
    "text": "from the offline side where I build"
  },
  {
    "startMs": 142000,
    "text": "simulation environments run the Arya"
  },
  {
    "startMs": 143760,
    "text": "agent over them and then track them in"
  },
  {
    "startMs": 145440,
    "text": "weave and my entire team does that as"
  },
  {
    "startMs": 147200,
    "text": "well. So I can see all of our"
  },
  {
    "startMs": 148400,
    "text": "performance in offline metrics. And then"
  },
  {
    "startMs": 150160,
    "text": "the other half of my team works on the"
  },
  {
    "startMs": 151920,
    "text": "production side of things, deploying the"
  },
  {
    "startMs": 153280,
    "text": "agent, logging the same things in the"
  },
  {
    "startMs": 154879,
    "text": "exact same format so that I can rip"
  },
  {
    "startMs": 156640,
    "text": "those production traces into our"
  },
  {
    "startMs": 158080,
    "text": "environments and then hill climb on them"
  },
  {
    "startMs": 159680,
    "text": "or resolve our errors. And it's a pretty"
  },
  {
    "startMs": 161599,
    "text": "nice flywheel. And then obviously the"
  },
  {
    "startMs": 163120,
    "text": "other thing that I'm here to talk about"
  },
  {
    "startMs": 164239,
    "text": "is Arya, the weights and devices agent"
  },
  {
    "startMs": 165920,
    "text": "itself, which is basically the thing"
  },
  {
    "startMs": 167440,
    "text": "that we use to now build itself because"
  },
  {
    "startMs": 169120,
    "text": "it's sophisticated enough that it can"
  },
  {
    "startMs": 170400,
    "text": "actually do that offline hill climbing"
  },
  {
    "startMs": 172319,
    "text": "uh by itself. So I'm going to just jump"
  },
  {
    "startMs": 174319,
    "text": "into a super quick live demo and then"
  },
  {
    "startMs": 175760,
    "text": "we'll talk through some other slides uh"
  },
  {
    "startMs": 177519,
    "text": "about how we built the offline"
  },
  {
    "startMs": 178800,
    "text": "evaluation framework. So if you hop into"
  },
  {
    "startMs": 180720,
    "text": "the weights and biases platform, this is"
  },
  {
    "startMs": 182319,
    "text": "a demo project that I called Arya"
  },
  {
    "startMs": 184319,
    "text": "researches Arya here. Uh and I'm just"
  },
  {
    "startMs": 186400,
    "text": "going to chat to Arya and ask it to"
  },
  {
    "startMs": 188080,
    "text": "basically do auto research for itself."
  },
  {
    "startMs": 190000,
    "text": "And so this is a relatively long prompt"
  },
  {
    "startMs": 191599,
    "text": "that I had it generate in actually one"
  },
  {
    "startMs": 192959,
    "text": "of these other sessions. Um but if we"
  },
  {
    "startMs": 195200,
    "text": "look at it, what it's going to do is"
  },
  {
    "startMs": 196480,
    "text": "it's going to take my codebase that is"
  },
  {
    "startMs": 198080,
    "text": "logged as a weights and biases artifact"
  },
  {
    "startMs": 199920,
    "text": "uh which we can see over here. It's"
  },
  {
    "startMs": 202480,
    "text": "going to launch training jobs uh against"
  },
  {
    "startMs": 204640,
    "text": "this codebase which is our offline"
  },
  {
    "startMs": 205920,
    "text": "evaluation framework. It's going to"
  },
  {
    "startMs": 207519,
    "text": "review production traces that we have,"
  },
  {
    "startMs": 209200,
    "text": "add new tasks to hill climb against uh"
  },
  {
    "startMs": 211519,
    "text": "and do stuff like that. And so, you"
  },
  {
    "startMs": 213200,
    "text": "know, this thing's going to cook for a"
  },
  {
    "startMs": 214560,
    "text": "little bit. The evaluations take a"
  },
  {
    "startMs": 215680,
    "text": "little bit of time to run. And so, we'll"
  },
  {
    "startMs": 217120,
    "text": "see Arya uh basically try to write a new"
  },
  {
    "startMs": 219120,
    "text": "variant for itself. The other side of"
  },
  {
    "startMs": 221920,
    "text": "this equation, right, is the production"
  },
  {
    "startMs": 223680,
    "text": "side of things. And so, this is a"
  },
  {
    "startMs": 225200,
    "text": "sanitized view of internal customer"
  },
  {
    "startMs": 227200,
    "text": "traces or internal traces on Arya. So"
  },
  {
    "startMs": 229840,
    "text": "we'll actually see very quickly uh that"
  },
  {
    "startMs": 231680,
    "text": "Arya will actually pop up here the exact"
  },
  {
    "startMs": 233280,
    "text": "conversation that we're running um right"
  },
  {
    "startMs": 235519,
    "text": "here. But in this trace I'm going to use"
  },
  {
    "startMs": 238480,
    "text": "this as an opportunity to take one of"
  },
  {
    "startMs": 240000,
    "text": "these production traces that are logged"
  },
  {
    "startMs": 241360,
    "text": "in weave and put them into my offline"
  },
  {
    "startMs": 243760,
    "text": "evaluation framework so that I can hill"
  },
  {
    "startMs": 245120,
    "text": "climb on them. And so I'm just going to"
  },
  {
    "startMs": 246319,
    "text": "do that live. We'll ask you know for a"
  },
  {
    "startMs": 247920,
    "text": "relatively simple task. Um this is going"
  },
  {
    "startMs": 249760,
    "text": "to be a little interesting because it's"
  },
  {
    "startMs": 250879,
    "text": "going to be slightly self-referential"
  },
  {
    "startMs": 252959,
    "text": "but I'll go here. I'll pop up in a new"
  },
  {
    "startMs": 255040,
    "text": "Arya tab and I'll just ask it hey take"
  },
  {
    "startMs": 257280,
    "text": "this trace log it into our offline evals"
  },
  {
    "startMs": 262160,
    "text": "I don't think spelling matters too much"
  },
  {
    "startMs": 263919,
    "text": "and then run the candidate and"
  },
  {
    "startMs": 266560,
    "text": "production agent on it"
  },
  {
    "startMs": 269600,
    "text": "and we'll have it cook for that as well"
  },
  {
    "startMs": 271120,
    "text": "and we'll see those evaluations get"
  },
  {
    "startMs": 272320,
    "text": "logged into weave as they're running um"
  },
  {
    "startMs": 274479,
    "text": "so that's basically the tight version of"
  },
  {
    "startMs": 276560,
    "text": "what I do basically every day which is"
  },
  {
    "startMs": 278320,
    "text": "look at production traces in our weights"
  },
  {
    "startMs": 279919,
    "text": "and biases weave platform run"
  },
  {
    "startMs": 281440,
    "text": "evaluations in our codebase which is our"
  },
  {
    "startMs": 283600,
    "text": "offline client evaluation sandbox uh and"
  },
  {
    "startMs": 285520,
    "text": "then deploy new versions of the agent uh"
  },
  {
    "startMs": 287600,
    "text": "and work with my team to do that. And if"
  },
  {
    "startMs": 289680,
    "text": "we look at our production project, this"
  },
  {
    "startMs": 291120,
    "text": "is, you know, a bunch of traces that we"
  },
  {
    "startMs": 292400,
    "text": "have over the last seven weeks of our"
  },
  {
    "startMs": 293600,
    "text": "nightly uh like CI jobs that run where"
  },
  {
    "startMs": 295759,
    "text": "we evaluate the agent in its production"
  },
  {
    "startMs": 297280,
    "text": "format and other candidate variants that"
  },
  {
    "startMs": 298800,
    "text": "we cut. And so we can see our relative"
  },
  {
    "startMs": 300479,
    "text": "performance over time. Uh like you know,"
  },
  {
    "startMs": 302479,
    "text": "for a while the CI broke, we had Arya"
  },
  {
    "startMs": 304400,
    "text": "have to fix itself last night, but you"
  },
  {
    "startMs": 305759,
    "text": "know, we're getting about like 66%"
  },
  {
    "startMs": 307120,
    "text": "performance on some of the tasks, etc."
  },
  {
    "startMs": 308960,
    "text": "And so we have a variety of tasks um"
  },
  {
    "startMs": 311199,
    "text": "that we train the agent to do. And so"
  },
  {
    "startMs": 313120,
    "text": "now I'm just going to jump into a"
  },
  {
    "startMs": 314639,
    "text": "material version of the presentation to"
  },
  {
    "startMs": 316080,
    "text": "go a little bit deeper into the theory"
  },
  {
    "startMs": 318000,
    "text": "behind building a simulation environment"
  },
  {
    "startMs": 319759,
    "text": "for offline hill climbing for basically"
  },
  {
    "startMs": 321600,
    "text": "building a software agent where"
  },
  {
    "startMs": 323120,
    "text": "traditionally you might expect that you"
  },
  {
    "startMs": 324560,
    "text": "are really robust if you're doing"
  },
  {
    "startMs": 326160,
    "text": "reinforcement learning but I mostly"
  },
  {
    "startMs": 328000,
    "text": "prompt engineer these days because the"
  },
  {
    "startMs": 329840,
    "text": "sophisticated models are relatively good"
  },
  {
    "startMs": 331520,
    "text": "at performing tasks at weights and"
  },
  {
    "startMs": 332800,
    "text": "biases and so we're really working on"
  },
  {
    "startMs": 334160,
    "text": "building skills for the software agent"
  },
  {
    "startMs": 335680,
    "text": "more than doing reinforcement learning."
  },
  {
    "startMs": 337039,
    "text": "But I think applying the same"
  },
  {
    "startMs": 338080,
    "text": "methodology to how you build the agent."
  },
  {
    "startMs": 340400,
    "text": "So the same robustness of how you"
  },
  {
    "startMs": 342000,
    "text": "simulate environments uh is really"
  },
  {
    "startMs": 343680,
    "text": "helpful for our work. So we benchmark a"
  },
  {
    "startMs": 346639,
    "text": "bite-wise identical version of the agent"
  },
  {
    "startMs": 348080,
    "text": "in our production environment and our"
  },
  {
    "startMs": 349600,
    "text": "simulated environment. There's a lot of"
  },
  {
    "startMs": 351039,
    "text": "text on these slides that I and Claude"
  },
  {
    "startMs": 352320,
    "text": "sort of slop out. Uh but you know we'll"
  },
  {
    "startMs": 354000,
    "text": "just go for the graphics. Um so I think"
  },
  {
    "startMs": 356400,
    "text": "the exciting part here is that both"
  },
  {
    "startMs": 358320,
    "text": "using the logging framework from weave"
  },
  {
    "startMs": 359919,
    "text": "as well as just in the way that we've"
  },
  {
    "startMs": 361039,
    "text": "designed the system our research and"
  },
  {
    "startMs": 362560,
    "text": "production code are exactly the same."
  },
  {
    "startMs": 363840,
    "text": "And in fact, there's like a 4-hour sync"
  },
  {
    "startMs": 365520,
    "text": "job that happens between production to"
  },
  {
    "startMs": 367120,
    "text": "our research environment so that we"
  },
  {
    "startMs": 368319,
    "text": "don't get any drift when you, you know,"
  },
  {
    "startMs": 370000,
    "text": "researchers are cutting new variants of"
  },
  {
    "startMs": 371360,
    "text": "the agent, new skills, etc., and doing"
  },
  {
    "startMs": 372880,
    "text": "their hill climbing. And so, we have"
  },
  {
    "startMs": 374160,
    "text": "this nice little tight loop that"
  },
  {
    "startMs": 375440,
    "text": "basically mirror each other on two sides"
  },
  {
    "startMs": 376800,
    "text": "of the stack from our deployment layer"
  },
  {
    "startMs": 378319,
    "text": "and our offline benchmarking layer. Uh,"
  },
  {
    "startMs": 381280,
    "text": "and then, you know, we expose internally"
  },
  {
    "startMs": 383759,
    "text": "as a team a bunch of run eval commands"
  },
  {
    "startMs": 385919,
    "text": "to generate score trajectories. And so"
  },
  {
    "startMs": 387840,
    "text": "that's what we were talking about with"
  },
  {
    "startMs": 389199,
    "text": "this like large project where we have a"
  },
  {
    "startMs": 390880,
    "text": "bunch of trajectory examples where if I"
  },
  {
    "startMs": 392400,
    "text": "click into one of these, you know, we"
  },
  {
    "startMs": 393919,
    "text": "can see like our relative scores of a"
  },
  {
    "startMs": 395440,
    "text": "variety of different metrics when it"
  },
  {
    "startMs": 396639,
    "text": "loads. Um, but essentially what we get"
  },
  {
    "startMs": 399360,
    "text": "is these super robust traces of the"
  },
  {
    "startMs": 401520,
    "text": "agent rolling out over time. And there's"
  },
  {
    "startMs": 403039,
    "text": "a lot of signal to reap from this. And I"
  },
  {
    "startMs": 404720,
    "text": "think the thesis behind this is generate"
  },
  {
    "startMs": 406479,
    "text": "a ton of traces and then decide"
  },
  {
    "startMs": 408560,
    "text": "what you're going to do with said"
  },
  {
    "startMs": 409919,
    "text": "ton of traces as you try to figure out"
  },
  {
    "startMs": 411840,
    "text": "how the agent performs. you're measuring"
  },
  {
    "startMs": 413919,
    "text": "emergent properties and then trying to"
  },
  {
    "startMs": 415680,
    "text": "align the agent in particular"
  },
  {
    "startMs": 416960,
    "text": "directions. And so the two ways that"
  },
  {
    "startMs": 418479,
    "text": "that happens is I look manually at the"
  },
  {
    "startMs": 420560,
    "text": "tasks or like we would do in that first"
  },
  {
    "startMs": 422800,
    "text": "example, I would ask Arya to review the"
  },
  {
    "startMs": 424720,
    "text": "roll out that it generated for itself or"
  },
  {
    "startMs": 426479,
    "text": "review some other rollout and decide"
  },
  {
    "startMs": 428080,
    "text": "what went wrong or what went well and"
  },
  {
    "startMs": 429680,
    "text": "try to reinforce that behavior through"
  },
  {
    "startMs": 431120,
    "text": "prompting or anything else. Uh and then"
  },
  {
    "startMs": 434160,
    "text": "you know the actual method behind the"
  },
  {
    "startMs": 436720,
    "text": "agent is relatively simple. We want to"
  },
  {
    "startMs": 438560,
    "text": "test on a variety of the different"
  },
  {
    "startMs": 440240,
    "text": "models both models offered through"
  },
  {
    "startMs": 441919,
    "text": "corewave inference models offered"
  },
  {
    "startMs": 443360,
    "text": "through you know the foundation model"
  },
  {
    "startMs": 444479,
    "text": "players etc. And so we define you know a"
  },
  {
    "startMs": 446639,
    "text": "relatively agnostic software stack for"
  },
  {
    "startMs": 448560,
    "text": "how we treat compaction and how we"
  },
  {
    "startMs": 450160,
    "text": "prepare context and how we assemble UI"
  },
  {
    "startMs": 452160,
    "text": "payloads etc. And the again mental"
  },
  {
    "startMs": 454639,
    "text": "pattern behind this is to really try to"
  },
  {
    "startMs": 456639,
    "text": "make it very very simple to have lots of"
  },
  {
    "startMs": 459440,
    "text": "mutations of the exact same"
  },
  {
    "startMs": 460639,
    "text": "configuration. You want to basically"
  },
  {
    "startMs": 462240,
    "text": "YAML define different configurations of"
  },
  {
    "startMs": 463919,
    "text": "the agent to sort of get multiple"
  },
  {
    "startMs": 465759,
    "text": "parallel uh variants and then test them"
  },
  {
    "startMs": 467680,
    "text": "all and see what happens because"
  },
  {
    "startMs": 469199,
    "text": "ultimately you can have insight into how"
  },
  {
    "startMs": 471039,
    "text": "to make the problem better. But if we"
  },
  {
    "startMs": 472479,
    "text": "take an adage from old reinforcement"
  },
  {
    "startMs": 474080,
    "text": "learning training or just simple model"
  },
  {
    "startMs": 475599,
    "text": "training, it's just better to run more"
  },
  {
    "startMs": 477919,
    "text": "experiments than fewer. And so that's"
  },
  {
    "startMs": 480000,
    "text": "the thesis behind the agent harness"
  },
  {
    "startMs": 481440,
    "text": "itself too. And so then you know we talk"
  },
  {
    "startMs": 484639,
    "text": "about like the sandbox stack behind the"
  },
  {
    "startMs": 486720,
    "text": "agent. And I think that you know there's"
  },
  {
    "startMs": 488319,
    "text": "a lot of mentions of code mode here at"
  },
  {
    "startMs": 490080,
    "text": "this conference and other things like"
  },
  {
    "startMs": 491199,
    "text": "that. And one thing that we really want"
  },
  {
    "startMs": 492639,
    "text": "to see happen because we want to build"
  },
  {
    "startMs": 493919,
    "text": "the best product for our customers is to"
  },
  {
    "startMs": 495759,
    "text": "just have a sandbox environment that"
  },
  {
    "startMs": 497520,
    "text": "they can do anything they wanted. So for"
  },
  {
    "startMs": 499520,
    "text": "example prior to yesterday I wasn't"
  },
  {
    "startMs": 502319,
    "text": "entirely certain that Arya was going to"
  },
  {
    "startMs": 503840,
    "text": "be able to do a bunch of parallel"
  },
  {
    "startMs": 505440,
    "text": "executions of itself. And so I asked it"
  },
  {
    "startMs": 507759,
    "text": "both while I was writing this"
  },
  {
    "startMs": 508879,
    "text": "presentation and preparing for this to"
  },
  {
    "startMs": 510560,
    "text": "just create the sandbox environment uh"
  },
  {
    "startMs": 512640,
    "text": "for itself to run a bunch of parallel"
  },
  {
    "startMs": 514719,
    "text": "executions of its own research loop."
  },
  {
    "startMs": 516560,
    "text": "Right. Uh, and so like having that"
  },
  {
    "startMs": 518240,
    "text": "unconstrained environment is really"
  },
  {
    "startMs": 519839,
    "text": "useful for getting the agent to do"
  },
  {
    "startMs": 521599,
    "text": "emergent things."
  },
  {
    "startMs": 524399,
    "text": "And so, you know, I I'm not really sure"
  },
  {
    "startMs": 526000,
    "text": "what the six phrases per record uh"
  },
  {
    "startMs": 527920,
    "text": "means, but I think from a theory"
  },
  {
    "startMs": 530560,
    "text": "standpoint, we want to build then a good"
  },
  {
    "startMs": 533040,
    "text": "pattern for generating these simulation"
  },
  {
    "startMs": 534800,
    "text": "environments. And so we have a"
  },
  {
    "startMs": 536560,
    "text": "relatively agnostic DAG that you might"
  },
  {
    "startMs": 538320,
    "text": "think about in a traditional machine"
  },
  {
    "startMs": 539440,
    "text": "learning context, which is like we take"
  },
  {
    "startMs": 540959,
    "text": "a configuration, which are these YAML"
  },
  {
    "startMs": 542560,
    "text": "files. We hydrate them. So we load the"
  },
  {
    "startMs": 544560,
    "text": "live data that we need to. We set up an"
  },
  {
    "startMs": 546399,
    "text": "environment and the environments and"
  },
  {
    "startMs": 547760,
    "text": "weights and biases are relatively"
  },
  {
    "startMs": 549040,
    "text": "expensive because we have lots of"
  },
  {
    "startMs": 550800,
    "text": "production data that we might want to"
  },
  {
    "startMs": 552000,
    "text": "test against or other things like that."
  },
  {
    "startMs": 553440,
    "text": "These are full machine learning training"
  },
  {
    "startMs": 554800,
    "text": "logs. We might have to simulate uh GPU"
  },
  {
    "startMs": 556959,
    "text": "executions if you're doing auto research"
  },
  {
    "startMs": 558720,
    "text": "etc. That's a relatively intensive"
  },
  {
    "startMs": 560320,
    "text": "process and so you might want to"
  },
  {
    "startMs": 561279,
    "text": "parallelize that. Then you would"
  },
  {
    "startMs": 562880,
    "text": "rehydrate because oftentimes there's"
  },
  {
    "startMs": 564320,
    "text": "runtime configurations that you can't"
  },
  {
    "startMs": 565920,
    "text": "encode in a YAML specification and so"
  },
  {
    "startMs": 568160,
    "text": "you hot patch that data back into the"
  },
  {
    "startMs": 570000,
    "text": "config config. Uh and then you run the"
  },
  {
    "startMs": 572240,
    "text": "agent and running the agent is trivial."
  },
  {
    "startMs": 573760,
    "text": "You run it, as I mentioned at the"
  },
  {
    "startMs": 575040,
    "text": "beginning, in the bite-wise identical"
  },
  {
    "startMs": 576480,
    "text": "version that you have in production, and"
  },
  {
    "startMs": 578320,
    "text": "then you score it. And scoring is where"
  },
  {
    "startMs": 580000,
    "text": "I spend a lot of my time, which is"
  },
  {
    "startMs": 581279,
    "text": "basically trying to think about the"
  },
  {
    "startMs": 582480,
    "text": "robustness of how you measure results."
  },
  {
    "startMs": 584880,
    "text": "Uh, and I imagine most people here are"
  },
  {
    "startMs": 586560,
    "text": "pretty good at thinking about things"
  },
  {
    "startMs": 587680,
    "text": "like that. And I think it's really worth"
  },
  {
    "startMs": 589360,
    "text": "spending the time once you have a"
  },
  {
    "startMs": 590560,
    "text": "configuration of how to run evaluations,"
  },
  {
    "startMs": 592720,
    "text": "thinking about the robustness of our"
  },
  {
    "startMs": 593920,
    "text": "emails. uh a Slack message that I"
  },
  {
    "startMs": 595920,
    "text": "probably can't show live on the screen"
  },
  {
    "startMs": 597040,
    "text": "that I got from one of my teammates"
  },
  {
    "startMs": 598160,
    "text": "today made me very happy where he"
  },
  {
    "startMs": 599680,
    "text": "basically just spent all of yesterday"
  },
  {
    "startMs": 600959,
    "text": "thinking about the health of our"
  },
  {
    "startMs": 602240,
    "text": "evaluations and the drift between our"
  },
  {
    "startMs": 604000,
    "text": "evaluations and production. Uh and I"
  },
  {
    "startMs": 605760,
    "text": "think that kind of mental model of like"
  },
  {
    "startMs": 607360,
    "text": "you know why are things working well why"
  },
  {
    "startMs": 608800,
    "text": "are things not working well? What are"
  },
  {
    "startMs": 610080,
    "text": "the gaps that we see in these two"
  },
  {
    "startMs": 611279,
    "text": "patterns is really useful. Um in our"
  },
  {
    "startMs": 613760,
    "text": "case weights and biases Arya scores"
  },
  {
    "startMs": 615680,
    "text": "itself in two patterns which is"
  },
  {
    "startMs": 616880,
    "text": "normatively which gives us basically did"
  },
  {
    "startMs": 618800,
    "text": "we pass a task or not and then"
  },
  {
    "startMs": 620399,
    "text": "relativistically where we can set styles"
  },
  {
    "startMs": 622480,
    "text": "based on one variant where it asks"
  },
  {
    "startMs": 624399,
    "text": "questions to the user and one variant"
  },
  {
    "startMs": 625760,
    "text": "where it doesn't ask questions to the"
  },
  {
    "startMs": 626800,
    "text": "user and we can sort of see which one"
  },
  {
    "startMs": 628399,
    "text": "behaves better uh with a relative"
  },
  {
    "startMs": 630160,
    "text": "scoring. So a pretty traditional"
  },
  {
    "startMs": 631680,
    "text": "formulation from a reinforcement"
  },
  {
    "startMs": 633120,
    "text": "learning standpoint. And then of course"
  },
  {
    "startMs": 634880,
    "text": "if you're running this in parallel you"
  },
  {
    "startMs": 636480,
    "text": "got to tear it down because you want to"
  },
  {
    "startMs": 637760,
    "text": "start it again and you don't want to"
  },
  {
    "startMs": 638959,
    "text": "clobber your teammates's work etc. Uh"
  },
  {
    "startMs": 640800,
    "text": "and so we'll see what happens in this"
  },
  {
    "startMs": 642160,
    "text": "project because a bunch of my teammates"
  },
  {
    "startMs": 643279,
    "text": "are running eval right now and so we'll"
  },
  {
    "startMs": 644880,
    "text": "see how Arya performs."
  },
  {
    "startMs": 647360,
    "text": "Uh and then you know more traditionally"
  },
  {
    "startMs": 649600,
    "text": "on tasks I think the eval tasks are just"
  },
  {
    "startMs": 651760,
    "text": "YAML specifications that we define as a"
  },
  {
    "startMs": 653839,
    "text": "starting condition of an environment"
  },
  {
    "startMs": 655440,
    "text": "with a bunch of user configurations as"
  },
  {
    "startMs": 657279,
    "text": "well as then an ending condition that we"
  },
  {
    "startMs": 658640,
    "text": "sort of want to get to. Uh in our case"
  },
  {
    "startMs": 660880,
    "text": "tasks are flows from users. So we"
  },
  {
    "startMs": 662640,
    "text": "simulate them in three ways. Uh we do"
  },
  {
    "startMs": 664720,
    "text": "just simple text which is like hey"
  },
  {
    "startMs": 666560,
    "text": "here's an instruction to Arya. You might"
  },
  {
    "startMs": 668160,
    "text": "imagine that the task that it generates"
  },
  {
    "startMs": 669920,
    "text": "in this chat over here um where it's"
  },
  {
    "startMs": 672320,
    "text": "logging a specific trace from production"
  },
  {
    "startMs": 674640,
    "text": "uh into our local framework actually"
  },
  {
    "startMs": 676880,
    "text": "looks like you know just a simple"
  },
  {
    "startMs": 679120,
    "text": "question that I asked it basically which"
  },
  {
    "startMs": 680560,
    "text": "was that question of Arya doing auto"
  },
  {
    "startMs": 681920,
    "text": "research itself um we can just ask it to"
  },
  {
    "startMs": 684240,
    "text": "continue"
  },
  {
    "startMs": 686959,
    "text": "um as it does that and that task is just"
  },
  {
    "startMs": 689680,
    "text": "a single question to it there's other"
  },
  {
    "startMs": 691360,
    "text": "variants of the tasks that exist as well"
  },
  {
    "startMs": 693279,
    "text": "where we simulate a language model that"
  },
  {
    "startMs": 695920,
    "text": "has a persona from a user and we ask it,"
  },
  {
    "startMs": 698000,
    "text": "hey, pretend to be this user, ask"
  },
  {
    "startMs": 699440,
    "text": "certain questions in a particular order"
  },
  {
    "startMs": 700640,
    "text": "so that we can simulate multi-turn"
  },
  {
    "startMs": 702160,
    "text": "environments and interactions with our"
  },
  {
    "startMs": 703519,
    "text": "agent. Uh, and then you just run a bunch"
  },
  {
    "startMs": 705040,
    "text": "of these. So, we have 886 tasks. We"
  },
  {
    "startMs": 707040,
    "text": "categorize them by levels. We expose"
  },
  {
    "startMs": 708560,
    "text": "them to our product team so that they"
  },
  {
    "startMs": 709920,
    "text": "can decide whether or not the tasks are"
  },
  {
    "startMs": 711279,
    "text": "good enough or they reflect things that"
  },
  {
    "startMs": 712560,
    "text": "we care about from our benchmarks, etc."
  },
  {
    "startMs": 714640,
    "text": "And then we just run them a bunch of"
  },
  {
    "startMs": 715839,
    "text": "times."
  },
  {
    "startMs": 717360,
    "text": "Yeah. And then, you know, the ultimate"
  },
  {
    "startMs": 719040,
    "text": "thing here is that the trajectory is"
  },
  {
    "startMs": 720480,
    "text": "like the meat and the data that I live"
  },
  {
    "startMs": 722000,
    "text": "behind. And so I look at our production"
  },
  {
    "startMs": 724320,
    "text": "traces that exist in our traffic and our"
  },
  {
    "startMs": 725839,
    "text": "offline traces that exist. And I really"
  },
  {
    "startMs": 727760,
    "text": "try to exploit the behavior patterns"
  },
  {
    "startMs": 729440,
    "text": "from both of those. And so we build"
  },
  {
    "startMs": 731040,
    "text": "tooling ourselves to understand our"
  },
  {
    "startMs": 733120,
    "text": "traces. Uh we have like this behavior"
  },
  {
    "startMs": 734720,
    "text": "trace project etc that we run in both"
  },
  {
    "startMs": 736720,
    "text": "prod and offline."
  },
  {
    "startMs": 739519,
    "text": "And yeah that's the eval flywheel. So"
  },
  {
    "startMs": 741600,
    "text": "you know fundamentally every production"
  },
  {
    "startMs": 743200,
    "text": "miss or every production goodness as"
  },
  {
    "startMs": 744800,
    "text": "well because I think it's useful to hill"
  },
  {
    "startMs": 746000,
    "text": "climb in a positive direction uh becomes"
  },
  {
    "startMs": 747920,
    "text": "a task for our agent framework. And so"
  },
  {
    "startMs": 749600,
    "text": "we use that to sort of define the agent"
  },
  {
    "startMs": 751279,
    "text": "and make it better."
  },
  {
    "startMs": 753760,
    "text": "And these are, you know, our real scores"
  },
  {
    "startMs": 755360,
    "text": "straight from weave. So we have a bunch"
  },
  {
    "startMs": 757440,
    "text": "of different categories that we measure"
  },
  {
    "startMs": 758639,
    "text": "ourselves on. I think the agent is"
  },
  {
    "startMs": 760000,
    "text": "really good at conceptually guiding you."
  },
  {
    "startMs": 761680,
    "text": "We want to make it better at doing some"
  },
  {
    "startMs": 763040,
    "text": "error analysis for projects. You know,"
  },
  {
    "startMs": 764480,
    "text": "we try to make these as difficult as"
  },
  {
    "startMs": 766079,
    "text": "possible for the agent to really test"
  },
  {
    "startMs": 767680,
    "text": "the metal of how this thing performs."
  },
  {
    "startMs": 770560,
    "text": "And so I'll hop back to Arya now."
  },
  {
    "startMs": 772160,
    "text": "There's sort of five minutes left um"
  },
  {
    "startMs": 773920,
    "text": "with sort of a demo of what it's"
  },
  {
    "startMs": 775680,
    "text": "actually doing. So right now it's going"
  },
  {
    "startMs": 776800,
    "text": "to run a bunch of code as it's trying to"
  },
  {
    "startMs": 778000,
    "text": "rip this production trace and log it"
  },
  {
    "startMs": 779760,
    "text": "into the framework. If I hop into this"
  },
  {
    "startMs": 782000,
    "text": "uh live demo that I just triggered with"
  },
  {
    "startMs": 783600,
    "text": "that nice long prompt, you know, Arya is"
  },
  {
    "startMs": 785519,
    "text": "going to tell me the research runs that"
  },
  {
    "startMs": 786800,
    "text": "it did. So we can click through some of"
  },
  {
    "startMs": 788000,
    "text": "these links. Uh I asked it to write a"
  },
  {
    "startMs": 790000,
    "text": "report using weights and biases reports"
  },
  {
    "startMs": 791760,
    "text": "from the uh run that it just had. If we"
  },
  {
    "startMs": 794560,
    "text": "scrub this, we can see that this is the"
  },
  {
    "startMs": 795839,
    "text": "most recent production trace that it"
  },
  {
    "startMs": 797040,
    "text": "just logged. It wrote itself a new task,"
  },
  {
    "startMs": 799200,
    "text": "ran it, and then scored it. And so this"
  },
  {
    "startMs": 801120,
    "text": "is like the production variant getting"
  },
  {
    "startMs": 802240,
    "text": "benchmarked on our specific task. If we"
  },
  {
    "startMs": 804079,
    "text": "look at the report that it generated,"
  },
  {
    "startMs": 805600,
    "text": "this is the demo that I asked it to do."
  },
  {
    "startMs": 807360,
    "text": "Um, so you know, it ran a real womb"
  },
  {
    "startMs": 810079,
    "text": "agent trace, turned it into a WBF"
  },
  {
    "startMs": 812320,
    "text": "regression task. WBAF is the factory"
  },
  {
    "startMs": 814800,
    "text": "that I build, which is the weights and"
  },
  {
    "startMs": 816240,
    "text": "biases agent factory. That's the"
  },
  {
    "startMs": 817600,
    "text": "principle behind our offline"
  },
  {
    "startMs": 818720,
    "text": "benchmarking. It identified that the"
  },
  {
    "startMs": 820720,
    "text": "problem was that we weren't calling"
  },
  {
    "startMs": 821760,
    "text": "weave.log, one of our SDK calls properly"
  },
  {
    "startMs": 823839,
    "text": "in the sandbox. It replicates the source"
  },
  {
    "startMs": 825920,
    "text": "trace, which we can click into here, uh,"
  },
  {
    "startMs": 827760,
    "text": "and then runs a bunch of variants of the"
  },
  {
    "startMs": 829200,
    "text": "agent to see how it makes itself better."
  },
  {
    "startMs": 830720,
    "text": "And so, you know, it added a hill climb"
  },
  {
    "startMs": 832959,
    "text": "target uh which was basically like"
  },
  {
    "startMs": 834560,
    "text": "here's what to do to fix it. Ran the"
  },
  {
    "startMs": 836560,
    "text": "trace and then we can see if it gives us"
  },
  {
    "startMs": 838000,
    "text": "a little explanation of exactly what"
  },
  {
    "startMs": 839760,
    "text": "happened at the end. Uh how did the prod"
  },
  {
    "startMs": 843199,
    "text": "versus candidate variant perform"
  },
  {
    "startMs": 847519,
    "text": "and you know this is now what I live in."
  },
  {
    "startMs": 849440,
    "text": "And so I think the flywheel here is"
  },
  {
    "startMs": 851600,
    "text": "exceptionally compelling for me where"
  },
  {
    "startMs": 853279,
    "text": "instead of going back to my cloud code"
  },
  {
    "startMs": 854880,
    "text": "and writing offline benchmarks and"
  },
  {
    "startMs": 856320,
    "text": "trying to think about what I want the"
  },
  {
    "startMs": 857279,
    "text": "agent to do, I'm just going to live in"
  },
  {
    "startMs": 858800,
    "text": "this platform instead and hop between my"
  },
  {
    "startMs": 861040,
    "text": "production tracing project where I'm"
  },
  {
    "startMs": 862560,
    "text": "looking at traces roll in as they come"
  },
  {
    "startMs": 864320,
    "text": "through over time where AR is, you know,"
  },
  {
    "startMs": 866000,
    "text": "building itself and then hop back into"
  },
  {
    "startMs": 867760,
    "text": "my offline evaluation project to sort of"
  },
  {
    "startMs": 869519,
    "text": "see what my team is doing, what I'm"
  },
  {
    "startMs": 871040,
    "text": "doing, and what changes I want to make."
  },
  {
    "startMs": 872880,
    "text": "what change did we make to the prod"
  },
  {
    "startMs": 875440,
    "text": "skill or to the agent variant"
  },
  {
    "startMs": 879120,
    "text": "etc. And you know to sort of like watch"
  },
  {
    "startMs": 882560,
    "text": "this thing roll out. I think a big"
  },
  {
    "startMs": 884079,
    "text": "challenge that we have is it's really"
  },
  {
    "startMs": 885760,
    "text": "easy to go like auto mode for some of"
  },
  {
    "startMs": 887440,
    "text": "these tasks where you want to just see"
  },
  {
    "startMs": 888560,
    "text": "the agent do everything. And I haven't"
  },
  {
    "startMs": 890639,
    "text": "written a line of code in maybe eight"
  },
  {
    "startMs": 891920,
    "text": "months because I just tell Claude to"
  },
  {
    "startMs": 893120,
    "text": "write all my code for me. And that's a"
  },
  {
    "startMs": 894720,
    "text": "really nice pattern, but it doesn't"
  },
  {
    "startMs": 896079,
    "text": "absolve yourself of the thought behind"
  },
  {
    "startMs": 897760,
    "text": "how we want to make improvements. And I"
  },
  {
    "startMs": 899600,
    "text": "think using these tools to improve"
  },
  {
    "startMs": 901279,
    "text": "themselves is very valuable because you"
  },
  {
    "startMs": 902880,
    "text": "get to spend more of your time in the"
  },
  {
    "startMs": 904560,
    "text": "gray of actually trying to think how we"
  },
  {
    "startMs": 906320,
    "text": "make this system better, right? And so"
  },
  {
    "startMs": 908160,
    "text": "if you're building an agent, I implore"
  },
  {
    "startMs": 909600,
    "text": "you to use weights and biases to both"
  },
  {
    "startMs": 910959,
    "text": "track that agent etc. Uh we have a"
  },
  {
    "startMs": 912959,
    "text": "really excellent model tracking platform"
  },
  {
    "startMs": 914480,
    "text": "as well. So some of the other demos that"
  },
  {
    "startMs": 916160,
    "text": "we've shown here are Arya training"
  },
  {
    "startMs": 917680,
    "text": "machine learning models on H200's"
  },
  {
    "startMs": 919600,
    "text": "running on core infrastructure and doing"
  },
  {
    "startMs": 921440,
    "text": "auto research for Karpathy's nano chat"
  },
  {
    "startMs": 923360,
    "text": "etc or full-scale production projects."
  },
  {
    "startMs": 925519,
    "text": "But I think the exciting thing in this"
  },
  {
    "startMs": 926959,
    "text": "case for me is that I don't need to"
  },
  {
    "startMs": 928560,
    "text": "leave this platform in order to do my"
  },
  {
    "startMs": 930000,
    "text": "job now right. I have full observability"
  },
  {
    "startMs": 931920,
    "text": "from a production standpoint. I have"
  },
  {
    "startMs": 933279,
    "text": "full observability from an offline"
  },
  {
    "startMs": 934639,
    "text": "standpoint. And if we look at, you know,"
  },
  {
    "startMs": 935920,
    "text": "the things that Arya is saying, it's"
  },
  {
    "startMs": 937600,
    "text": "running lots of evaluations on my behalf"
  },
  {
    "startMs": 939519,
    "text": "to improve itself. And that auto mode"
  },
  {
    "startMs": 941839,
    "text": "that I just mentioned is really exciting"
  },
  {
    "startMs": 944079,
    "text": "because now I get to spend all of my"
  },
  {
    "startMs": 945600,
    "text": "time thinking about how I can make this"
  },
  {
    "startMs": 946959,
    "text": "system reinforce itself better. The"
  },
  {
    "startMs": 948320,
    "text": "tight little guard rails that I can put"
  },
  {
    "startMs": 949680,
    "text": "around it to make it a more useful tool"
  },
  {
    "startMs": 951600,
    "text": "for all of you and then inherently for"
  },
  {
    "startMs": 953839,
    "text": "me. Um, but yeah, that's a that's a"
  },
  {
    "startMs": 956639,
    "text": "quick demo. We can we can look at the"
  },
  {
    "startMs": 958079,
    "text": "production call trace that we have here"
  },
  {
    "startMs": 959360,
    "text": "from Arya. This is the evaluation that"
  },
  {
    "startMs": 961199,
    "text": "it just ran. So if we look into like the"
  },
  {
    "startMs": 963040,
    "text": "actual trace, we'll see that, you know,"
  },
  {
    "startMs": 964959,
    "text": "AR did a bunch of work. It ran this"
  },
  {
    "startMs": 966320,
    "text": "predict and score column. We have a"
  },
  {
    "startMs": 967759,
    "text": "bunch of tool calls that ran. Uh and you"
  },
  {
    "startMs": 969680,
    "text": "know, if we hop back into the chat,"
  },
  {
    "startMs": 970800,
    "text": "we'll see the actual variant that it"
  },
  {
    "startMs": 971920,
    "text": "ran. Um it looks like the candidate"
  },
  {
    "startMs": 973759,
    "text": "variant just had a tight little prompt"
  },
  {
    "startMs": 975199,
    "text": "that we injected into the system prompt"
  },
  {
    "startMs": 976639,
    "text": "or in one of the skills to sort of solve"
  },
  {
    "startMs": 978720,
    "text": "that exact SDK error uh and mitigate it."
  },
  {
    "startMs": 981199,
    "text": "And that kind of replication from"
  },
  {
    "startMs": 982720,
    "text": "production to simulation to agent then"
  },
  {
    "startMs": 985199,
    "text": "defining the improvement pattern is the"
  },
  {
    "startMs": 987040,
    "text": "thing that I really would like to leave"
  },
  {
    "startMs": 988240,
    "text": "everybody with."
  },
  {
    "startMs": 990800,
    "text": "So that's coreweave Arya. Uh welcome to"
  },
  {
    "startMs": 993680,
    "text": "weights and biases. I hope that was an"
  },
  {
    "startMs": 995040,
    "text": "interesting talk in sort of how we"
  },
  {
    "startMs": 996240,
    "text": "decide to build uh our offline"
  },
  {
    "startMs": 998079,
    "text": "evaluation metrics, how we reap data"
  },
  {
    "startMs": 999759,
    "text": "from production and the flywheel of"
  },
  {
    "startMs": 1001279,
    "text": "trying to build these agents and the"
  },
  {
    "startMs": 1002880,
    "text": "things that I think about on a"
  },
  {
    "startMs": 1004000,
    "text": "day-to-day basis. Uh and yeah, let's get"
  },
  {
    "startMs": 1006399,
    "text": "started. Thank you all so much."
  },
  {
    "startMs": 1024431,
    "text": ">> [music]"
  }
]