[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 13280,
    "text": "Okay. Hello everyone and thank you for"
  },
  {
    "startMs": 15679,
    "text": "attending this session. My name is Tim"
  },
  {
    "startMs": 17760,
    "text": "Sweeney, a principal engineer at Weights"
  },
  {
    "startMs": 20160,
    "text": "and Biases and Coreweave. And for the"
  },
  {
    "startMs": 22240,
    "text": "next 20 minutes, we're going to talk"
  },
  {
    "startMs": 23600,
    "text": "about Arya, our new AI research and"
  },
  {
    "startMs": 26160,
    "text": "iteration agent. Let's go ahead and get"
  },
  {
    "startMs": 28160,
    "text": "started."
  },
  {
    "startMs": 29599,
    "text": "So, uh, first off, just by way of making"
  },
  {
    "startMs": 32079,
    "text": "some noise, some clapping, uh, who here,"
  },
  {
    "startMs": 35280,
    "text": "um, identifies as an ML researcher?"
  },
  {
    "startMs": 37440,
    "text": "You're someone that trains models,"
  },
  {
    "startMs": 38960,
    "text": "trains the brain?"
  },
  {
    "startMs": 41520,
    "text": "I heard one. Wow. Okay. Great work."
  },
  {
    "startMs": 43520,
    "text": "Great work. Uh, what about who here is"
  },
  {
    "startMs": 45600,
    "text": "the applied engineer, the namesake of"
  },
  {
    "startMs": 47440,
    "text": "this conference? Who here actually"
  },
  {
    "startMs": 48800,
    "text": "builds the bots?"
  },
  {
    "startMs": 50960,
    "text": ">> Okay, good. Expected much more. And who"
  },
  {
    "startMs": 53520,
    "text": "here is in AI management? You are"
  },
  {
    "startMs": 55199,
    "text": "helping fund this compute."
  },
  {
    "startMs": 58160,
    "text": "Okay. Okay. Nice. From the back. Lovely."
  },
  {
    "startMs": 61359,
    "text": "Um, well, now that I know a little bit"
  },
  {
    "startMs": 62800,
    "text": "about you, just a little bit about me."
  },
  {
    "startMs": 64400,
    "text": "Uh, again, my name is Tim. I have a"
  },
  {
    "startMs": 66159,
    "text": "masters in machine learning, uh, and"
  },
  {
    "startMs": 67920,
    "text": "reinforcement learning from Georgia"
  },
  {
    "startMs": 69360,
    "text": "Tech. So, I've been that, uh, researcher"
  },
  {
    "startMs": 71920,
    "text": "currently building Weights and Biases"
  },
  {
    "startMs": 73680,
    "text": "agent, Arya. So, identify as that"
  },
  {
    "startMs": 75840,
    "text": "applied engineer. And in a previous life"
  },
  {
    "startMs": 78080,
    "text": "was the PM of Twitter's ML stack. So, I"
  },
  {
    "startMs": 81040,
    "text": "hope you hopefully can connect with you"
  },
  {
    "startMs": 82400,
    "text": "middle management as well. [laughter]"
  },
  {
    "startMs": 84495,
    "text": ">> [gasps]"
  },
  {
    "startMs": 84960,
    "text": ">> Um today's agenda is kind of broken into"
  },
  {
    "startMs": 87439,
    "text": "three sections and hopefully each of you"
  },
  {
    "startMs": 89119,
    "text": "personas walk away with something"
  },
  {
    "startMs": 90479,
    "text": "valuable. So first we're going to learn"
  },
  {
    "startMs": 92400,
    "text": "about Arya itself and how it can"
  },
  {
    "startMs": 94320,
    "text": "supercharge your AI and ML workflows."
  },
  {
    "startMs": 97119,
    "text": "We're going to dive into auto research"
  },
  {
    "startMs": 98640,
    "text": "and see that live in a live demo in just"
  },
  {
    "startMs": 100640,
    "text": "a moment."
  },
  {
    "startMs": 102079,
    "text": "Then we're going to pull back the"
  },
  {
    "startMs": 103200,
    "text": "curtain and learn how we use weights and"
  },
  {
    "startMs": 104960,
    "text": "biases and uh coreweave to actually"
  },
  {
    "startMs": 107280,
    "text": "build Arya because a lot of you in the"
  },
  {
    "startMs": 109200,
    "text": "audience are building agents yourself"
  },
  {
    "startMs": 110640,
    "text": "and we believe a lot of these components"
  },
  {
    "startMs": 112159,
    "text": "can help you in your endeavors. And then"
  },
  {
    "startMs": 114720,
    "text": "towards the end we'll just take a step"
  },
  {
    "startMs": 116159,
    "text": "back and identify a few key tips and"
  },
  {
    "startMs": 118000,
    "text": "tricks for making sure that you're able"
  },
  {
    "startMs": 119520,
    "text": "to productionize your systems"
  },
  {
    "startMs": 120880,
    "text": "effectively."
  },
  {
    "startMs": 122640,
    "text": "For those of you who might not be"
  },
  {
    "startMs": 123920,
    "text": "familiar, Weights and Biases is the"
  },
  {
    "startMs": 125759,
    "text": "world's leading AI development platform."
  },
  {
    "startMs": 127920,
    "text": "We've been in business now for nine"
  },
  {
    "startMs": 129440,
    "text": "years and have happily joined the core"
  },
  {
    "startMs": 131680,
    "text": "family about a year ago. Uh we have a"
  },
  {
    "startMs": 133920,
    "text": "number of products in our suite but are"
  },
  {
    "startMs": 135680,
    "text": "really known for our models training"
  },
  {
    "startMs": 137280,
    "text": "inference and weave stack which really"
  },
  {
    "startMs": 139200,
    "text": "helps collect data uh about the AI"
  },
  {
    "startMs": 141760,
    "text": "development and machine learning"
  },
  {
    "startMs": 142959,
    "text": "workflows and makes that information"
  },
  {
    "startMs": 144959,
    "text": "actionable and uh enables users to make"
  },
  {
    "startMs": 147440,
    "text": "the best decisions about what to do"
  },
  {
    "startMs": 148959,
    "text": "next."
  },
  {
    "startMs": 150560,
    "text": "So without further ado, let's go ahead"
  },
  {
    "startMs": 152000,
    "text": "and dive into Arya, our agent. Uh we'll"
  },
  {
    "startMs": 154000,
    "text": "show a demo and then we'll get back to"
  },
  {
    "startMs": 155280,
    "text": "some slides."
  },
  {
    "startMs": 160160,
    "text": "Okay, beautiful."
  },
  {
    "startMs": 162400,
    "text": "Let's make this a bit bigger. Holler at"
  },
  {
    "startMs": 164000,
    "text": "me if you need it to be bigger. So, uh,"
  },
  {
    "startMs": 166080,
    "text": "what you're looking at here is a weights"
  },
  {
    "startMs": 168160,
    "text": "and biases workspace. For you, for"
  },
  {
    "startMs": 170080,
    "text": "anybody that isn't familiar, on the"
  },
  {
    "startMs": 171840,
    "text": "lefth hand side, I actually see a list"
  },
  {
    "startMs": 173680,
    "text": "of a bunch of different experiments. In"
  },
  {
    "startMs": 175680,
    "text": "this particular project, I have over 200"
  },
  {
    "startMs": 177920,
    "text": "training jobs. And on the right hand"
  },
  {
    "startMs": 180000,
    "text": "side, I see a scatter plot of, in this"
  },
  {
    "startMs": 182400,
    "text": "case, declining metrics, which is good."
  },
  {
    "startMs": 184560,
    "text": "means our loss is going down over time."
  },
  {
    "startMs": 186800,
    "text": "And this view would be very familiar for"
  },
  {
    "startMs": 188560,
    "text": "anyone that uses our tool. Now, to"
  },
  {
    "startMs": 190879,
    "text": "ground this, we're actually uh uh using"
  },
  {
    "startMs": 193040,
    "text": "the Carpathy Auto Research project,"
  },
  {
    "startMs": 195040,
    "text": "which I'm sure many of you are familiar"
  },
  {
    "startMs": 196640,
    "text": "with, but if you're not, it's just a"
  },
  {
    "startMs": 198400,
    "text": "very simple project that trains an LLM,"
  },
  {
    "startMs": 200879,
    "text": "and it's a great foundation for auto"
  },
  {
    "startMs": 203200,
    "text": "research type demonstrations because"
  },
  {
    "startMs": 205040,
    "text": "it's a very simple codebase and allows"
  },
  {
    "startMs": 207120,
    "text": "us to improve iteratively over time. So,"
  },
  {
    "startMs": 210159,
    "text": "let's jump back to the project and open"
  },
  {
    "startMs": 211840,
    "text": "up Arya by clicking this blue button in"
  },
  {
    "startMs": 213840,
    "text": "the upper right. When I click this"
  },
  {
    "startMs": 215680,
    "text": "button, I'm uh presented with the"
  },
  {
    "startMs": 217360,
    "text": "familiar chat interface with, you know,"
  },
  {
    "startMs": 219120,
    "text": "how can I help you today, a few call to"
  },
  {
    "startMs": 220959,
    "text": "actions, and you know, I can add"
  },
  {
    "startMs": 223120,
    "text": "different context in my project or maybe"
  },
  {
    "startMs": 225599,
    "text": "add images, etc. Um, everyone here is"
  },
  {
    "startMs": 228400,
    "text": "agent builders, so I don't need to bore"
  },
  {
    "startMs": 229840,
    "text": "you with the details of what an agent"
  },
  {
    "startMs": 231440,
    "text": "interface looks like. But let's go ahead"
  },
  {
    "startMs": 233440,
    "text": "and just, you know, enter in a basic"
  },
  {
    "startMs": 235519,
    "text": "intro here. Let's say, \"Hello, Arya."
  },
  {
    "startMs": 238000,
    "text": "You're on stage at AI World's Fair 2026."
  },
  {
    "startMs": 240640,
    "text": "Please introduce yourself.\" So, it's"
  },
  {
    "startMs": 242400,
    "text": "going to go ahead and chug along and"
  },
  {
    "startMs": 243920,
    "text": "hopefully emit some sort of nice emoji."
  },
  {
    "startMs": 246239,
    "text": "Yay. He I'm Arya. I'm talking to the"
  },
  {
    "startMs": 248640,
    "text": "audience. Great. But now, let's dive"
  },
  {
    "startMs": 250720,
    "text": "into the meat of why you came here. So,"
  },
  {
    "startMs": 253280,
    "text": "I'm going to open up this chat here. And"
  },
  {
    "startMs": 255439,
    "text": "this is a longunning chat where I've"
  },
  {
    "startMs": 257120,
    "text": "been running again over 200 experiments"
  },
  {
    "startMs": 259120,
    "text": "using the auto research loop. Um, it"
  },
  {
    "startMs": 262000,
    "text": "helped me download the code, set up my"
  },
  {
    "startMs": 264080,
    "text": "launch jobs, set up my GPUs, and is able"
  },
  {
    "startMs": 266400,
    "text": "to autonomously iterate on the code"
  },
  {
    "startMs": 268240,
    "text": "itself and the hyperparameters."
  },
  {
    "startMs": 270320,
    "text": "We'll take a look at what it's doing in"
  },
  {
    "startMs": 271759,
    "text": "a moment, but while we're doing this,"
  },
  {
    "startMs": 273280,
    "text": "I'm going to kick off a live iteration"
  },
  {
    "startMs": 275280,
    "text": "right here. So, what I'm going to say is"
  },
  {
    "startMs": 277759,
    "text": "please conduct another batch of"
  },
  {
    "startMs": 279360,
    "text": "experiments. You are on stage at the AI"
  },
  {
    "startMs": 281759,
    "text": "Engineer Worlds Fair 2026 and we're"
  },
  {
    "startMs": 284160,
    "text": "hoping to find the best model live. I"
  },
  {
    "startMs": 286320,
    "text": "believe in you. Uh, because we know we"
  },
  {
    "startMs": 288080,
    "text": "have to encourage our models. Um, so"
  },
  {
    "startMs": 290479,
    "text": "it's been doing this for a while. What"
  },
  {
    "startMs": 291759,
    "text": "it what it's doing here is it's saying,"
  },
  {
    "startMs": 293280,
    "text": "\"Okay, great. Um, I don't want to make a"
  },
  {
    "startMs": 295280,
    "text": "big architecture swing. That feels a"
  },
  {
    "startMs": 297040,
    "text": "little bit too risky.\" So, it's probably"
  },
  {
    "startMs": 298960,
    "text": "going to go for uh some modifications to"
  },
  {
    "startMs": 301120,
    "text": "the hyperparameters. And then it's"
  },
  {
    "startMs": 303120,
    "text": "kicking off a shell call here that is"
  },
  {
    "startMs": 305360,
    "text": "actually um executing that uh executing"
  },
  {
    "startMs": 308160,
    "text": "that experimentation loop. And we're"
  },
  {
    "startMs": 310240,
    "text": "going to check in on this periodically"
  },
  {
    "startMs": 311600,
    "text": "throughout this presentation."
  },
  {
    "startMs": 313600,
    "text": "But I want to help explain what's going"
  },
  {
    "startMs": 315360,
    "text": "on behind the scenes. So behind the"
  },
  {
    "startMs": 317440,
    "text": "scenes I have set up a weights and"
  },
  {
    "startMs": 319120,
    "text": "biases launch queue. Launch is our our"
  },
  {
    "startMs": 321759,
    "text": "product that allows you to connect your"
  },
  {
    "startMs": 323360,
    "text": "compute clusters and allows humans and"
  },
  {
    "startMs": 325520,
    "text": "agents to launch longunning"
  },
  {
    "startMs": 327120,
    "text": "experimentation jobs particularly by"
  },
  {
    "startMs": 329039,
    "text": "leveraging GPUs."
  },
  {
    "startMs": 331280,
    "text": "Here I'm looking at a uh a terminal"
  },
  {
    "startMs": 333600,
    "text": "output of my Kubernetes cluster where"
  },
  {
    "startMs": 335360,
    "text": "we're actually seeing live execution of"
  },
  {
    "startMs": 337759,
    "text": "experiments happening. So this is"
  },
  {
    "startMs": 339120,
    "text": "happening live right here. This is not a"
  },
  {
    "startMs": 340880,
    "text": "fake demo. Um great. And if we jump"
  },
  {
    "startMs": 344720,
    "text": "back, we see that at this point it"
  },
  {
    "startMs": 346639,
    "text": "started the cues and now it is simply"
  },
  {
    "startMs": 348720,
    "text": "polling and waiting for our work to be"
  },
  {
    "startMs": 350880,
    "text": "complete. So we'll jump back to that in"
  },
  {
    "startMs": 352639,
    "text": "a in a moment. But before but let's dive"
  },
  {
    "startMs": 355039,
    "text": "into a few other examples. So uh"
  },
  {
    "startMs": 357520,
    "text": "something else that is interesting you"
  },
  {
    "startMs": 359440,
    "text": "can do is maybe you might want to ask it"
  },
  {
    "startMs": 362000,
    "text": "something like please summarize the"
  },
  {
    "startMs": 363360,
    "text": "highest performing runs in this project."
  },
  {
    "startMs": 365440,
    "text": "This use case would be something like"
  },
  {
    "startMs": 366720,
    "text": "maybe a new user come or a new uh team"
  },
  {
    "startMs": 368720,
    "text": "member is joining your project and want"
  },
  {
    "startMs": 370319,
    "text": "to understand the research. Um or maybe"
  },
  {
    "startMs": 372720,
    "text": "you've uh someone's been doing some work"
  },
  {
    "startMs": 374560,
    "text": "while you were on PTO and you want to"
  },
  {
    "startMs": 376000,
    "text": "get caught up. We'll see what this comes"
  },
  {
    "startMs": 377680,
    "text": "up with in a moment. Some other"
  },
  {
    "startMs": 379600,
    "text": "pre-anned uh examples are finding"
  },
  {
    "startMs": 381759,
    "text": "patterns in your project. So here we can"
  },
  {
    "startMs": 384800,
    "text": "see that I asked it, hey, can you find"
  },
  {
    "startMs": 386319,
    "text": "some patterns in this research? And we"
  },
  {
    "startMs": 388319,
    "text": "see that um it identified that a new"
  },
  {
    "startMs": 390800,
    "text": "family of models emerged as the as the"
  },
  {
    "startMs": 393039,
    "text": "auto uh auto research was happening. Uh"
  },
  {
    "startMs": 395919,
    "text": "it identified that batch size seems to"
  },
  {
    "startMs": 397840,
    "text": "be a really high high uh lever uh"
  },
  {
    "startMs": 400319,
    "text": "parameter. It identified an"
  },
  {
    "startMs": 402240,
    "text": "architectural recipe that seemed to be"
  },
  {
    "startMs": 404319,
    "text": "quite promising and a number of other"
  },
  {
    "startMs": 406479,
    "text": "insights that would have taken me hours"
  },
  {
    "startMs": 408319,
    "text": "or days to discover on my own. And Arya"
  },
  {
    "startMs": 410800,
    "text": "is able to do it right for me directly"
  },
  {
    "startMs": 412560,
    "text": "in the interface that I already live."
  },
  {
    "startMs": 415360,
    "text": "Not only is it able to emit text based"
  },
  {
    "startMs": 417680,
    "text": "uh textbased outputs, but it also deeply"
  },
  {
    "startMs": 420080,
    "text": "integrates with a number of weights and"
  },
  {
    "startMs": 421759,
    "text": "biases visualization utilities. So here"
  },
  {
    "startMs": 424400,
    "text": "I've actually asked it to emit a weights"
  },
  {
    "startMs": 426319,
    "text": "and biases report which for those who"
  },
  {
    "startMs": 428240,
    "text": "aren't familiar is essentially a"
  },
  {
    "startMs": 429680,
    "text": "markdown file on steroids. It's got uh"
  },
  {
    "startMs": 432160,
    "text": "embedded embedded plots, charts and and"
  },
  {
    "startMs": 434319,
    "text": "and graphics. And so here uh you know"
  },
  {
    "startMs": 436960,
    "text": "it's talked about the thesis of the"
  },
  {
    "startMs": 438400,
    "text": "project. It's it's emitted a number of"
  },
  {
    "startMs": 440720,
    "text": "of data panels. And uh I actually think"
  },
  {
    "startMs": 443680,
    "text": "it's quite interesting. It used um one"
  },
  {
    "startMs": 445599,
    "text": "of our more esoteric panels, the uh"
  },
  {
    "startMs": 447759,
    "text": "parameter importance chart to uh tell me"
  },
  {
    "startMs": 450479,
    "text": "the correlation of various different"
  },
  {
    "startMs": 452000,
    "text": "parameters within this uh within this"
  },
  {
    "startMs": 453840,
    "text": "training job."
  },
  {
    "startMs": 455919,
    "text": "Uh in addition to uh reports, it's also"
  },
  {
    "startMs": 458720,
    "text": "great at working with workspaces. So if"
  },
  {
    "startMs": 460880,
    "text": "you're a weights and biases user, uh you"
  },
  {
    "startMs": 463120,
    "text": "spend a lot of your time uh designing"
  },
  {
    "startMs": 465039,
    "text": "and working with workspaces. Well, Arya"
  },
  {
    "startMs": 468000,
    "text": "is actually customtuned and prompted to"
  },
  {
    "startMs": 470400,
    "text": "really understand how to build"
  },
  {
    "startMs": 471680,
    "text": "workspaces, build plots, and complement"
  },
  {
    "startMs": 474160,
    "text": "that that data analytics with real live"
  },
  {
    "startMs": 476479,
    "text": "graphics using the built-in proprietary"
  },
  {
    "startMs": 478879,
    "text": "charts that weights and biases users"
  },
  {
    "startMs": 480479,
    "text": "know and love. Um, so with that, let's"
  },
  {
    "startMs": 483199,
    "text": "go ahead and check back on some of our"
  },
  {
    "startMs": 484479,
    "text": "our prompts. We can see that the please"
  },
  {
    "startMs": 486400,
    "text": "summarize this project prompt is cooking"
  },
  {
    "startMs": 488720,
    "text": "away. It's querying weights and biases."
  },
  {
    "startMs": 490479,
    "text": "It's applying patches. It's writing its"
  },
  {
    "startMs": 492160,
    "text": "own code. So, we'll come back and check"
  },
  {
    "startMs": 493440,
    "text": "on that in a moment. and our longunning"
  },
  {
    "startMs": 495919,
    "text": "training job is uh still pulling for the"
  },
  {
    "startMs": 498400,
    "text": "results. We can see that we're cooking"
  },
  {
    "startMs": 500080,
    "text": "away on our GPUs. So, we're we're frying"
  },
  {
    "startMs": 502800,
    "text": "some GPUs [music] and doing some data"
  },
  {
    "startMs": 504240,
    "text": "science all live. And while that's"
  },
  {
    "startMs": 506000,
    "text": "cooking, let's go ahead and jump back to"
  },
  {
    "startMs": 507440,
    "text": "the presentation. We'll come back in a"
  },
  {
    "startMs": 508960,
    "text": "moment."
  },
  {
    "startMs": 510560,
    "text": "Uh"
  },
  {
    "startMs": 513200,
    "text": "oh, no, we're not looking at a"
  },
  {
    "startMs": 514880,
    "text": "dictionary. We're looking at a PO."
  },
  {
    "startMs": 516719,
    "text": "Great. Uh okay, so quick recap here."
  },
  {
    "startMs": 518959,
    "text": "What did Arya show? What did we show in"
  },
  {
    "startMs": 520560,
    "text": "these last five minutes? First, we show"
  },
  {
    "startMs": 522320,
    "text": "that uh Arya can serve as your data"
  },
  {
    "startMs": 524480,
    "text": "science companion right inside of"
  },
  {
    "startMs": 526080,
    "text": "Weights and Biases, helping you discover"
  },
  {
    "startMs": 528160,
    "text": "insights that you wouldn't you wouldn't"
  },
  {
    "startMs": 529839,
    "text": "be able to discover as your experiments"
  },
  {
    "startMs": 531600,
    "text": "and as your team size grows."
  },
  {
    "startMs": 535040,
    "text": "Next, we address the problem of"
  },
  {
    "startMs": 537040,
    "text": "complicated reporting and complicated"
  },
  {
    "startMs": 538720,
    "text": "plotting. Weights and biases users are"
  },
  {
    "startMs": 540959,
    "text": "are really want to turn their insights"
  },
  {
    "startMs": 542959,
    "text": "into visual communication tools. They"
  },
  {
    "startMs": 544880,
    "text": "want to communicate with their peers and"
  },
  {
    "startMs": 546320,
    "text": "their colleagues. So Arya's built from"
  },
  {
    "startMs": 548560,
    "text": "the ground up to understand those"
  },
  {
    "startMs": 549920,
    "text": "primitives and help co-pilot and drive"
  },
  {
    "startMs": 552080,
    "text": "right along right alongside in the UI"
  },
  {
    "startMs": 555279,
    "text": "and announcing now today for the first"
  },
  {
    "startMs": 557120,
    "text": "time we are releasing Arya on our iOS"
  },
  {
    "startMs": 559440,
    "text": "device or on our iOS app. So uh uh Arya"
  },
  {
    "startMs": 563279,
    "text": "released on Monday and our iOS app now"
  },
  {
    "startMs": 566000,
    "text": "has Arya built in. So if you're"
  },
  {
    "startMs": 567920,
    "text": "conducting hyperparameter tuning jobs,"
  },
  {
    "startMs": 569760,
    "text": "if you're training models, or if you're"
  },
  {
    "startMs": 571120,
    "text": "just researching within the weights and"
  },
  {
    "startMs": 572399,
    "text": "biases ecosystem, you can go touched"
  },
  {
    "startMs": 574640,
    "text": "grass at Yerba Buena uh gardens and"
  },
  {
    "startMs": 577519,
    "text": "steer your uh hyperparameter tuning jobs"
  },
  {
    "startMs": 579680,
    "text": "all from your mobile device. And what is"
  },
  {
    "startMs": 581920,
    "text": "this all building up to? This is"
  },
  {
    "startMs": 583279,
    "text": "building up to an a fully automated"
  },
  {
    "startMs": 585200,
    "text": "endto-end research platform where we're"
  },
  {
    "startMs": 587200,
    "text": "not seeking to replace uh RL"
  },
  {
    "startMs": 589279,
    "text": "researchers, but complement your"
  },
  {
    "startMs": 590560,
    "text": "workflows. Arya's great at orchestrating"
  },
  {
    "startMs": 593040,
    "text": "jobs, understanding GPU workloads,"
  },
  {
    "startMs": 595440,
    "text": "responding to events within the within"
  },
  {
    "startMs": 597040,
    "text": "the Wandi ecosystem, and listening to"
  },
  {
    "startMs": 599200,
    "text": "researchers, uh, uh, looking up archive"
  },
  {
    "startMs": 601600,
    "text": "papers, and collaborating on hypothesis."
  },
  {
    "startMs": 604000,
    "text": "So, we can let Arya drive the mechanics"
  },
  {
    "startMs": 605839,
    "text": "that you don't want to deal with while"
  },
  {
    "startMs": 607279,
    "text": "you focus on the new ideas, new"
  },
  {
    "startMs": 608959,
    "text": "architectures, and new parameters that"
  },
  {
    "startMs": 610399,
    "text": "you wanted to try."
  },
  {
    "startMs": 612720,
    "text": "Um, great. So, that's Arya in a"
  },
  {
    "startMs": 614959,
    "text": "nutshell. We're really hoping that you"
  },
  {
    "startMs": 616320,
    "text": "give it a shot. And uh we'll jump back"
  },
  {
    "startMs": 618399,
    "text": "to the auto research at the end and see"
  },
  {
    "startMs": 620079,
    "text": "if we got a new best record. But before"
  },
  {
    "startMs": 622399,
    "text": "we do that, let's talk about how we use"
  },
  {
    "startMs": 624399,
    "text": "weights and biases and coreweave to"
  },
  {
    "startMs": 626240,
    "text": "actually build Arya. So now speaking to"
  },
  {
    "startMs": 628000,
    "text": "a lot of the the AI agent builders in"
  },
  {
    "startMs": 630160,
    "text": "the room, here's a quick architecture on"
  },
  {
    "startMs": 632959,
    "text": "the lefth hand side. You see that we"
  },
  {
    "startMs": 634240,
    "text": "have a web client, iOS client that"
  },
  {
    "startMs": 636000,
    "text": "communicates with our API server that"
  },
  {
    "startMs": 637920,
    "text": "then dumps data into our turn database"
  },
  {
    "startMs": 639920,
    "text": "and is worked on by our harness, our our"
  },
  {
    "startMs": 641920,
    "text": "worker harness. This is sort of"
  },
  {
    "startMs": 643839,
    "text": "archetypical of probably what most of"
  },
  {
    "startMs": 645440,
    "text": "you are all building in the room and is"
  },
  {
    "startMs": 647200,
    "text": "exactly what we have on our back end."
  },
  {
    "startMs": 649279,
    "text": "But that harness worker is a magic is a"
  },
  {
    "startMs": 651200,
    "text": "is a magic box and it connects to a"
  },
  {
    "startMs": 653120,
    "text": "number of important utilities. First is"
  },
  {
    "startMs": 655040,
    "text": "a sandbox where it can execute arbitrary"
  },
  {
    "startMs": 657040,
    "text": "shell calls uh do do Python data science"
  },
  {
    "startMs": 660079,
    "text": "etc. And we invite you to try coreweave"
  },
  {
    "startMs": 662320,
    "text": "weights and biases sandbox to fit into"
  },
  {
    "startMs": 664399,
    "text": "your architecture."
  },
  {
    "startMs": 666320,
    "text": "Next up you need an LLM provider of"
  },
  {
    "startMs": 668160,
    "text": "course and so if you're maybe using GLM"
  },
  {
    "startMs": 670880,
    "text": "5.2 two or one of your fine-tuned"
  },
  {
    "startMs": 672640,
    "text": "models. We invite you to use uh weights"
  },
  {
    "startMs": 674399,
    "text": "and biases inference and connect that to"
  },
  {
    "startMs": 676240,
    "text": "your worker as well."
  },
  {
    "startMs": 678640,
    "text": "If you're like us, you need to run"
  },
  {
    "startMs": 680160,
    "text": "longunning workloads outside of the main"
  },
  {
    "startMs": 682560,
    "text": "loop of the agent where you're actually"
  },
  {
    "startMs": 684160,
    "text": "training for day for sometimes days at a"
  },
  {
    "startMs": 686240,
    "text": "time. Weights and biases launch can"
  },
  {
    "startMs": 688160,
    "text": "actually help facilitate that and"
  },
  {
    "startMs": 689760,
    "text": "coreweave GPUs can help make that"
  },
  {
    "startMs": 691360,
    "text": "compute even better."
  },
  {
    "startMs": 693680,
    "text": "And then lastly, and really most"
  },
  {
    "startMs": 695040,
    "text": "importantly, we need an observability"
  },
  {
    "startMs": 696720,
    "text": "layer. It's critical that your agents"
  },
  {
    "startMs": 698880,
    "text": "are able to log out their what's going"
  },
  {
    "startMs": 700560,
    "text": "on with their sessions, their turns,"
  },
  {
    "startMs": 702160,
    "text": "their tool calls, any errors they're hap"
  },
  {
    "startMs": 703920,
    "text": "that that's happening, etc. Uh we have a"
  },
  {
    "startMs": 706480,
    "text": "product called Weights and Biases Weave"
  },
  {
    "startMs": 708000,
    "text": "that we log 100% of our traces to where"
  },
  {
    "startMs": 710320,
    "text": "us and our team can learn from. And"
  },
  {
    "startMs": 712240,
    "text": "that's where we move from production to"
  },
  {
    "startMs": 714560,
    "text": "offline where our team is able to use"
  },
  {
    "startMs": 716800,
    "text": "Weights and Biases Weave to drive"
  },
  {
    "startMs": 718240,
    "text": "insights and identify behaviors,"
  },
  {
    "startMs": 720240,
    "text": "implement tasks with tasks which are"
  },
  {
    "startMs": 722560,
    "text": "essentially unit tests for your models"
  },
  {
    "startMs": 724320,
    "text": "and evaluate those models in a loop."
  },
  {
    "startMs": 727120,
    "text": "We have a model repository which you"
  },
  {
    "startMs": 728880,
    "text": "might choose to use weights and biases"
  },
  {
    "startMs": 730399,
    "text": "artifacts to store your agents or models"
  },
  {
    "startMs": 732720,
    "text": "and you we emit our evaluation results"
  },
  {
    "startMs": 735120,
    "text": "to weave where we have a common"
  },
  {
    "startMs": 736560,
    "text": "dashboard that we can make go no-go"
  },
  {
    "startMs": 738399,
    "text": "decisions on various prompt changes or"
  },
  {
    "startMs": 740320,
    "text": "architectural changes that then feeds"
  },
  {
    "startMs": 743040,
    "text": "into a research loop which we call our"
  },
  {
    "startMs": 744800,
    "text": "improvement loop where we form"
  },
  {
    "startMs": 746320,
    "text": "hypotheses implement candidate agents"
  },
  {
    "startMs": 748240,
    "text": "and analyze the evals. So we have two"
  },
  {
    "startMs": 750720,
    "text": "sort of complimentary yet adversarial"
  },
  {
    "startMs": 752720,
    "text": "research loops going on going on offline"
  },
  {
    "startMs": 755360,
    "text": "feeding data from weights and biases"
  },
  {
    "startMs": 757440,
    "text": "weave ultimately to identify the best"
  },
  {
    "startMs": 759519,
    "text": "model so that we can promote that to"
  },
  {
    "startMs": 761200,
    "text": "production through our registry and"
  },
  {
    "startMs": 762720,
    "text": "close the data flywheel. So in the next"
  },
  {
    "startMs": 765519,
    "text": "just uh three seven minutes or so we'll"
  },
  {
    "startMs": 767920,
    "text": "just talk about uh weights and biases"
  },
  {
    "startMs": 769600,
    "text": "weave and show how we as a team actually"
  },
  {
    "startMs": 771760,
    "text": "use weave to facilitate this workflow"
  },
  {
    "startMs": 773839,
    "text": "and we believe this is something that"
  },
  {
    "startMs": 775040,
    "text": "you would benefit from as well all of"
  },
  {
    "startMs": 776639,
    "text": "you agent builders in the room."
  },
  {
    "startMs": 779440,
    "text": "Yes, another demo. Great."
  },
  {
    "startMs": 783040,
    "text": "Okay. Okay, we have new responses. So,"
  },
  {
    "startMs": 785680,
    "text": "it's going to be exciting when we open"
  },
  {
    "startMs": 786959,
    "text": "this up later. See if uh we've got some"
  },
  {
    "startMs": 789200,
    "text": "better metrics. Um, okay. Let me zoom"
  },
  {
    "startMs": 791839,
    "text": "out just a little bit here. So, here I'm"
  },
  {
    "startMs": 793680,
    "text": "looking at the agent dashboard. This is"
  },
  {
    "startMs": 795600,
    "text": "the live weights and biases agent or"
  },
  {
    "startMs": 798560,
    "text": "Arya agent dashboard uh built in weave."
  },
  {
    "startMs": 801360,
    "text": "Man, that is a lot of uh branded"
  },
  {
    "startMs": 803440,
    "text": "buzzwords there. This is the dashboard"
  },
  {
    "startMs": 805040,
    "text": "that you would get if you use our tool."
  },
  {
    "startMs": 806880,
    "text": "and uh you have a you know uh span"
  },
  {
    "startMs": 809279,
    "text": "volume, conversation volume, token"
  },
  {
    "startMs": 811120,
    "text": "tracking, etc. Think of this as like a"
  },
  {
    "startMs": 813600,
    "text": "uh a bird's eye view of your agent. For"
  },
  {
    "startMs": 816720,
    "text": "me, however, I really like this"
  },
  {
    "startMs": 818160,
    "text": "conversations view, which I do have"
  },
  {
    "startMs": 819920,
    "text": "pre-loaded in this tab. This"
  },
  {
    "startMs": 821760,
    "text": "conversations view is a live feed of all"
  },
  {
    "startMs": 824240,
    "text": "of the conversations that are going"
  },
  {
    "startMs": 825519,
    "text": "through Arya, but it's filtered down to"
  },
  {
    "startMs": 827360,
    "text": "just the internal employees. So, it's a"
  },
  {
    "startMs": 829360,
    "text": "little bit of a of a reduced set here."
  },
  {
    "startMs": 831760,
    "text": "Um what I what I love is this middle"
  },
  {
    "startMs": 833920,
    "text": "spans view which gives me a visual"
  },
  {
    "startMs": 836000,
    "text": "indicator of the topology of a trace."
  },
  {
    "startMs": 838720,
    "text": "Different colors and and shapes indicate"
  },
  {
    "startMs": 841040,
    "text": "different things that are happening"
  },
  {
    "startMs": 842320,
    "text": "within the agent. So things like tool"
  },
  {
    "startMs": 844160,
    "text": "calls, LLM calls, thinking blocks, etc."
  },
  {
    "startMs": 846959,
    "text": "which really help me understand again"
  },
  {
    "startMs": 848480,
    "text": "the shape and topology of that"
  },
  {
    "startMs": 850160,
    "text": "particular conversation. I can of course"
  },
  {
    "startMs": 852800,
    "text": "open up one of these conversations and"
  },
  {
    "startMs": 854800,
    "text": "view our our conversation view where I"
  },
  {
    "startMs": 857120,
    "text": "can see the system prompt, the user"
  },
  {
    "startMs": 858800,
    "text": "message, shell calls, reasoning blocks,"
  },
  {
    "startMs": 861519,
    "text": "etc. This is where my research lead,"
  },
  {
    "startMs": 863920,
    "text": "myself and my PM go to add notes, add"
  },
  {
    "startMs": 866320,
    "text": "feedback, add emojis, and talk about and"
  },
  {
    "startMs": 868880,
    "text": "discover those insights and those"
  },
  {
    "startMs": 870399,
    "text": "behavioral nuances we spoke about"
  },
  {
    "startMs": 872079,
    "text": "earlier so that we can turn them into"
  },
  {
    "startMs": 873760,
    "text": "tasks."
  },
  {
    "startMs": 875360,
    "text": "Arya's built in to the weights and"
  },
  {
    "startMs": 877120,
    "text": "biases system as well. Here you'll see a"
  },
  {
    "startMs": 879120,
    "text": "summarize button and these are sprinkled"
  },
  {
    "startMs": 880880,
    "text": "throughout the weights and biases"
  },
  {
    "startMs": 882079,
    "text": "application. I simply click summarize"
  },
  {
    "startMs": 884560,
    "text": "and we start a new chat contextualized"
  },
  {
    "startMs": 887040,
    "text": "to the thing that I'm looking at. So it"
  },
  {
    "startMs": 889519,
    "text": "it sees this and says give me a brief"
  },
  {
    "startMs": 891440,
    "text": "summary of this particular conversation."
  },
  {
    "startMs": 893839,
    "text": "So if you if you're paying a attention"
  },
  {
    "startMs": 895680,
    "text": "closely, you'll realize that what we're"
  },
  {
    "startMs": 897360,
    "text": "doing is using Arya to analyze Arya's"
  },
  {
    "startMs": 900000,
    "text": "own conversations to then make"
  },
  {
    "startMs": 901360,
    "text": "recommendations about how to improve"
  },
  {
    "startMs": 902880,
    "text": "Arya all within the UI."
  },
  {
    "startMs": 906079,
    "text": "Um okay, great. While that's cooking"
  },
  {
    "startMs": 908000,
    "text": "away, I want to show you the last item"
  },
  {
    "startMs": 910079,
    "text": "uh within the Weave ecosystem here, and"
  },
  {
    "startMs": 912240,
    "text": "that's signals. We've heard a lot today"
  },
  {
    "startMs": 914160,
    "text": "about the value of evals and the value"
  },
  {
    "startMs": 916000,
    "text": "of LLM judges. Weave actually offers an"
  },
  {
    "startMs": 918800,
    "text": "integrated LLM judge experience. So"
  },
  {
    "startMs": 921199,
    "text": "here, if I zoom out a little bit, you'll"
  },
  {
    "startMs": 923120,
    "text": "see that I have a user frustration"
  },
  {
    "startMs": 925120,
    "text": "signal, a lowquality response signal,"
  },
  {
    "startMs": 927040,
    "text": "ask user signal, etc. These are LLM"
  },
  {
    "startMs": 929680,
    "text": "judges that run live against against our"
  },
  {
    "startMs": 932160,
    "text": "live traffic. And we can see various"
  },
  {
    "startMs": 934160,
    "text": "different signals like user frustration"
  },
  {
    "startMs": 935839,
    "text": "moments or lowquality responses. These"
  },
  {
    "startMs": 938480,
    "text": "help our team identify these clusters of"
  },
  {
    "startMs": 940560,
    "text": "behavior for us to go fix in next week's"
  },
  {
    "startMs": 942639,
    "text": "iteration. Let's go ahead and do a live"
  },
  {
    "startMs": 944880,
    "text": "look and see what it says. Um this says"
  },
  {
    "startMs": 947680,
    "text": "the user explicitly states that I'm not"
  },
  {
    "startMs": 949600,
    "text": "satisfied with the loss curve. It looks"
  },
  {
    "startMs": 951519,
    "text": "bad and it apparently that indicates"
  },
  {
    "startMs": 953839,
    "text": "frustration. So here we can see an LLM"
  },
  {
    "startMs": 956240,
    "text": "judges live reasoning for why that"
  },
  {
    "startMs": 958079,
    "text": "particular flag was uh indicated."
  },
  {
    "startMs": 962079,
    "text": "Uh let's see, four minutes left."
  },
  {
    "startMs": 963519,
    "text": "Perfect. Um so, uh with that, I've been"
  },
  {
    "startMs": 966079,
    "text": "using the term task a lot. And so what"
  },
  {
    "startMs": 968320,
    "text": "we're do, what I've showed so far is is"
  },
  {
    "startMs": 970240,
    "text": "this live production loop where we are"
  },
  {
    "startMs": 972079,
    "text": "are are are tracing our our prod logs."
  },
  {
    "startMs": 974399,
    "text": "We're looking at them as humans, maybe"
  },
  {
    "startMs": 976160,
    "text": "even using LLMs to complement that"
  },
  {
    "startMs": 977839,
    "text": "analysis. And what we end up doing is"
  },
  {
    "startMs": 980000,
    "text": "transforming those into tasks. Now, this"
  },
  {
    "startMs": 982240,
    "text": "gets a bit technical here, but our tasks"
  },
  {
    "startMs": 984399,
    "text": "are all described as YAML files. You can"
  },
  {
    "startMs": 986480,
    "text": "think of a task as essentially a unit"
  },
  {
    "startMs": 988240,
    "text": "test for your model. So here we say we"
  },
  {
    "startMs": 990720,
    "text": "have a an example user prompt that says"
  },
  {
    "startMs": 993279,
    "text": "check this run and that run. Both of"
  },
  {
    "startMs": 995440,
    "text": "these are giving good results. What can"
  },
  {
    "startMs": 997199,
    "text": "we learn from this? What's the"
  },
  {
    "startMs": 998320,
    "text": "difference? So this is an example of"
  },
  {
    "startMs": 1000079,
    "text": "something we want Arya to be good at for"
  },
  {
    "startMs": 1002240,
    "text": "all of you. And after the uh requisite"
  },
  {
    "startMs": 1005680,
    "text": "metadata we see that we've defined an"
  },
  {
    "startMs": 1007680,
    "text": "LLM judge. So here we've defined what"
  },
  {
    "startMs": 1010079,
    "text": "correctness means in the context of that"
  },
  {
    "startMs": 1012160,
    "text": "question."
  },
  {
    "startMs": 1013839,
    "text": "And we've then we've defined a second"
  },
  {
    "startMs": 1015920,
    "text": "LLM judge that determines if the"
  },
  {
    "startMs": 1018160,
    "text": "insights are actually interesting."
  },
  {
    "startMs": 1020501,
    "text": "[laughter] And then we've uh defined a"
  },
  {
    "startMs": 1022720,
    "text": "third rule-based judge that says were"
  },
  {
    "startMs": 1025120,
    "text": "you able to actually generate a result"
  },
  {
    "startMs": 1027280,
    "text": "within just six tool calls meaning it"
  },
  {
    "startMs": 1029600,
    "text": "got there with some degree of"
  },
  {
    "startMs": 1030959,
    "text": "expediency. These are all then clustered"
  },
  {
    "startMs": 1033520,
    "text": "together into we have about like 200 of"
  },
  {
    "startMs": 1035439,
    "text": "these. They're all clustered together"
  },
  {
    "startMs": 1037039,
    "text": "into an eval suite that runs nightly."
  },
  {
    "startMs": 1039360,
    "text": "And again we use weave to track all"
  },
  {
    "startMs": 1041280,
    "text": "those evals. So here, I know it's a bit"
  },
  {
    "startMs": 1043760,
    "text": "small on this screen, but what you're"
  },
  {
    "startMs": 1045520,
    "text": "looking at is a listing of every night's"
  },
  {
    "startMs": 1047520,
    "text": "eval. This is literally two nights ago,"
  },
  {
    "startMs": 1049280,
    "text": "the evaluation for our candidate model"
  },
  {
    "startMs": 1051280,
    "text": "got 73% on our production or on our eval"
  },
  {
    "startMs": 1054720,
    "text": "suite against the 72% that our prod"
  },
  {
    "startMs": 1057039,
    "text": "model got, which means we're definitely"
  },
  {
    "startMs": 1058480,
    "text": "going to push that forward uh this"
  },
  {
    "startMs": 1060080,
    "text": "Friday. Uh and we can see a kind of a a"
  },
  {
    "startMs": 1062480,
    "text": "performance plot on the right. So these"
  },
  {
    "startMs": 1064480,
    "text": "utilities are what you would get out of"
  },
  {
    "startMs": 1065760,
    "text": "the box if you're uh if you decide to"
  },
  {
    "startMs": 1067440,
    "text": "pick up weave and use this tool. Um,"
  },
  {
    "startMs": 1070400,
    "text": "jumping back to the last conversation we"
  },
  {
    "startMs": 1072480,
    "text": "had where it asked me where we asked,"
  },
  {
    "startMs": 1074240,
    "text": "uh, can you please give a quick summary"
  },
  {
    "startMs": 1076240,
    "text": "of this trace, we see that it actually"
  },
  {
    "startMs": 1078559,
    "text": "analyzed the conversation, understood"
  },
  {
    "startMs": 1080480,
    "text": "what the user was doing, and then"
  },
  {
    "startMs": 1082400,
    "text": "ultimately decided that this was a"
  },
  {
    "startMs": 1084799,
    "text": "pretty strong trace. Um, let's see,"
  },
  {
    "startMs": 1087520,
    "text": "we've got two and a half minutes left,"
  },
  {
    "startMs": 1088960,
    "text": "so let's just quickly recap here. Uh,"
  },
  {
    "startMs": 1091200,
    "text": "first off, uh, what we use weave to do"
  },
  {
    "startMs": 1093039,
    "text": "is a, collect production traffic. Super"
  },
  {
    "startMs": 1095039,
    "text": "critical to collect all of your"
  },
  {
    "startMs": 1096400,
    "text": "production traffic so you can learn and"
  },
  {
    "startMs": 1097840,
    "text": "iterate. Secondly, we use it to generate"
  },
  {
    "startMs": 1100400,
    "text": "insights both as humans as well. We we"
  },
  {
    "startMs": 1102880,
    "text": "do it as humans. We use Arya and we use"
  },
  {
    "startMs": 1104799,
    "text": "LLM judges to identify those behavioral"
  },
  {
    "startMs": 1106960,
    "text": "nuances. We then enrich our tasks. We"
  },
  {
    "startMs": 1110080,
    "text": "implement models and we evaluate using"
  },
  {
    "startMs": 1112240,
    "text": "weights and biases weave as a shared"
  },
  {
    "startMs": 1114240,
    "text": "dashboard where we can make decisions"
  },
  {
    "startMs": 1115760,
    "text": "together as a team that then ultimately"
  },
  {
    "startMs": 1118480,
    "text": "allows us to promote the best model"
  },
  {
    "startMs": 1120400,
    "text": "forward with confidence."
  },
  {
    "startMs": 1123200,
    "text": "So speaking of confident"
  },
  {
    "startMs": 1124480,
    "text": "productionization, let me speak uh"
  },
  {
    "startMs": 1126240,
    "text": "briefly to the managers in the room. So"
  },
  {
    "startMs": 1128640,
    "text": "a few tips for being successful here."
  },
  {
    "startMs": 1131039,
    "text": "First is um invest in agent-oriented"
  },
  {
    "startMs": 1133760,
    "text": "observability. Uh I'm a bit biased. I"
  },
  {
    "startMs": 1136320,
    "text": "believe that weights and biases weave is"
  },
  {
    "startMs": 1137919,
    "text": "the uh observability platform of the"
  },
  {
    "startMs": 1139679,
    "text": "future. Uh but pick your favorite"
  },
  {
    "startMs": 1141120,
    "text": "flavor. Whatever it is, log your"
  },
  {
    "startMs": 1143200,
    "text": "sessions, log your turns, log your tools"
  },
  {
    "startMs": 1144880,
    "text": "and feedback. This introduces an ability"
  },
  {
    "startMs": 1147039,
    "text": "to catch a new class of bugs in our"
  },
  {
    "startMs": 1148960,
    "text": "world called behavioral bugs. Not"
  },
  {
    "startMs": 1150880,
    "text": "exceptions, not performance, but"
  },
  {
    "startMs": 1152320,
    "text": "behavioral bugs."
  },
  {
    "startMs": 1154480,
    "text": "Next up, tasks and evals are the new"
  },
  {
    "startMs": 1156799,
    "text": "world of CI. You've heard a lot about"
  },
  {
    "startMs": 1158320,
    "text": "this. If you are a software engineer,"
  },
  {
    "startMs": 1159919,
    "text": "you've written unit tests your whole"
  },
  {
    "startMs": 1161120,
    "text": "life. You must develop a practice where"
  },
  {
    "startMs": 1163120,
    "text": "your researchers are sitting on the same"
  },
  {
    "startMs": 1164640,
    "text": "scrum team as you developing tasks and"
  },
  {
    "startMs": 1166640,
    "text": "you're viewing the performance metrics"
  },
  {
    "startMs": 1168320,
    "text": "as true go no-go decisions. But in order"
  },
  {
    "startMs": 1171760,
    "text": "to complement that, you must use humans"
  },
  {
    "startMs": 1173679,
    "text": "as a necessary judge. There are"
  },
  {
    "startMs": 1175600,
    "text": "behavioral nuances that LLM will not"
  },
  {
    "startMs": 1177600,
    "text": "catch. You must be using your product"
  },
  {
    "startMs": 1179520,
    "text": "and you must be manually reviewing these"
  },
  {
    "startMs": 1181440,
    "text": "traces as a team at the end of the week"
  },
  {
    "startMs": 1183520,
    "text": "on a board looking at the best and worst"
  },
  {
    "startMs": 1185360,
    "text": "traces to understand how your model is"
  },
  {
    "startMs": 1187520,
    "text": "performing."
  },
  {
    "startMs": 1189440,
    "text": "And then lastly, um just maybe one one"
  },
  {
    "startMs": 1191919,
    "text": "more tip is to add value through context"
  },
  {
    "startMs": 1193919,
    "text": "and tools. It can be really tempting to"
  },
  {
    "startMs": 1196480,
    "text": "uh try to overengineer the harness and"
  },
  {
    "startMs": 1198160,
    "text": "do a bunch of creative stuff around"
  },
  {
    "startMs": 1199360,
    "text": "memory and things like this. We found"
  },
  {
    "startMs": 1201200,
    "text": "that a a lot of lowhanging fruit can be"
  },
  {
    "startMs": 1203120,
    "text": "ascertained through simply giving your"
  },
  {
    "startMs": 1204799,
    "text": "agent context about your business"
  },
  {
    "startMs": 1206160,
    "text": "domain, the underlying uh primitives"
  },
  {
    "startMs": 1208320,
    "text": "that you have available and your"
  },
  {
    "startMs": 1209840,
    "text": "particular uh business data. Um so with"
  },
  {
    "startMs": 1212799,
    "text": "that, let's go ahead and check in on our"
  },
  {
    "startMs": 1215039,
    "text": "uh our our research agent here and let's"
  },
  {
    "startMs": 1219120,
    "text": "go ahead and toggle our workspace. And"
  },
  {
    "startMs": 1221120,
    "text": "what we should be seeing is yes indeed a"
  },
  {
    "startMs": 1224640,
    "text": "little dot that uh oh okay our previous"
  },
  {
    "startMs": 1227760,
    "text": "dot which was done at lunch was 5.83."
  },
  {
    "startMs": 1230159,
    "text": "831. This got 5.833. So we were right on"
  },
  {
    "startMs": 1233840,
    "text": "the edge of having a live improvement,"
  },
  {
    "startMs": 1235600,
    "text": "but pretty darn close. Uh so that's what"
  },
  {
    "startMs": 1237840,
    "text": "the uh that's what the model was able to"
  },
  {
    "startMs": 1239600,
    "text": "produce. It actually uh ran uh quite a"
  },
  {
    "startMs": 1242880,
    "text": "few tests here. I see I'm over time, so"
  },
  {
    "startMs": 1244400,
    "text": "I will click close pretty soon. But we"
  },
  {
    "startMs": 1246320,
    "text": "ran 12 different experiments within that"
  },
  {
    "startMs": 1248000,
    "text": "experiment batch and uh we'll be running"
  },
  {
    "startMs": 1250000,
    "text": "more all night. So please try out Arya,"
  },
  {
    "startMs": 1252480,
    "text": "scan the QR codes, check out the docs."
  },
  {
    "startMs": 1254240,
    "text": "Uh we really love to see what you do"
  },
  {
    "startMs": 1256080,
    "text": "with it and um looking forward to"
  },
  {
    "startMs": 1257919,
    "text": "serving you. Thank you very much."
  },
  {
    "startMs": 1273334,
    "text": ">> [music]"
  }
]