[
  {
    "startMs": 12960,
    "text": "Hello everyone. How's everyone doing?"
  },
  {
    "startMs": 17840,
    "text": "Are you guys ready for some more loops?"
  },
  {
    "startMs": 21119,
    "text": "Yeah. My name is Roland. My co-founder"
  },
  {
    "startMs": 24400,
    "text": "and I were in this mythical place called"
  },
  {
    "startMs": 27119,
    "text": "XAI working hard on agent infra and we"
  },
  {
    "startMs": 30880,
    "text": "realized there's something new that has"
  },
  {
    "startMs": 33200,
    "text": "to be done in a standalone way. So we"
  },
  {
    "startMs": 35680,
    "text": "left a few months ago to really figure"
  },
  {
    "startMs": 38559,
    "text": "out okay what's the next stage of how we"
  },
  {
    "startMs": 41040,
    "text": "should deploy these always on longunning"
  },
  {
    "startMs": 43840,
    "text": "horizon tasks. Um and I'm happy to"
  },
  {
    "startMs": 47600,
    "text": "announce we have a few findings that we"
  },
  {
    "startMs": 49600,
    "text": "would like to present you. Um, and this"
  },
  {
    "startMs": 52559,
    "text": "talk it's all about um, how you should"
  },
  {
    "startMs": 55199,
    "text": "productize these ideas in ways that can"
  },
  {
    "startMs": 58000,
    "text": "scale with your customers. Um, you've"
  },
  {
    "startMs": 62399,
    "text": "heard a lot about auto research. Um, we"
  },
  {
    "startMs": 65280,
    "text": "think there's a blueprint for 2026 and"
  },
  {
    "startMs": 67280,
    "text": "beyond on how you should think about"
  },
  {
    "startMs": 68720,
    "text": "auto research and it really comes down"
  },
  {
    "startMs": 72240,
    "text": "to three ideas."
  },
  {
    "startMs": 74479,
    "text": "Let's go through the first one. The loop"
  },
  {
    "startMs": 77360,
    "text": "is the product."
  },
  {
    "startMs": 81280,
    "text": "We're all familiar with this. We've"
  },
  {
    "startMs": 83360,
    "text": "started with everything goes down to RL"
  },
  {
    "startMs": 86560,
    "text": "chief for models and how you should"
  },
  {
    "startMs": 88640,
    "text": "train the model to become better and"
  },
  {
    "startMs": 90479,
    "text": "better reasoning. We then quickly moved"
  },
  {
    "startMs": 92640,
    "text": "to harnesses and how the model is a"
  },
  {
    "startMs": 95119,
    "text": "commodity and it's all about the"
  },
  {
    "startMs": 96640,
    "text": "harness. And now we're talking about"
  },
  {
    "startMs": 98560,
    "text": "loops and how you should build these"
  },
  {
    "startMs": 100320,
    "text": "loops uh and not touch code anymore. But"
  },
  {
    "startMs": 103439,
    "text": "what does it really mean and why is"
  },
  {
    "startMs": 105280,
    "text": "everyone saying that?"
  },
  {
    "startMs": 107920,
    "text": "Do you guys remember clawbot?"
  },
  {
    "startMs": 110560,
    "text": "That was the original um original name"
  },
  {
    "startMs": 113200,
    "text": "of what is now now now known as open"
  },
  {
    "startMs": 115840,
    "text": "claw. And this guy AJ built the first"
  },
  {
    "startMs": 120320,
    "text": "loop around cloudbot."
  },
  {
    "startMs": 123280,
    "text": "What he did was to find a way to talk to"
  },
  {
    "startMs": 126640,
    "text": "dealers and talk to Reddit users to get"
  },
  {
    "startMs": 129840,
    "text": "bigger discounts on a car. He followed"
  },
  {
    "startMs": 132640,
    "text": "these four steps. Um and is really open"
  },
  {
    "startMs": 135599,
    "text": "call the one that did it. Go on Reddit,"
  },
  {
    "startMs": 138959,
    "text": "find prices, find inventory, talk to the"
  },
  {
    "startMs": 142640,
    "text": "dealers,"
  },
  {
    "startMs": 145280,
    "text": "put dealers head-to-head, and try to"
  },
  {
    "startMs": 147440,
    "text": "figure out how to make them outbid each"
  },
  {
    "startMs": 149680,
    "text": "other,"
  },
  {
    "startMs": 151200,
    "text": "have a verifiable way to know when the"
  },
  {
    "startMs": 153920,
    "text": "price is right, and then lock in, get"
  },
  {
    "startMs": 158080,
    "text": "the car, and it worked. Um, probably"
  },
  {
    "startMs": 161200,
    "text": "this was when all the Mac minis were uh"
  },
  {
    "startMs": 164080,
    "text": "selling off the shelves, but this was"
  },
  {
    "startMs": 166160,
    "text": "the first real example of loop is the"
  },
  {
    "startMs": 169360,
    "text": "product and something that probably"
  },
  {
    "startMs": 171840,
    "text": "should be a startup at this point. Um,"
  },
  {
    "startMs": 173920,
    "text": "but we've seen how this became a recipe"
  },
  {
    "startMs": 177200,
    "text": "for everyone to build loops. But let's"
  },
  {
    "startMs": 179920,
    "text": "take a step back. Why are we here? Um,"
  },
  {
    "startMs": 183440,
    "text": "we really think models have been trained"
  },
  {
    "startMs": 185519,
    "text": "with this loop in mind. And it comes"
  },
  {
    "startMs": 187840,
    "text": "from this idea of udala loops. It's a"
  },
  {
    "startMs": 190959,
    "text": "terminology coined back in 1970s by the"
  },
  {
    "startMs": 195200,
    "text": "US air force and is the idea of these um"
  },
  {
    "startMs": 199200,
    "text": "jet fighters how to react in fast-paced"
  },
  {
    "startMs": 202879,
    "text": "environments."
  },
  {
    "startMs": 204720,
    "text": "If you think of models calling tools and"
  },
  {
    "startMs": 207200,
    "text": "taking observations, it's it's what"
  },
  {
    "startMs": 209680,
    "text": "we've been trained on uh as humans but"
  },
  {
    "startMs": 212080,
    "text": "also as as agents. Now, now what happens"
  },
  {
    "startMs": 215280,
    "text": "when you put strong signals and"
  },
  {
    "startMs": 217680,
    "text": "verifiable work uh at the other ends?"
  },
  {
    "startMs": 221519,
    "text": "You get to these workers or cloud code"
  },
  {
    "startMs": 224560,
    "text": "agents. Um and and what matters here is"
  },
  {
    "startMs": 228640,
    "text": "the quality of the signal determines the"
  },
  {
    "startMs": 231760,
    "text": "uh success rate of the loop and the uh"
  },
  {
    "startMs": 235040,
    "text": "quality of the verifi verifier um um is"
  },
  {
    "startMs": 239680,
    "text": "able to calibrate if that success is"
  },
  {
    "startMs": 242319,
    "text": "actually correct or not. But there's"
  },
  {
    "startMs": 245040,
    "text": "another loop here. Um what happens when"
  },
  {
    "startMs": 247680,
    "text": "you take that and feed it back into the"
  },
  {
    "startMs": 249760,
    "text": "signal? And this is what looping around"
  },
  {
    "startMs": 252959,
    "text": "is all about is how do you generate"
  },
  {
    "startMs": 255760,
    "text": "these artifacts at the end of the first"
  },
  {
    "startMs": 257519,
    "text": "loop to then run a second loop on and"
  },
  {
    "startMs": 260799,
    "text": "have a way to continuously improve. And"
  },
  {
    "startMs": 264320,
    "text": "this goes to my second point. System"
  },
  {
    "startMs": 266960,
    "text": "dillation is the mode and is really the"
  },
  {
    "startMs": 270160,
    "text": "ability to understand what went well and"
  },
  {
    "startMs": 274000,
    "text": "wrong in the first loop and know how to"
  },
  {
    "startMs": 277360,
    "text": "process that in the second one."
  },
  {
    "startMs": 280080,
    "text": "So how do we tune these AI systems? Each"
  },
  {
    "startMs": 283120,
    "text": "loop generates useful information around"
  },
  {
    "startMs": 285520,
    "text": "harnesses, profiles, evals,"
  },
  {
    "startMs": 288400,
    "text": "models, resources, tools, and the"
  },
  {
    "startMs": 291199,
    "text": "environment. What you really want is"
  },
  {
    "startMs": 295199,
    "text": "to have a way to keep this portable, to"
  },
  {
    "startMs": 297440,
    "text": "have a way to version this and to evolve"
  },
  {
    "startMs": 299680,
    "text": "it over time. If you think about data"
  },
  {
    "startMs": 303759,
    "text": "recipes in research, this is how RL"
  },
  {
    "startMs": 307360,
    "text": "started to work really well. you"
  },
  {
    "startMs": 309039,
    "text": "understood the recipes and how to"
  },
  {
    "startMs": 311199,
    "text": "continuously change the recipe to combat"
  },
  {
    "startMs": 314160,
    "text": "some of the behaviors that may happen"
  },
  {
    "startMs": 315840,
    "text": "around hallucinations around reward"
  },
  {
    "startMs": 317759,
    "text": "hacking and then you get to a stack"
  },
  {
    "startMs": 320240,
    "text": "which is your final data recipe. We"
  },
  {
    "startMs": 322880,
    "text": "don't have that for harnesses. We don't"
  },
  {
    "startMs": 324320,
    "text": "have that for like AI systems in the"
  },
  {
    "startMs": 326639,
    "text": "general term. So we thought there's"
  },
  {
    "startMs": 328960,
    "text": "space for something like that. something"
  },
  {
    "startMs": 331039,
    "text": "that contains the evils and contains the"
  },
  {
    "startMs": 333520,
    "text": "tweaks and the human judgment and all"
  },
  {
    "startMs": 336000,
    "text": "these things that are not predetermined"
  },
  {
    "startMs": 337840,
    "text": "at the beginning but they're defined as"
  },
  {
    "startMs": 340320,
    "text": "you learn more about your agent acting"
  },
  {
    "startMs": 342960,
    "text": "in in in the environment."
  },
  {
    "startMs": 347039,
    "text": "We think recipes can be applied to this"
  },
  {
    "startMs": 349280,
    "text": "and we should use the same name. So an"
  },
  {
    "startMs": 352080,
    "text": "agent recipe is really something that"
  },
  {
    "startMs": 354080,
    "text": "enables you to create reproducible"
  },
  {
    "startMs": 356240,
    "text": "frontier AI systems. It's something that"
  },
  {
    "startMs": 359120,
    "text": "allows you to have a mode that keeps"
  },
  {
    "startMs": 362960,
    "text": "getting better over time, which is not"
  },
  {
    "startMs": 364960,
    "text": "tied to any platform or any provider."
  },
  {
    "startMs": 367520,
    "text": "It's something that you control lives in"
  },
  {
    "startMs": 370240,
    "text": "your company and is agnostic to the"
  },
  {
    "startMs": 373600,
    "text": "models and providers you use. And loops"
  },
  {
    "startMs": 376639,
    "text": "should focus on this. Loops should be"
  },
  {
    "startMs": 378479,
    "text": "the way you distill these systems into"
  },
  {
    "startMs": 380240,
    "text": "recipes."
  },
  {
    "startMs": 381759,
    "text": "Failure patterns should become judges"
  },
  {
    "startMs": 383919,
    "text": "and evals. Repeated behavior should"
  },
  {
    "startMs": 386240,
    "text": "become skills and prompts. user"
  },
  {
    "startMs": 387919,
    "text": "frustration, extensions and memories to"
  },
  {
    "startMs": 389680,
    "text": "your harness and so on. You we're all"
  },
  {
    "startMs": 392800,
    "text": "familiar with this, but we didn't have"
  },
  {
    "startMs": 394160,
    "text": "the the the right like terminology of"
  },
  {
    "startMs": 396000,
    "text": "how we should think about it and how we"
  },
  {
    "startMs": 398400,
    "text": "should define it. And we think recipes"
  },
  {
    "startMs": 400880,
    "text": "is a way to put everything together into"
  },
  {
    "startMs": 405440,
    "text": "a git repo and treat it as your ongoing"
  },
  {
    "startMs": 410639,
    "text": "um strategy for for uh building these"
  },
  {
    "startMs": 413520,
    "text": "self-improving systems. So we are"
  },
  {
    "startMs": 416319,
    "text": "introspection but you can think of"
  },
  {
    "startMs": 418080,
    "text": "introspection as the way you generate"
  },
  {
    "startMs": 419919,
    "text": "these recipes. So they're recipes for"
  },
  {
    "startMs": 422160,
    "text": "introspecting on your on your system. We"
  },
  {
    "startMs": 425360,
    "text": "wanted to build something that is"
  },
  {
    "startMs": 427280,
    "text": "portable and provider agnostic. So we"
  },
  {
    "startMs": 429840,
    "text": "built our um approach to recipes on the"
  },
  {
    "startMs": 433360,
    "text": "pi harness and on harbor for evals."
  },
  {
    "startMs": 437120,
    "text": "We baked it into uh git repos so uh"
  },
  {
    "startMs": 441919,
    "text": "everything could be versioned and agents"
  },
  {
    "startMs": 444080,
    "text": "could have a way to continuously track"
  },
  {
    "startMs": 446319,
    "text": "how this change and why and is meant to"
  },
  {
    "startMs": 449039,
    "text": "be owned by you but managed by your"
  },
  {
    "startMs": 450880,
    "text": "agents. And this is how products should"
  },
  {
    "startMs": 453039,
    "text": "really be built going forward. It's"
  },
  {
    "startMs": 454880,
    "text": "something that treats the owner as the"
  },
  {
    "startMs": 458639,
    "text": "um almost like the the the higher taste"
  },
  {
    "startMs": 462319,
    "text": "um personality in the room. But agents"
  },
  {
    "startMs": 464880,
    "text": "should try to calibrate themselves to to"
  },
  {
    "startMs": 467280,
    "text": "the taste of the of the maker. So we"
  },
  {
    "startMs": 470240,
    "text": "think recipes should be basically"
  },
  {
    "startMs": 472479,
    "text": "encoding the taste of the makers into"
  },
  {
    "startMs": 474879,
    "text": "how you build these agents. And if I"
  },
  {
    "startMs": 477199,
    "text": "want to use someone else's recipe, I"
  },
  {
    "startMs": 479599,
    "text": "should be able to also bring that taste."
  },
  {
    "startMs": 481520,
    "text": "It's not just the harness, it's not just"
  },
  {
    "startMs": 483039,
    "text": "the model, is how did you arrive at this"
  },
  {
    "startMs": 485919,
    "text": "particular recipe and why? And that's"
  },
  {
    "startMs": 488560,
    "text": "kind of like what uh what is behind uh"
  },
  {
    "startMs": 492000,
    "text": "reproducible"
  },
  {
    "startMs": 493919,
    "text": "um uh products and services around"
  },
  {
    "startMs": 496240,
    "text": "agents. Um we have an early release of"
  },
  {
    "startMs": 499520,
    "text": "recipes is called pi. Recipes. It's very"
  },
  {
    "startMs": 503120,
    "text": "similar to what skills uh used to be in"
  },
  {
    "startMs": 505360,
    "text": "2025 but is going a step forward. And"
  },
  {
    "startMs": 508400,
    "text": "this is what do I need to have a"
  },
  {
    "startMs": 510560,
    "text": "frontier agent is everything about how"
  },
  {
    "startMs": 513120,
    "text": "do I codify paste into evals? How do I"
  },
  {
    "startMs": 516000,
    "text": "run evals? How do we have the loops to"
  },
  {
    "startMs": 518479,
    "text": "continuously improve those evals over"
  },
  {
    "startMs": 520159,
    "text": "time? How do we process signals and know"
  },
  {
    "startMs": 522719,
    "text": "what are the right signals to to use? Um"
  },
  {
    "startMs": 525519,
    "text": "what are the right tools to work with"
  },
  {
    "startMs": 527120,
    "text": "certain models? How do I have different"
  },
  {
    "startMs": 529279,
    "text": "profiles of the harness to work with"
  },
  {
    "startMs": 531040,
    "text": "different models? Um and everything in"
  },
  {
    "startMs": 533680,
    "text": "between. So have a look at what we've"
  },
  {
    "startMs": 536480,
    "text": "been building here. It's still early uh"
  },
  {
    "startMs": 539040,
    "text": "but hopefully it's useful enough for you"
  },
  {
    "startMs": 541440,
    "text": "guys to to get going. And we feel this"
  },
  {
    "startMs": 543760,
    "text": "is going to grow into something that um"
  },
  {
    "startMs": 545920,
    "text": "really allows you to to use uh different"
  },
  {
    "startMs": 548880,
    "text": "um almost like different the to to be"
  },
  {
    "startMs": 551200,
    "text": "able to use the taste of of different"
  },
  {
    "startMs": 553360,
    "text": "makers as recipes for your agent."
  },
  {
    "startMs": 558160,
    "text": "And finally, the last point is valued"
  },
  {
    "startMs": 560080,
    "text": "work per watt. And why is this the score"
  },
  {
    "startMs": 562240,
    "text": "to really optimize for? Think of how um"
  },
  {
    "startMs": 566160,
    "text": "cursor and cognition went from building"
  },
  {
    "startMs": 568560,
    "text": "the best product to then building the"
  },
  {
    "startMs": 571519,
    "text": "best evals for the product and finally"
  },
  {
    "startMs": 573920,
    "text": "building the best models based on the"
  },
  {
    "startMs": 576080,
    "text": "previous two artifacts. We think this is"
  },
  {
    "startMs": 578880,
    "text": "like the recipe for everything going"
  },
  {
    "startMs": 580880,
    "text": "forward. Um code was the first domain"
  },
  {
    "startMs": 583200,
    "text": "where this um was successful. Um"
  },
  {
    "startMs": 586080,
    "text": "everything beyond customer support,"
  },
  {
    "startMs": 588320,
    "text": "legal research um everything is going to"
  },
  {
    "startMs": 591519,
    "text": "come down to this idea. How much value"
  },
  {
    "startMs": 593440,
    "text": "am I getting per watt? Um, how do I"
  },
  {
    "startMs": 596160,
    "text": "measure the value is the first step and"
  },
  {
    "startMs": 599120,
    "text": "how do I know I'm getting a good deal on"
  },
  {
    "startMs": 601680,
    "text": "that value is the second. And maybe this"
  },
  {
    "startMs": 604720,
    "text": "makes it a bit more clear. We've all"
  },
  {
    "startMs": 607200,
    "text": "started from a base harness and a base"
  },
  {
    "startMs": 611040,
    "text": "set of evals and we went to go to the"
  },
  {
    "startMs": 613440,
    "text": "frontier. Um, and you only go through"
  },
  {
    "startMs": 615839,
    "text": "that by running these systems in prod."
  },
  {
    "startMs": 617760,
    "text": "There's no way you you know what"
  },
  {
    "startMs": 619440,
    "text": "frontier is before you uh you start. Um"
  },
  {
    "startMs": 622640,
    "text": "but the the the last step here which is"
  },
  {
    "startMs": 625279,
    "text": "what is requiring a lot of research um"
  },
  {
    "startMs": 628320,
    "text": "is okay once you've reached frontier how"
  },
  {
    "startMs": 630959,
    "text": "do we make this um uh economically"
  },
  {
    "startMs": 634560,
    "text": "viable which is how do we not spend more"
  },
  {
    "startMs": 637200,
    "text": "than than uh we need for generating this"
  },
  {
    "startMs": 639839,
    "text": "amount of value um and we think we have"
  },
  {
    "startMs": 642640,
    "text": "the building blocks now to make this"
  },
  {
    "startMs": 645120,
    "text": "accessible and pretty efficient in the"
  },
  {
    "startMs": 647519,
    "text": "sense of you've seen all these"
  },
  {
    "startMs": 649360,
    "text": "fine-tuning APIs all the infrastructure"
  },
  {
    "startMs": 651600,
    "text": "that has been abstracted away for you to"
  },
  {
    "startMs": 654399,
    "text": "do do this process is just the knowhow"
  },
  {
    "startMs": 656720,
    "text": "that uh is not there yet and this is"
  },
  {
    "startMs": 659600,
    "text": "what we we we hope we can like push for"
  },
  {
    "startMs": 661920,
    "text": "the knowhow for knowing how to codify"
  },
  {
    "startMs": 664320,
    "text": "taste into evals and how to validate"
  },
  {
    "startMs": 666480,
    "text": "that in experiments um and you you've"
  },
  {
    "startMs": 669279,
    "text": "you've heard a lot about evals in"
  },
  {
    "startMs": 670800,
    "text": "experiments before but you didn't really"
  },
  {
    "startMs": 672720,
    "text": "think of them of like what are they is"
  },
  {
    "startMs": 674880,
    "text": "it's not just tests is is really what is"
  },
  {
    "startMs": 677519,
    "text": "the taste of the creator that agents"
  },
  {
    "startMs": 680399,
    "text": "should be able to reproduce and"
  },
  {
    "startMs": 682640,
    "text": "self-improve around. And no one has"
  },
  {
    "startMs": 685440,
    "text": "thought of how do I make this as"
  },
  {
    "startMs": 687760,
    "text": "portable enough? How how do I make my"
  },
  {
    "startMs": 689760,
    "text": "taste as an artist or as a software"
  },
  {
    "startMs": 693120,
    "text": "developer um something that anyone can"
  },
  {
    "startMs": 695600,
    "text": "download in their brain and be able to"
  },
  {
    "startMs": 697360,
    "text": "be a onetoone replica to me? And this is"
  },
  {
    "startMs": 699760,
    "text": "kind of like what RL is is is about now"
  },
  {
    "startMs": 702959,
    "text": "is how do we uh turn these um taste"
  },
  {
    "startMs": 707279,
    "text": "makers into uh environments and evals"
  },
  {
    "startMs": 710480,
    "text": "around them so then we can move them"
  },
  {
    "startMs": 712240,
    "text": "into the weights. But um there's more"
  },
  {
    "startMs": 714800,
    "text": "than that. Um you can think of the"
  },
  {
    "startMs": 716959,
    "text": "worker as the inner loop and it"
  },
  {
    "startMs": 718959,
    "text": "generates all these artifacts. But how"
  },
  {
    "startMs": 720880,
    "text": "you look at the artifacts and know what"
  },
  {
    "startMs": 723600,
    "text": "to change is the taste. Uh and this is"
  },
  {
    "startMs": 726399,
    "text": "what creates candidates of what you"
  },
  {
    "startMs": 729360,
    "text": "should change and how you should adapt"
  },
  {
    "startMs": 730959,
    "text": "based on that. And experiments is what"
  },
  {
    "startMs": 733200,
    "text": "how you self-calibrate that okay my"
  },
  {
    "startMs": 735760,
    "text": "taste is actually validated in"
  },
  {
    "startMs": 737920,
    "text": "production with users and we make sure"
  },
  {
    "startMs": 740000,
    "text": "that not only the maker is happy through"
  },
  {
    "startMs": 742320,
    "text": "the um offline evals but the end users"
  },
  {
    "startMs": 745360,
    "text": "are happy as well and they agree with"
  },
  {
    "startMs": 747279,
    "text": "what we consider good."
  },
  {
    "startMs": 750480,
    "text": "Let's go through a practical example of"
  },
  {
    "startMs": 752240,
    "text": "how this works."
  },
  {
    "startMs": 756000,
    "text": "Let's take a baseline um agent which"
  },
  {
    "startMs": 758959,
    "text": "could be a talent sourcing agent. Um and"
  },
  {
    "startMs": 761920,
    "text": "this is a very classical case of"
  },
  {
    "startMs": 765360,
    "text": "everyone is doing recruiting differently"
  },
  {
    "startMs": 767279,
    "text": "and is very much"
  },
  {
    "startMs": 769680,
    "text": "about not what is good recruiting but"
  },
  {
    "startMs": 773120,
    "text": "who is leading that recruiting that"
  },
  {
    "startMs": 775519,
    "text": "considers recruiting as good. So in this"
  },
  {
    "startMs": 777519,
    "text": "case we're starting with something"
  },
  {
    "startMs": 778639,
    "text": "pretty simple. um a a bunch of tools,"
  },
  {
    "startMs": 781360,
    "text": "web search, LinkedIn, uh a bunch of sub"
  },
  {
    "startMs": 784160,
    "text": "aents that have been pre-popularized by"
  },
  {
    "startMs": 787040,
    "text": "harnesses like codeex and cloud code and"
  },
  {
    "startMs": 789920,
    "text": "uh system instruction which is about"
  },
  {
    "startMs": 791920,
    "text": "your recruiter."
  },
  {
    "startMs": 794959,
    "text": "First step is really understand the"
  },
  {
    "startMs": 796800,
    "text": "signals. So you can think of patterns as"
  },
  {
    "startMs": 800000,
    "text": "being a way to look at the traces,"
  },
  {
    "startMs": 802079,
    "text": "extract some common um behaviors or"
  },
  {
    "startMs": 805680,
    "text": "common user frustrations and turn them"
  },
  {
    "startMs": 808000,
    "text": "into like a cluster. So let's say this"
  },
  {
    "startMs": 810720,
    "text": "idea of uh the agent is going uh and"
  },
  {
    "startMs": 813600,
    "text": "reaching out to a lot of big tech"
  },
  {
    "startMs": 815200,
    "text": "employees. As a recruiter, you don't"
  },
  {
    "startMs": 817440,
    "text": "really want that. You want to find"
  },
  {
    "startMs": 818800,
    "text": "hidden gems. You don't want to try to"
  },
  {
    "startMs": 820959,
    "text": "hire John Carmarmac. But an agent would"
  },
  {
    "startMs": 824240,
    "text": "think that's, oh, John Carmarmac is"
  },
  {
    "startMs": 825920,
    "text": "great. why would I not reach out to him?"
  },
  {
    "startMs": 827920,
    "text": "Um, so so this is a behavior that you"
  },
  {
    "startMs": 830959,
    "text": "you'd never think of codifying, but you"
  },
  {
    "startMs": 833600,
    "text": "discover the agent tends to do that. Um,"
  },
  {
    "startMs": 837040,
    "text": "patterns is how you discover these"
  },
  {
    "startMs": 838959,
    "text": "signals and inform you what you should"
  },
  {
    "startMs": 841279,
    "text": "do next."
  },
  {
    "startMs": 843600,
    "text": "um calibration judges and evals is how"
  },
  {
    "startMs": 847600,
    "text": "we used to think about how do we codify"
  },
  {
    "startMs": 850480,
    "text": "these these behaviors into um something"
  },
  {
    "startMs": 853279,
    "text": "that can try to uh apply the same"
  },
  {
    "startMs": 855279,
    "text": "judgment across traces and across uh"
  },
  {
    "startMs": 857440,
    "text": "execution. So let's say we we build an"
  },
  {
    "startMs": 860639,
    "text": "agent that looks at a trajectory and um"
  },
  {
    "startMs": 864480,
    "text": "identifies exactly that pattern. Hey,"
  },
  {
    "startMs": 866399,
    "text": "did did this agent reach out to Google"
  },
  {
    "startMs": 868720,
    "text": "employees instead of trying to uh find"
  },
  {
    "startMs": 871199,
    "text": "hidden gems on GitHub? Um, and the"
  },
  {
    "startMs": 874399,
    "text": "calibration bit and the eval generation"
  },
  {
    "startMs": 876959,
    "text": "bit is not that hard. It it it should be"
  },
  {
    "startMs": 879519,
    "text": "doable by agents to build. You just need"
  },
  {
    "startMs": 881680,
    "text": "a human in the loop to say, hey, um,"
  },
  {
    "startMs": 883680,
    "text": "this is the approach we're taking. Do"
  },
  {
    "startMs": 885680,
    "text": "you agree with this"
  },
  {
    "startMs": 888160,
    "text": "judgment? Do you really agree that we"
  },
  {
    "startMs": 890560,
    "text": "should look more towards hidden gems"
  },
  {
    "startMs": 892639,
    "text": "rather than reach out to um um big tech"
  },
  {
    "startMs": 896959,
    "text": "employees? And that's about it. You"
  },
  {
    "startMs": 898959,
    "text": "don't need the human to actually build"
  },
  {
    "startMs": 900480,
    "text": "the evals. You need them to calibrate"
  },
  {
    "startMs": 902639,
    "text": "the evals. And agents should be the ones"
  },
  {
    "startMs": 905199,
    "text": "that really take the the the taste of"
  },
  {
    "startMs": 907600,
    "text": "the maker and and put them in into code."
  },
  {
    "startMs": 909920,
    "text": "Once you have this, it's pretty easy to"
  },
  {
    "startMs": 912160,
    "text": "create recipe candidates. And this"
  },
  {
    "startMs": 913920,
    "text": "should be the the diffs that you really"
  },
  {
    "startMs": 915920,
    "text": "want to taste. Um, and"
  },
  {
    "startMs": 919360,
    "text": "you can have a pretty good offline evil"
  },
  {
    "startMs": 921360,
    "text": "set around this, but the the the test"
  },
  {
    "startMs": 923920,
    "text": "here is when you go to prod. So, do the"
  },
  {
    "startMs": 927360,
    "text": "end user agree with your taste of not"
  },
  {
    "startMs": 930639,
    "text": "hitting up um big tech uh employees,"
  },
  {
    "startMs": 934720,
    "text": "right? And this is kind of like what you"
  },
  {
    "startMs": 936079,
    "text": "want is you build a product that really"
  },
  {
    "startMs": 938160,
    "text": "emphasizes your taste and then you you"
  },
  {
    "startMs": 940800,
    "text": "make sure that your users appreciate and"
  },
  {
    "startMs": 943279,
    "text": "value that taste. and AB tests have been"
  },
  {
    "startMs": 946160,
    "text": "a way to to to make sure that that's the"
  },
  {
    "startMs": 948399,
    "text": "case. Um so with a multi-arm banded um"
  },
  {
    "startMs": 951920,
    "text": "scenario for example you you'd be able"
  },
  {
    "startMs": 953600,
    "text": "to do that pretty well. So once you"
  },
  {
    "startMs": 955199,
    "text": "validate okay I have great taste and my"
  },
  {
    "startMs": 957279,
    "text": "users believe uh I have great taste as"
  },
  {
    "startMs": 959600,
    "text": "well that's when you promote and that's"
  },
  {
    "startMs": 961120,
    "text": "kind of when you go to to the next"
  },
  {
    "startMs": 963600,
    "text": "version of an agent recipe. The secret"
  },
  {
    "startMs": 966880,
    "text": "is you keep doing this over and over"
  },
  {
    "startMs": 968320,
    "text": "again and you know how to continuously"
  },
  {
    "startMs": 970560,
    "text": "codify your taste and your um what what"
  },
  {
    "startMs": 973600,
    "text": "what good is to you into an agent that"
  },
  {
    "startMs": 976639,
    "text": "can reproduce the same service or"
  },
  {
    "startMs": 978480,
    "text": "product uh for other people and they"
  },
  {
    "startMs": 980880,
    "text": "also agree you have great taste and you"
  },
  {
    "startMs": 983040,
    "text": "have great execution. And this is really"
  },
  {
    "startMs": 984959,
    "text": "kind of like the the secret of building"
  },
  {
    "startMs": 986399,
    "text": "good loops is okay can can someone"
  },
  {
    "startMs": 988959,
    "text": "iterate on my um system in a way as uh"
  },
  {
    "startMs": 993600,
    "text": "you know um a good example here is like"
  },
  {
    "startMs": 995759,
    "text": "Miranda from the Delor product right"
  },
  {
    "startMs": 998320,
    "text": "what would Miranda do uh in certain"
  },
  {
    "startMs": 1000399,
    "text": "cases and you kind of want to codify"
  },
  {
    "startMs": 1002399,
    "text": "that that thinking into like agents that"
  },
  {
    "startMs": 1004399,
    "text": "can do the same stuff at a higher level."
  },
  {
    "startMs": 1008320,
    "text": "So the takeaways are this. Um the loop"
  },
  {
    "startMs": 1010240,
    "text": "is the product. You try to automate"
  },
  {
    "startMs": 1011920,
    "text": "yourself as the u as a um higher level"
  },
  {
    "startMs": 1016320,
    "text": "judge and you want to make sure your"
  },
  {
    "startMs": 1019360,
    "text": "second loop agents are able to apply the"
  },
  {
    "startMs": 1022000,
    "text": "same judgment to to the agents you're"
  },
  {
    "startMs": 1024000,
    "text": "trying to to to push to prod. Second bit"
  },
  {
    "startMs": 1027280,
    "text": "system dissolation is the mode. So, how"
  },
  {
    "startMs": 1029120,
    "text": "do you continuously inject that taste"
  },
  {
    "startMs": 1031280,
    "text": "into these uh workers and they how how"
  },
  {
    "startMs": 1033679,
    "text": "they continuously self-verify and work"
  },
  {
    "startMs": 1035679,
    "text": "together is uh the biggest thing that"
  },
  {
    "startMs": 1038160,
    "text": "you should focus on and the faster you"
  },
  {
    "startMs": 1040160,
    "text": "do it uh the the the faster you you"
  },
  {
    "startMs": 1043039,
    "text": "build a defensible"
  },
  {
    "startMs": 1045039,
    "text": "um approach to to becoming a vertical AI"
  },
  {
    "startMs": 1047760,
    "text": "company. And finally, valued work per"
  },
  {
    "startMs": 1050320,
    "text": "watt is how you should measure um am I"
  },
  {
    "startMs": 1053679,
    "text": "making progress or not. So first make"
  },
  {
    "startMs": 1056240,
    "text": "sure that uh the the the work you're"
  },
  {
    "startMs": 1058640,
    "text": "generating is valuable. Second make sure"
  },
  {
    "startMs": 1060799,
    "text": "that the economics makes sense and the"
  },
  {
    "startMs": 1064000,
    "text": "um the the difference in price is is"
  },
  {
    "startMs": 1065919,
    "text": "basically what um people would would"
  },
  {
    "startMs": 1068160,
    "text": "switch away from cloud code to to"
  },
  {
    "startMs": 1070320,
    "text": "something you provide."
  },
  {
    "startMs": 1073039,
    "text": "We've been thinking a lot about these"
  },
  {
    "startMs": 1074400,
    "text": "ideas and we're building some very"
  },
  {
    "startMs": 1076480,
    "text": "interesting products around how to"
  },
  {
    "startMs": 1077919,
    "text": "deploy this in production. We'd love to"
  },
  {
    "startMs": 1079760,
    "text": "hear from you. would love to get um to"
  },
  {
    "startMs": 1082559,
    "text": "to understand more about how how certain"
  },
  {
    "startMs": 1085840,
    "text": "um vertical SAS companies are are"
  },
  {
    "startMs": 1087919,
    "text": "looking to go to prod with um or how"
  },
  {
    "startMs": 1090960,
    "text": "agent labs have been thinking about this"
  },
  {
    "startMs": 1093200,
    "text": "idea of um um creating these like auto"
  },
  {
    "startMs": 1096240,
    "text": "research uh labs around their their own"
  },
  {
    "startMs": 1099200,
    "text": "products. um get in touch. Uh we're"
  },
  {
    "startMs": 1101600,
    "text": "gonna be around the block for for"
  },
  {
    "startMs": 1103679,
    "text": "chatting more about this. And thank you"
  },
  {
    "startMs": 1105840,
    "text": "very much."
  }
]