[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12640,
    "text": ">> Um well, thanks all for your time."
  },
  {
    "startMs": 14320,
    "text": "Really appreciate you dropping by and"
  },
  {
    "startMs": 16960,
    "text": "it's always a a great honor to speak at"
  },
  {
    "startMs": 18760,
    "text": "the world's fair. So, I'll do my best to"
  },
  {
    "startMs": 21000,
    "text": "uh"
  },
  {
    "startMs": 21560,
    "text": "give you guys some valuable insights and"
  },
  {
    "startMs": 23520,
    "text": "um"
  },
  {
    "startMs": 24080,
    "text": "yeah, hopefully make it worth your time."
  },
  {
    "startMs": 26040,
    "text": "So, my name's Maximilian Piros and today"
  },
  {
    "startMs": 28400,
    "text": "I'll be talking about mouse power."
  },
  {
    "startMs": 31160,
    "text": "And this is a talk about measuring"
  },
  {
    "startMs": 33200,
    "text": "agents through mental models."
  },
  {
    "startMs": 35760,
    "text": "But before I get into talking about"
  },
  {
    "startMs": 37280,
    "text": "measuring agents, I'm going to talk"
  },
  {
    "startMs": 39040,
    "text": "through a bit about how I use them every"
  },
  {
    "startMs": 40760,
    "text": "day. And it might seem familiar to you,"
  },
  {
    "startMs": 43040,
    "text": "but just to level set, we'll go through"
  },
  {
    "startMs": 44280,
    "text": "it. So, uh I tend to background them"
  },
  {
    "startMs": 46880,
    "text": "like I'm sure a lot of you people are as"
  },
  {
    "startMs": 48640,
    "text": "well."
  },
  {
    "startMs": 49520,
    "text": "Um so, while my active attention is"
  },
  {
    "startMs": 51600,
    "text": "focusing on one thing, like perhaps"
  },
  {
    "startMs": 53520,
    "text": "giving this talk to you, I still want to"
  },
  {
    "startMs": 55400,
    "text": "make some progress on peripheral tasks."
  },
  {
    "startMs": 57200,
    "text": "So, I'll keep my attention focused on"
  },
  {
    "startMs": 60520,
    "text": "giving this talk, while my agents can"
  },
  {
    "startMs": 63240,
    "text": "help me explore some designs in the"
  },
  {
    "startMs": 64720,
    "text": "background cuz I think that uh my slides"
  },
  {
    "startMs": 67200,
    "text": "need a bit of work. So, you know, I've"
  },
  {
    "startMs": 68680,
    "text": "got my design system already set up."
  },
  {
    "startMs": 70440,
    "text": "I've got some guidance uh given to my"
  },
  {
    "startMs": 72640,
    "text": "agents. So, I'll kick off an agent to"
  },
  {
    "startMs": 74520,
    "text": "just try to explore some different"
  },
  {
    "startMs": 76000,
    "text": "directions on the type type two"
  },
  {
    "startMs": 77560,
    "text": "treatment and the layout. And um you"
  },
  {
    "startMs": 80240,
    "text": "know, just try to get as many"
  },
  {
    "startMs": 81320,
    "text": "explorations as possible."
  },
  {
    "startMs": 83200,
    "text": "But of course, uh one agent's never"
  },
  {
    "startMs": 84680,
    "text": "enough. So, I like to kick off a bunch"
  },
  {
    "startMs": 86720,
    "text": "in parallel. You know, I've got a lot of"
  },
  {
    "startMs": 88280,
    "text": "slides to get through. So, I need all of"
  },
  {
    "startMs": 90640,
    "text": "my agents on and exploring it in"
  },
  {
    "startMs": 92040,
    "text": "different directions and"
  },
  {
    "startMs": 94040,
    "text": "uh hopefully I can get some interesting"
  },
  {
    "startMs": 95720,
    "text": "things to make my slides a bit better."
  },
  {
    "startMs": 98240,
    "text": "And uh hopefully they can finish the job"
  },
  {
    "startMs": 100480,
    "text": "pretty soon cuz we're obviously kind of"
  },
  {
    "startMs": 102000,
    "text": "up against uh the timeline the the uh"
  },
  {
    "startMs": 104240,
    "text": "deadline here."
  },
  {
    "startMs": 106080,
    "text": "So, um"
  },
  {
    "startMs": 107560,
    "text": "this is generally how I work. I'm sure"
  },
  {
    "startMs": 108880,
    "text": "it's probably familiar to a lot of you"
  },
  {
    "startMs": 110080,
    "text": "where we're just trying to kick off"
  },
  {
    "startMs": 111440,
    "text": "agents for as much as possible in"
  },
  {
    "startMs": 113120,
    "text": "parallel cuz it always feels like"
  },
  {
    "startMs": 115080,
    "text": "there's just way more research to do. We"
  },
  {
    "startMs": 117960,
    "text": "want it to be as thorough as possible."
  },
  {
    "startMs": 119720,
    "text": "There's way more design explorations to"
  },
  {
    "startMs": 121480,
    "text": "do. So, whenever our main focus is on"
  },
  {
    "startMs": 123800,
    "text": "one thing, why not kick a bunch of"
  },
  {
    "startMs": 125400,
    "text": "agents off in parallel and just try to"
  },
  {
    "startMs": 127760,
    "text": "maximize your time. And it's a lot of"
  },
  {
    "startMs": 130119,
    "text": "fun, of course, until you get the bill."
  },
  {
    "startMs": 133120,
    "text": "And then you start to wonder, was it all"
  },
  {
    "startMs": 135760,
    "text": "worth it? Right? Did"
  },
  {
    "startMs": 137600,
    "text": "did you vibe code too hard?"
  },
  {
    "startMs": 140400,
    "text": "Were you token maxing too much? Like,"
  },
  {
    "startMs": 142600,
    "text": "could you have been more efficient in"
  },
  {
    "startMs": 144680,
    "text": "how you approached"
  },
  {
    "startMs": 146120,
    "text": "your sequencing your agents?"
  },
  {
    "startMs": 148520,
    "text": "And so, this is what I'm going to get"
  },
  {
    "startMs": 149600,
    "text": "into today. It's It's how do we evaluate"
  },
  {
    "startMs": 151760,
    "text": "the token cost? And specifically, how do"
  },
  {
    "startMs": 153760,
    "text": "we help our customers value it?"
  },
  {
    "startMs": 156040,
    "text": "So, for the past year and a half, I've"
  },
  {
    "startMs": 158120,
    "text": "had the pleasure of working as a"
  },
  {
    "startMs": 159360,
    "text": "founding designer at a company called"
  },
  {
    "startMs": 160800,
    "text": "території and we focus on computer use"
  },
  {
    "startMs": 163040,
    "text": "models. These are models that learn to"
  },
  {
    "startMs": 165120,
    "text": "use a computer like a human would and"
  },
  {
    "startMs": 167400,
    "text": "the use case for them is when you can't"
  },
  {
    "startMs": 169959,
    "text": "get information from an API or an MCP,"
  },
  {
    "startMs": 172640,
    "text": "why not just send an agent out to use"
  },
  {
    "startMs": 174959,
    "text": "the computer like a human would and then"
  },
  {
    "startMs": 176440,
    "text": "we can extract all types of data and"
  },
  {
    "startMs": 178360,
    "text": "manipulate it in ways that let us access"
  },
  {
    "startMs": 181160,
    "text": "all the stuff that wasn't accessible"
  },
  {
    "startMs": 183239,
    "text": "previously. So, obviously less efficient"
  },
  {
    "startMs": 184800,
    "text": "than APIs and MCPs, but as a last"
  },
  {
    "startMs": 187400,
    "text": "resort, just have an agent go use the"
  },
  {
    "startMs": 189560,
    "text": "computer and try to get the information."
  },
  {
    "startMs": 192320,
    "text": "Here's the U території agent using the U"
  },
  {
    "startMs": 194760,
    "text": "території website."
  },
  {
    "startMs": 196160,
    "text": "Um checking out its own benchmark. So,"
  },
  {
    "startMs": 198360,
    "text": "kind of it's admire itself in a way. So,"
  },
  {
    "startMs": 201440,
    "text": "yeah, it gets it gets a bit weird. Like,"
  },
  {
    "startMs": 203680,
    "text": "and a lot of what I do as a founding"
  },
  {
    "startMs": 205440,
    "text": "designer there is talk to customers, try"
  },
  {
    "startMs": 207280,
    "text": "to understand how can we make agents as"
  },
  {
    "startMs": 209360,
    "text": "intuitive as possible, how do we figure"
  },
  {
    "startMs": 211200,
    "text": "out the mental models they're using to"
  },
  {
    "startMs": 213000,
    "text": "value the use cases they want to send"
  },
  {
    "startMs": 215239,
    "text": "out agents for."
  },
  {
    "startMs": 216680,
    "text": "And a lot of them do seem pretty"
  },
  {
    "startMs": 219000,
    "text": "confused so far. A lot of people are"
  },
  {
    "startMs": 221040,
    "text": "excited about agents, but the phrase"
  },
  {
    "startMs": 223160,
    "text": "that comes up quite often is that they"
  },
  {
    "startMs": 225000,
    "text": "feel like they're just scratching the"
  },
  {
    "startMs": 226000,
    "text": "surface. Um"
  },
  {
    "startMs": 227440,
    "text": "it seems like it's not quite intuitive"
  },
  {
    "startMs": 228840,
    "text": "how we can best use them yet. And so, in"
  },
  {
    "startMs": 231400,
    "text": "a lot of my uh customer discussions,"
  },
  {
    "startMs": 233560,
    "text": "it's it always comes down to a question"
  },
  {
    "startMs": 235240,
    "text": "of like, what is the best way to use use"
  },
  {
    "startMs": 237040,
    "text": "uh to use agents? What are the best use"
  },
  {
    "startMs": 238840,
    "text": "cases for them? And how do I think about"
  },
  {
    "startMs": 240400,
    "text": "the trade-offs with regards to token"
  },
  {
    "startMs": 242480,
    "text": "costs relative to value? So, I think"
  },
  {
    "startMs": 244600,
    "text": "we're still kind of building this muscle"
  },
  {
    "startMs": 246200,
    "text": "today. And this leads me to the thesis"
  },
  {
    "startMs": 248920,
    "text": "of the talk, which is that I think"
  },
  {
    "startMs": 250600,
    "text": "agents have a measurement problem."
  },
  {
    "startMs": 253160,
    "text": "And uh as an example, here's me at work"
  },
  {
    "startMs": 255959,
    "text": "trying to measure some agents, and one"
  },
  {
    "startMs": 257760,
    "text": "of my co-workers took this photo and"
  },
  {
    "startMs": 259320,
    "text": "told me I looked like"
  },
  {
    "startMs": 260600,
    "text": "I was uh trying to solve the mystery of"
  },
  {
    "startMs": 262160,
    "text": "Pepe Silvia. So,"
  },
  {
    "startMs": 263880,
    "text": "uh"
  },
  {
    "startMs": 264440,
    "text": "as you can see, it's it's"
  },
  {
    "startMs": 266400,
    "text": "it's not a it's not an easy task to to"
  },
  {
    "startMs": 268720,
    "text": "measure agents. But, I'm sure some of"
  },
  {
    "startMs": 270360,
    "text": "you are saying, \"Hold on a sec. Like,"
  },
  {
    "startMs": 272760,
    "text": "what is this guy talking about? I've got"
  },
  {
    "startMs": 274520,
    "text": "a fleet of agents working for me right"
  },
  {
    "startMs": 275840,
    "text": "now. We're building our next"
  },
  {
    "startMs": 277480,
    "text": "million-dollar app as we speak, and I'm"
  },
  {
    "startMs": 279640,
    "text": "having a a totally fine time uh"
  },
  {
    "startMs": 281800,
    "text": "measuring my agents.\" Uh to which I will"
  },
  {
    "startMs": 283960,
    "text": "agree with you, uh but then I will point"
  },
  {
    "startMs": 285880,
    "text": "you to the mandatory Upton Sinclair"
  },
  {
    "startMs": 288200,
    "text": "quote to remind us all that everybody in"
  },
  {
    "startMs": 290360,
    "text": "this room is very biased, and we're"
  },
  {
    "startMs": 292720,
    "text": "early adopters, and we're very excited"
  },
  {
    "startMs": 295400,
    "text": "to explore this new technology, but it"
  },
  {
    "startMs": 297040,
    "text": "doesn't mean that we represent the"
  },
  {
    "startMs": 298200,
    "text": "people that ultimately we're going to be"
  },
  {
    "startMs": 300160,
    "text": "trying to help adopt this technology."
  },
  {
    "startMs": 302800,
    "text": "And so, you know, I think it's important"
  },
  {
    "startMs": 304800,
    "text": "to remind ourselves that in some way or"
  },
  {
    "startMs": 306200,
    "text": "another, we probably are selling tokens,"
  },
  {
    "startMs": 308840,
    "text": "whether it's indirectly or directly. And"
  },
  {
    "startMs": 311000,
    "text": "so, when we think about our own token"
  },
  {
    "startMs": 312240,
    "text": "usage, uh is it really representative of"
  },
  {
    "startMs": 314560,
    "text": "all the people out there who have never"
  },
  {
    "startMs": 315840,
    "text": "touched an agent yet? Uh some people are"
  },
  {
    "startMs": 318120,
    "text": "still copy and pasting into ChatGPT."
  },
  {
    "startMs": 320880,
    "text": "I may be [REDACTED] to one of these people,"
  },
  {
    "startMs": 322640,
    "text": "and despite how much I tried to get her"
  },
  {
    "startMs": 324080,
    "text": "to try out agents, she's not let me uh"
  },
  {
    "startMs": 326600,
    "text": "set her up with it yet. And so, um as a"
  },
  {
    "startMs": 329040,
    "text": "reminder, uh when we think about helping"
  },
  {
    "startMs": 331080,
    "text": "people adopt agents, you know, all the"
  },
  {
    "startMs": 332720,
    "text": "people across the world that we think"
  },
  {
    "startMs": 334440,
    "text": "could get as much uh excitement and"
  },
  {
    "startMs": 336000,
    "text": "value as as we do when we run off"
  },
  {
    "startMs": 337880,
    "text": "parallel agents, let's uh just remember"
  },
  {
    "startMs": 339440,
    "text": "this quote."
  },
  {
    "startMs": 340603,
    "text": ">> [snorts]"
  },
  {
    "startMs": 341120,
    "text": ">> And um"
  },
  {
    "startMs": 342440,
    "text": "so, it really boils down to the age-old"
  },
  {
    "startMs": 344800,
    "text": "problem of a new technology."
  },
  {
    "startMs": 347120,
    "text": "And of course, there's tons of history"
  },
  {
    "startMs": 349160,
    "text": "we can go to to study how people solved"
  },
  {
    "startMs": 351480,
    "text": "this in the past. We have this really"
  },
  {
    "startMs": 353080,
    "text": "exciting new thing, but we haven't quite"
  },
  {
    "startMs": 355400,
    "text": "uh figured out the right ways to"
  },
  {
    "startMs": 356360,
    "text": "communicate it."
  },
  {
    "startMs": 357760,
    "text": "And so for this talk, I'll go back to"
  },
  {
    "startMs": 359400,
    "text": "the 1700s and we can take some notes"
  },
  {
    "startMs": 361920,
    "text": "from when James Watt was trying to sell"
  },
  {
    "startMs": 364440,
    "text": "steam engines."
  },
  {
    "startMs": 366000,
    "text": "And at the time he decided that uh a"
  },
  {
    "startMs": 368000,
    "text": "great use case for his steam engines was"
  },
  {
    "startMs": 369960,
    "text": "trying to replace a horse gin. And these"
  },
  {
    "startMs": 373120,
    "text": "are the was the power source of a mill"
  },
  {
    "startMs": 375160,
    "text": "at the time. So when you're"
  },
  {
    "startMs": 376960,
    "text": "for uh let's say a brewery and you you"
  },
  {
    "startMs": 378800,
    "text": "need some power source to to grind your"
  },
  {
    "startMs": 381200,
    "text": "barley or whatever. I don't know. I'm"
  },
  {
    "startMs": 382080,
    "text": "not I'm not like a big brewery guy, so I"
  },
  {
    "startMs": 383720,
    "text": "don't know exactly what how it's made,"
  },
  {
    "startMs": 385320,
    "text": "but you need a power source and the"
  },
  {
    "startMs": 387080,
    "text": "power source at the time that was common"
  },
  {
    "startMs": 389080,
    "text": "was you hooked a horse up to a rotary"
  },
  {
    "startMs": 390760,
    "text": "arm and the horse walked in a circle and"
  },
  {
    "startMs": 392560,
    "text": "that's how you generated your power."
  },
  {
    "startMs": 394440,
    "text": "Uh and seems crazy today maybe, but um"
  },
  {
    "startMs": 397080,
    "text": "at the time was commonplace and Watt"
  },
  {
    "startMs": 398640,
    "text": "thought, you know, it would be much"
  },
  {
    "startMs": 400320,
    "text": "better than a horse is like a very"
  },
  {
    "startMs": 401680,
    "text": "efficient machine."
  },
  {
    "startMs": 403240,
    "text": "Although he um rightfully acknowledged"
  },
  {
    "startMs": 405560,
    "text": "that one of the big barriers to adopting"
  },
  {
    "startMs": 407720,
    "text": "it would be this cognitive dissonance of"
  },
  {
    "startMs": 410280,
    "text": "trying to tell people who kind of think"
  },
  {
    "startMs": 412120,
    "text": "in horses, how do you adapt to this to"
  },
  {
    "startMs": 414520,
    "text": "this uh cold machine that's kind of"
  },
  {
    "startMs": 416840,
    "text": "intimidating and scary and perhaps uh"
  },
  {
    "startMs": 420000,
    "text": "somebody's going to say it's going to"
  },
  {
    "startMs": 420960,
    "text": "solve all your problems, but you you"
  },
  {
    "startMs": 422400,
    "text": "can't quite see the vision yet. So uh"
  },
  {
    "startMs": 424640,
    "text": "perhaps that sounds familiar to any of"
  },
  {
    "startMs": 426440,
    "text": "us working in agents today."
  },
  {
    "startMs": 428280,
    "text": "And Watt's solution was that he needed"
  },
  {
    "startMs": 430160,
    "text": "to understand um the mental model of"
  },
  {
    "startMs": 432160,
    "text": "these people and specifically to create"
  },
  {
    "startMs": 433800,
    "text": "a metric that would help him uh give"
  },
  {
    "startMs": 436280,
    "text": "some baseline of the relative"
  },
  {
    "startMs": 437760,
    "text": "improvement in efficiency."
  },
  {
    "startMs": 439720,
    "text": "And so he literally studied uh horse"
  },
  {
    "startMs": 442680,
    "text": "gins and tried to get some kind of"
  },
  {
    "startMs": 445680,
    "text": "armchair measurements of of how is uh"
  },
  {
    "startMs": 448080,
    "text": "like where the mechanics and the average"
  },
  {
    "startMs": 449840,
    "text": "um performance of it and eventually came"
  },
  {
    "startMs": 452480,
    "text": "to a metric called horsepower,"
  },
  {
    "startMs": 455440,
    "text": "which may sound familiar."
  },
  {
    "startMs": 457120,
    "text": "And uh he used this measure to, you"
  },
  {
    "startMs": 459720,
    "text": "know, this was to quantify the the"
  },
  {
    "startMs": 461480,
    "text": "general power that the horses were were"
  },
  {
    "startMs": 463440,
    "text": "um creating at the time and then he"
  },
  {
    "startMs": 465040,
    "text": "could use that as a basis to show the"
  },
  {
    "startMs": 466520,
    "text": "multiplier of efficiency that a steam"
  },
  {
    "startMs": 468080,
    "text": "engine could provide."
  },
  {
    "startMs": 469600,
    "text": "And this metric was not very scientific"
  },
  {
    "startMs": 472320,
    "text": "at the time. It was not necessarily even"
  },
  {
    "startMs": 474760,
    "text": "accurate, you could say. But the main"
  },
  {
    "startMs": 477000,
    "text": "thing it did was it communicated an"
  },
  {
    "startMs": 479240,
    "text": "increase in value and so this led people"
  },
  {
    "startMs": 481560,
    "text": "who love horses"
  },
  {
    "startMs": 483400,
    "text": "let them kind of calibrate their"
  },
  {
    "startMs": 487000,
    "text": "the the efficiency gains that they could"
  },
  {
    "startMs": 488480,
    "text": "get by by attempting to adopt a steam"
  },
  {
    "startMs": 491920,
    "text": "engine. So not even necessarily um"
  },
  {
    "startMs": 494480,
    "text": "what you would get when you use it, but"
  },
  {
    "startMs": 496040,
    "text": "what would get you over the limit of"
  },
  {
    "startMs": 497480,
    "text": "trying it out in the first place."
  },
  {
    "startMs": 500400,
    "text": "And you know, it's it's it's a pretty"
  },
  {
    "startMs": 501919,
    "text": "big feat because like although um"
  },
  {
    "startMs": 504360,
    "text": "he had efficiency on his side with"
  },
  {
    "startMs": 506320,
    "text": "regards to this metric"
  },
  {
    "startMs": 507800,
    "text": "you know, let's be honest regardless of"
  },
  {
    "startMs": 509720,
    "text": "how efficient this was"
  },
  {
    "startMs": 511960,
    "text": "horses just have"
  },
  {
    "startMs": 513599,
    "text": "great vibes. So like it's kind of hard"
  },
  {
    "startMs": 516000,
    "text": "to beat the vibes of horses and so he"
  },
  {
    "startMs": 517360,
    "text": "knew he had to kind of overcome the"
  },
  {
    "startMs": 519080,
    "text": "emotion"
  },
  {
    "startMs": 520320,
    "text": "and actually speak to to something that"
  },
  {
    "startMs": 522320,
    "text": "gave them an ability to calculate the"
  },
  {
    "startMs": 524440,
    "text": "ROI."
  },
  {
    "startMs": 526200,
    "text": "And oh, sorry. Skipped something."
  },
  {
    "startMs": 528880,
    "text": "And so yeah, the the lesson being if"
  },
  {
    "startMs": 530960,
    "text": "we're not able to give something that is"
  },
  {
    "startMs": 532839,
    "text": "a tangible ROI for our customers, then"
  },
  {
    "startMs": 535680,
    "text": "it's very hard for us to communicate"
  },
  {
    "startMs": 537200,
    "text": "value."
  },
  {
    "startMs": 538640,
    "text": "And I think we only need look to our own"
  },
  {
    "startMs": 540880,
    "text": "industry to see all the examples where"
  },
  {
    "startMs": 543560,
    "text": "other people in the technology sector"
  },
  {
    "startMs": 545120,
    "text": "are failing to calculate good ROIs as"
  },
  {
    "startMs": 547200,
    "text": "well."
  },
  {
    "startMs": 548000,
    "text": "And so we might in this room think this"
  },
  {
    "startMs": 549600,
    "text": "is some sort of solved problem."
  },
  {
    "startMs": 551480,
    "text": "But if you look to the other engineers"
  },
  {
    "startMs": 553800,
    "text": "in the world who are perhaps not as AI"
  },
  {
    "startMs": 555680,
    "text": "pilled they're theoretically very smart"
  },
  {
    "startMs": 558680,
    "text": "and should be able to figure out how to"
  },
  {
    "startMs": 560200,
    "text": "calculate this pretty well, but then you"
  },
  {
    "startMs": 561600,
    "text": "get these scenarios where people are"
  },
  {
    "startMs": 563520,
    "text": "blowing through their entire"
  },
  {
    "startMs": 565320,
    "text": "token budget for a year and they're"
  },
  {
    "startMs": 567320,
    "text": "blowing through it in in a quarter or"
  },
  {
    "startMs": 569480,
    "text": "they're like dealing with token"
  },
  {
    "startMs": 570920,
    "text": "leaderboards and such and so obviously"
  },
  {
    "startMs": 573080,
    "text": "the incentives haven't quite aligned and"
  },
  {
    "startMs": 574839,
    "text": "we haven't perhaps got the right measure"
  },
  {
    "startMs": 576280,
    "text": "of value in terms of the technology"
  },
  {
    "startMs": 578200,
    "text": "sector itself. And so how then do we end"
  },
  {
    "startMs": 580600,
    "text": "up scaling past past that and talk to"
  },
  {
    "startMs": 582960,
    "text": "people who have no idea what we're"
  },
  {
    "startMs": 584440,
    "text": "talking about, but still try to provide"
  },
  {
    "startMs": 586360,
    "text": "them um a measure of the like increase"
  },
  {
    "startMs": 589280,
    "text": "efficiency with agents."
  },
  {
    "startMs": 591560,
    "text": "And so right now I think we're kind of"
  },
  {
    "startMs": 592720,
    "text": "in this doom loop where we're"
  },
  {
    "startMs": 594720,
    "text": "we're overspending and we're underusing."
  },
  {
    "startMs": 596920,
    "text": "Uh this is a term I borrowed from Ramp"
  },
  {
    "startMs": 598520,
    "text": "um and they have a great blog post on"
  },
  {
    "startMs": 599720,
    "text": "this. And so it's kind of this vicious"
  },
  {
    "startMs": 601600,
    "text": "cycle where we're just token maxing and"
  },
  {
    "startMs": 604000,
    "text": "ourselves into austerity"
  },
  {
    "startMs": 606080,
    "text": "kind of dropping out of the loop until"
  },
  {
    "startMs": 607920,
    "text": "we get more FOMO to to get activated"
  },
  {
    "startMs": 610400,
    "text": "enough to try it again."
  },
  {
    "startMs": 612160,
    "text": "And so I think we have to break this"
  },
  {
    "startMs": 613320,
    "text": "loop and I think the way we do that is"
  },
  {
    "startMs": 614800,
    "text": "by getting better measures of that will"
  },
  {
    "startMs": 616720,
    "text": "communicate value."
  },
  {
    "startMs": 619120,
    "text": "Some people are obviously on the right"
  },
  {
    "startMs": 620800,
    "text": "track. There was this chart floating"
  },
  {
    "startMs": 622200,
    "text": "around on X recently that the Coinbase"
  },
  {
    "startMs": 624760,
    "text": "Coinbase CEO posted where they hadn't"
  },
  {
    "startMs": 627160,
    "text": "really started changing the defaults of"
  },
  {
    "startMs": 629400,
    "text": "what models they will start with and"
  },
  {
    "startMs": 631560,
    "text": "trying to only save the frontier models"
  },
  {
    "startMs": 633040,
    "text": "for the hardest tasks and as a result"
  },
  {
    "startMs": 635400,
    "text": "saw some good"
  },
  {
    "startMs": 636920,
    "text": "saw AI spend start to diverge from token"
  },
  {
    "startMs": 639120,
    "text": "usage."
  },
  {
    "startMs": 640240,
    "text": "And uh this is a good start. Uh Ramp"
  },
  {
    "startMs": 642000,
    "text": "also, as I mentioned, has a great blog"
  },
  {
    "startMs": 643400,
    "text": "post about this. Uh but I think the"
  },
  {
    "startMs": 644760,
    "text": "problem is still that it's too focused"
  },
  {
    "startMs": 647280,
    "text": "on tokens."
  },
  {
    "startMs": 648760,
    "text": "And tokens are of course uh useful as a"
  },
  {
    "startMs": 652280,
    "text": "measurement of an internal system, but"
  },
  {
    "startMs": 654720,
    "text": "at at the end of the day they're just an"
  },
  {
    "startMs": 656280,
    "text": "output. And so the tokens need to then"
  },
  {
    "startMs": 659040,
    "text": "be traced very cleanly to an outcome."
  },
  {
    "startMs": 661360,
    "text": "So how many"
  },
  {
    "startMs": 662680,
    "text": "uh bug how many bugs did the tokens uh"
  },
  {
    "startMs": 665800,
    "text": "how sorry how many um"
  },
  {
    "startMs": 667480,
    "text": "bugs squashed did the tokens that we"
  },
  {
    "startMs": 669040,
    "text": "bought um sorry, totally butchered that."
  },
  {
    "startMs": 672040,
    "text": "Um how many uh bugs got squashed with"
  },
  {
    "startMs": 674080,
    "text": "the to with our token spend? How many uh"
  },
  {
    "startMs": 676480,
    "text": "support requests got closed, etc. So"
  },
  {
    "startMs": 678320,
    "text": "clean outcomes and then cleanly tying"
  },
  {
    "startMs": 680720,
    "text": "those to"
  },
  {
    "startMs": 682160,
    "text": "to progress on our objectives. And so"
  },
  {
    "startMs": 684160,
    "text": "without uh a very tight measure of ROI,"
  },
  {
    "startMs": 686400,
    "text": "this becomes very hard to do."
  },
  {
    "startMs": 689320,
    "text": "And I think I'll take this further and"
  },
  {
    "startMs": 691160,
    "text": "um say that it need not even be the the"
  },
  {
    "startMs": 693560,
    "text": "broader um, technology industry where"
  },
  {
    "startMs": 695720,
    "text": "it's encountering this problem, but also"
  },
  {
    "startMs": 698280,
    "text": "many of us in this room perhaps are."
  },
  {
    "startMs": 700400,
    "text": "And uh, although we're all probably"
  },
  {
    "startMs": 702200,
    "text": "enjoying uh, coding with uh, with"
  },
  {
    "startMs": 704320,
    "text": "various agents and feeling like it it's"
  },
  {
    "startMs": 706600,
    "text": "it does feel like there's something"
  },
  {
    "startMs": 707960,
    "text": "there in terms of the increase in"
  },
  {
    "startMs": 709640,
    "text": "ability and efficiency."
  },
  {
    "startMs": 711360,
    "text": "Um, the problem of course is that we're"
  },
  {
    "startMs": 712720,
    "text": "all kind of dying by a thousand pull"
  },
  {
    "startMs": 714040,
    "text": "requests. And so, uh, even Anthropic who"
  },
  {
    "startMs": 717760,
    "text": "has uh, some people on the team have"
  },
  {
    "startMs": 719440,
    "text": "claimed have solved coding, uh, they"
  },
  {
    "startMs": 721800,
    "text": "have also admitted that they've not"
  },
  {
    "startMs": 722840,
    "text": "solved code review. And so, as a result,"
  },
  {
    "startMs": 725840,
    "text": "um, the the bottleneck is now shifted to"
  },
  {
    "startMs": 728360,
    "text": "the human review where the efficiency"
  },
  {
    "startMs": 730120,
    "text": "gains from coding agents aren't quite"
  },
  {
    "startMs": 731760,
    "text": "are aren't seen yet because we spend"
  },
  {
    "startMs": 733640,
    "text": "most of the time reviewing the code and"
  },
  {
    "startMs": 735040,
    "text": "we've not figured out how to scale that"
  },
  {
    "startMs": 736840,
    "text": "in tandem with uh, the generation of the"
  },
  {
    "startMs": 738880,
    "text": "code itself."
  },
  {
    "startMs": 740200,
    "text": "And so, the bottleneck ends up shifting"
  },
  {
    "startMs": 742760,
    "text": "to the verification side and thus we"
  },
  {
    "startMs": 744880,
    "text": "don't have a way to uh, measure value at"
  },
  {
    "startMs": 747040,
    "text": "scale and uh, to judge quality at at the"
  },
  {
    "startMs": 749960,
    "text": "same speed. And so, again going back to"
  },
  {
    "startMs": 751920,
    "text": "the ROI calculations, we generate all"
  },
  {
    "startMs": 753960,
    "text": "this code, but how do we know uh, we"
  },
  {
    "startMs": 756040,
    "text": "don't know that enough of it is good to"
  },
  {
    "startMs": 757440,
    "text": "justify the spend."
  },
  {
    "startMs": 760280,
    "text": "And of course, maybe uh, code review was"
  },
  {
    "startMs": 762480,
    "text": "always flawed, uh, but it's just that"
  },
  {
    "startMs": 764160,
    "text": "agents are now exposing it for"
  },
  {
    "startMs": 766320,
    "text": "uh, the are exposing the actual problem."
  },
  {
    "startMs": 769240,
    "text": "Um, but I like this quote by Noah Hein"
  },
  {
    "startMs": 771240,
    "text": "who from a a post about how to solve"
  },
  {
    "startMs": 773120,
    "text": "code review where he's mentioning"
  },
  {
    "startMs": 774760,
    "text": "specifically that the assumptions"
  },
  {
    "startMs": 776200,
    "text": "underneath code review are what's now"
  },
  {
    "startMs": 777880,
    "text": "being uh, what needs to be revisited."
  },
  {
    "startMs": 779760,
    "text": "So, you have to uh, check our priors to"
  },
  {
    "startMs": 782320,
    "text": "try to figure out a new basis for um,"
  },
  {
    "startMs": 784640,
    "text": "how we can code review in the age of"
  },
  {
    "startMs": 786080,
    "text": "agents."
  },
  {
    "startMs": 787160,
    "text": "And I'm not going to go into how to"
  },
  {
    "startMs": 789040,
    "text": "solve code review. I think that's"
  },
  {
    "startMs": 789960,
    "text": "definitely better uh, a talk that's"
  },
  {
    "startMs": 791920,
    "text": "better given by somebody else and um, is"
  },
  {
    "startMs": 794360,
    "text": "totally different subject, but uh, what"
  },
  {
    "startMs": 796120,
    "text": "I think is important for this talk is"
  },
  {
    "startMs": 797720,
    "text": "why does code review feel like it is"
  },
  {
    "startMs": 799200,
    "text": "solvable? And I think that Noah is"
  },
  {
    "startMs": 800880,
    "text": "hitting on something important here"
  },
  {
    "startMs": 802080,
    "text": "which is that as a as a culture uh, code"
  },
  {
    "startMs": 804640,
    "text": "review has a a good uh,"
  },
  {
    "startMs": 806880,
    "text": "convergence on shared assumptions and"
  },
  {
    "startMs": 808800,
    "text": "that lets you"
  },
  {
    "startMs": 810120,
    "text": "um"
  },
  {
    "startMs": 811400,
    "text": "that lets you measure things at scale"
  },
  {
    "startMs": 813600,
    "text": "when we can all kind of converge on the"
  },
  {
    "startMs": 814800,
    "text": "measurement and it becomes somewhat of"
  },
  {
    "startMs": 818280,
    "text": "clear rubric and so the task at hand now"
  },
  {
    "startMs": 821280,
    "text": "is we have to adopt we have to adapt"
  },
  {
    "startMs": 824720,
    "text": "those assumptions for the agentic age."
  },
  {
    "startMs": 829320,
    "text": "And so"
  },
  {
    "startMs": 830600,
    "text": "we can we need to go if we're able to do"
  },
  {
    "startMs": 832680,
    "text": "that then we can go from execution at"
  },
  {
    "startMs": 834280,
    "text": "the speed of computer to measurement at"
  },
  {
    "startMs": 836280,
    "text": "the speed of computer and of course the"
  },
  {
    "startMs": 838080,
    "text": "measurements need to fit the mental"
  },
  {
    "startMs": 839840,
    "text": "models of the customers using it."
  },
  {
    "startMs": 841960,
    "text": "And"
  },
  {
    "startMs": 843240,
    "text": "I think the lesson here being that if"
  },
  {
    "startMs": 845080,
    "text": "you're going to"
  },
  {
    "startMs": 846760,
    "text": "think of how to build an agent for"
  },
  {
    "startMs": 848600,
    "text": "something you also have to think about"
  },
  {
    "startMs": 850000,
    "text": "how do you help the customers build"
  },
  {
    "startMs": 852480,
    "text": "build or at least create a method for"
  },
  {
    "startMs": 854840,
    "text": "verifying that the output is good and so"
  },
  {
    "startMs": 857160,
    "text": "it's not enough to build it we also have"
  },
  {
    "startMs": 858560,
    "text": "to help them"
  },
  {
    "startMs": 860280,
    "text": "we also we also also have to help them"
  },
  {
    "startMs": 861680,
    "text": "get to clear ROI calculations to justify"
  },
  {
    "startMs": 864520,
    "text": "their spend."
  },
  {
    "startMs": 866000,
    "text": "And so this brings me to the idea of"
  },
  {
    "startMs": 869320,
    "text": "mouse power which could be the"
  },
  {
    "startMs": 871200,
    "text": "equivalent of horsepower for the agentic"
  },
  {
    "startMs": 873120,
    "text": "age just as James Watt was able to show"
  },
  {
    "startMs": 875880,
    "text": "a measure of efficiency relative to the"
  },
  {
    "startMs": 877480,
    "text": "horses in in the gins in the horse gins"
  },
  {
    "startMs": 880600,
    "text": "that were the source of power at the"
  },
  {
    "startMs": 881440,
    "text": "time we perhaps can also figure out how"
  },
  {
    "startMs": 884480,
    "text": "do we create a baseline of efficiency"
  },
  {
    "startMs": 886640,
    "text": "for the way we use computers today and"
  },
  {
    "startMs": 889200,
    "text": "can then demonstrate how much better or"
  },
  {
    "startMs": 891600,
    "text": "perhaps more performant on certain"
  },
  {
    "startMs": 893080,
    "text": "vectors an agent could be at that task."
  },
  {
    "startMs": 896040,
    "text": "Um and"
  },
  {
    "startMs": 897640,
    "text": "of course it's not as easy perhaps as"
  },
  {
    "startMs": 899839,
    "text": "easy a task as he had back then where he"
  },
  {
    "startMs": 901200,
    "text": "could just study the horse gin cuz it's"
  },
  {
    "startMs": 903120,
    "text": "not as if we can create some method to"
  },
  {
    "startMs": 905600,
    "text": "measure cursor movements and like figure"
  },
  {
    "startMs": 907760,
    "text": "out the delta of how much more efficient"
  },
  {
    "startMs": 909800,
    "text": "an agent could move them and thus we can"
  },
  {
    "startMs": 911960,
    "text": "say yeah agents are this much more"
  },
  {
    "startMs": 913240,
    "text": "performant than humans at these tasks."
  },
  {
    "startMs": 915480,
    "text": "Trust me I've I've tried I had Claude"
  },
  {
    "startMs": 918080,
    "text": "vibe code me this measurement device and"
  },
  {
    "startMs": 920480,
    "text": "I thought maybe if I can figure out the"
  },
  {
    "startMs": 922400,
    "text": "movement like the potential movement"
  },
  {
    "startMs": 924320,
    "text": "across the screen and measure how fast"
  },
  {
    "startMs": 926280,
    "text": "it went, I could get some clean measure"
  },
  {
    "startMs": 928280,
    "text": "of mouse power. Uh but of course this is"
  },
  {
    "startMs": 930760,
    "text": "only joking. Um this is of course um"
  },
  {
    "startMs": 933520,
    "text": "like a fool's errand because information"
  },
  {
    "startMs": 935480,
    "text": "space is just way too high dimensional"
  },
  {
    "startMs": 937640,
    "text": "and so I think mouse power is is never"
  },
  {
    "startMs": 940320,
    "text": "going to be a metric of course, but it's"
  },
  {
    "startMs": 941800,
    "text": "more so an idea. Which the idea being if"
  },
  {
    "startMs": 944520,
    "text": "you're going to sell somebody an agent,"
  },
  {
    "startMs": 945760,
    "text": "you also have to help them with the with"
  },
  {
    "startMs": 947880,
    "text": "the rubric of how do we actually verify"
  },
  {
    "startMs": 950120,
    "text": "that this agent is doing good work and"
  },
  {
    "startMs": 952040,
    "text": "thus we can uh have a good measure of"
  },
  {
    "startMs": 954160,
    "text": "saying that these tokens are worth it."
  },
  {
    "startMs": 956400,
    "text": "Um so how to do that of course is is"
  },
  {
    "startMs": 958400,
    "text": "really up to you and I won't be able to"
  },
  {
    "startMs": 960320,
    "text": "tell you how do you I don't have any"
  },
  {
    "startMs": 962280,
    "text": "good frameworks for how do you figure"
  },
  {
    "startMs": 964160,
    "text": "out the right measurements to to help"
  },
  {
    "startMs": 965720,
    "text": "provide anybody you're building an agent"
  },
  {
    "startMs": 967600,
    "text": "for. Uh but what I can do is give a"
  },
  {
    "startMs": 969800,
    "text": "principle uh give an idea that I've been"
  },
  {
    "startMs": 971520,
    "text": "kicking around which is based um"
  },
  {
    "startMs": 973840,
    "text": "in information theory. So going back to"
  },
  {
    "startMs": 976880,
    "text": "Claude Shannon's ideas about measuring"
  },
  {
    "startMs": 978880,
    "text": "entropy and information."
  },
  {
    "startMs": 980720,
    "text": "Uh entropy being uh the uncertainty of a"
  },
  {
    "startMs": 982920,
    "text": "probability distribution and of course"
  },
  {
    "startMs": 985280,
    "text": "very much the basis of how we train"
  },
  {
    "startMs": 987240,
    "text": "agents today."
  },
  {
    "startMs": 988600,
    "text": "Things like cross entropy and such uh"
  },
  {
    "startMs": 990160,
    "text": "being a big factor in determining how"
  },
  {
    "startMs": 991839,
    "text": "capable an agent is."
  },
  {
    "startMs": 993480,
    "text": "Um I think that entropy's an interesting"
  },
  {
    "startMs": 995040,
    "text": "idea to think through with regards to"
  },
  {
    "startMs": 996800,
    "text": "not just the performance of an agent,"
  },
  {
    "startMs": 999000,
    "text": "but also the task that we're setting"
  },
  {
    "startMs": 1000680,
    "text": "them out to to perform on."
  },
  {
    "startMs": 1002960,
    "text": "And so uh I put together this matrix"
  },
  {
    "startMs": 1005320,
    "text": "which uh it maps on the x-axis axis the"
  },
  {
    "startMs": 1009200,
    "text": "uncertainty in the steps it takes to"
  },
  {
    "startMs": 1010960,
    "text": "perform a task. And so when we're"
  },
  {
    "startMs": 1012960,
    "text": "thinking of building an agent, I think"
  },
  {
    "startMs": 1014480,
    "text": "it's not enough to just think what would"
  },
  {
    "startMs": 1016080,
    "text": "be a valuable task for the agent to do,"
  },
  {
    "startMs": 1018080,
    "text": "but also thinking about how um how much"
  },
  {
    "startMs": 1021040,
    "text": "uncertainty are in the steps to perform"
  },
  {
    "startMs": 1022800,
    "text": "that task itself. So an example would be"
  },
  {
    "startMs": 1025760,
    "text": "uh booking a flight has uh much less"
  },
  {
    "startMs": 1028000,
    "text": "uncertainty than let's say painting a"
  },
  {
    "startMs": 1029520,
    "text": "masterpiece, right? Because you know"
  },
  {
    "startMs": 1031560,
    "text": "there's certain information that has to"
  },
  {
    "startMs": 1033120,
    "text": "be that has to happen in the flight"
  },
  {
    "startMs": 1034560,
    "text": "purchase. There has to be a departing"
  },
  {
    "startMs": 1036600,
    "text": "destination, arriving destination."
  },
  {
    "startMs": 1038560,
    "text": "There's going to be a seat chosen. It"
  },
  {
    "startMs": 1039880,
    "text": "might be by the person. It might just be"
  },
  {
    "startMs": 1041400,
    "text": "random."
  },
  {
    "startMs": 1042520,
    "text": "But these things have to happen for that"
  },
  {
    "startMs": 1044199,
    "text": "task to be completed. And on the other"
  },
  {
    "startMs": 1045959,
    "text": "hand, there is the task of like painting"
  },
  {
    "startMs": 1048520,
    "text": "a masterpiece, right? And who knows what"
  },
  {
    "startMs": 1050480,
    "text": "the steps are to that? Maybe you can get"
  },
  {
    "startMs": 1052600,
    "text": "an agent to do it, but it would be very"
  },
  {
    "startMs": 1054000,
    "text": "hard to figure out how we can actually"
  },
  {
    "startMs": 1056520,
    "text": "create a a relatively predictable"
  },
  {
    "startMs": 1059120,
    "text": "pathway to that."
  },
  {
    "startMs": 1060600,
    "text": "But then on the other axis is the the"
  },
  {
    "startMs": 1063520,
    "text": "uncertainty in the acceptance criteria"
  },
  {
    "startMs": 1065120,
    "text": "itself. So not just can the agent"
  },
  {
    "startMs": 1067320,
    "text": "perform the task, but can we help"
  },
  {
    "startMs": 1069320,
    "text": "somebody actually or is is there"
  },
  {
    "startMs": 1071320,
    "text": "actually a a clean rubric for how it's"
  },
  {
    "startMs": 1073560,
    "text": "graded? And so thinking about ideas on"
  },
  {
    "startMs": 1076200,
    "text": "on these two axes and where they"
  },
  {
    "startMs": 1077600,
    "text": "intersect, perhaps gives us a better"
  },
  {
    "startMs": 1079720,
    "text": "guide for how to build agents and we can"
  },
  {
    "startMs": 1081960,
    "text": "run through a few examples. So if we"
  },
  {
    "startMs": 1085080,
    "text": "look at the at the left side, your right"
  },
  {
    "startMs": 1087520,
    "text": "side."
  },
  {
    "startMs": 1089120,
    "text": "Yes."
  },
  {
    "startMs": 1090080,
    "text": "No, you're left as well."
  },
  {
    "startMs": 1092040,
    "text": "Then"
  },
  {
    "startMs": 1093320,
    "text": "No, last speaker was also a confused"
  },
  {
    "startMs": 1095320,
    "text": "about."
  },
  {
    "startMs": 1096720,
    "text": "So uh"
  },
  {
    "startMs": 1097960,
    "text": "Yeah, on the left side when uncertainty"
  },
  {
    "startMs": 1100480,
    "text": "in the task steps are low, then it's a"
  },
  {
    "startMs": 1102880,
    "text": "very it's a very predictable outcome or"
  },
  {
    "startMs": 1105120,
    "text": "it's a very predictable pathway to"
  },
  {
    "startMs": 1106640,
    "text": "achieve that goal. And so then, you"
  },
  {
    "startMs": 1108840,
    "text": "know, why would you waste tokens? Just"
  },
  {
    "startMs": 1110560,
    "text": "write a script. On the other side, when"
  },
  {
    "startMs": 1113800,
    "text": "the"
  },
  {
    "startMs": 1114640,
    "text": "the steps to do perform the task are"
  },
  {
    "startMs": 1116080,
    "text": "very high in in uncertainty, then you"
  },
  {
    "startMs": 1119080,
    "text": "you have very unpredictable information."
  },
  {
    "startMs": 1120960,
    "text": "And so it's probably at risk of being"
  },
  {
    "startMs": 1123000,
    "text": "out of distribution in pre-training and"
  },
  {
    "startMs": 1125120,
    "text": "probably has very sparse rewards for"
  },
  {
    "startMs": 1126600,
    "text": "reinforcement learning. And so perhaps"
  },
  {
    "startMs": 1128680,
    "text": "it's not a a good task for an agent"
  },
  {
    "startMs": 1130120,
    "text": "because it's just much harder to figure"
  },
  {
    "startMs": 1132080,
    "text": "out how to actually model that data."
  },
  {
    "startMs": 1134520,
    "text": "And so obviously in the middle is is um"
  },
  {
    "startMs": 1138600,
    "text": "is so I'm I think I'm out of time, but"
  },
  {
    "startMs": 1140400,
    "text": "I'm not getting kicked off yet."
  },
  {
    "startMs": 1142920,
    "text": "So I'll just finish this up quickly. Um"
  },
  {
    "startMs": 1144280,
    "text": "so yeah, in the middle is is probably"
  },
  {
    "startMs": 1145560,
    "text": "the sweet spot, but then on the other"
  },
  {
    "startMs": 1146800,
    "text": "axis,"
  },
  {
    "startMs": 1147920,
    "text": "what's the uncertainty in verifying that"
  },
  {
    "startMs": 1149360,
    "text": "this is actually valuable? So when you"
  },
  {
    "startMs": 1151360,
    "text": "have high uncertainty in the acceptance"
  },
  {
    "startMs": 1153000,
    "text": "criteria, you pretty much are in a spot"
  },
  {
    "startMs": 1154880,
    "text": "where verification is indistinguishable"
  },
  {
    "startMs": 1156720,
    "text": "from execution. So, why would you build"
  },
  {
    "startMs": 1158840,
    "text": "an agent for something that to verify"
  },
  {
    "startMs": 1161000,
    "text": "was useful, a person pretty much has to"
  },
  {
    "startMs": 1162920,
    "text": "do the work again. So, like waste of"
  },
  {
    "startMs": 1165480,
    "text": "tokens obviously."
  },
  {
    "startMs": 1167080,
    "text": "And then it leaves that that middle area"
  },
  {
    "startMs": 1169120,
    "text": "where you have this interesting"
  },
  {
    "startMs": 1170200,
    "text": "intersection of tasks that are um"
  },
  {
    "startMs": 1173720,
    "text": "they're not too uncertain in that"
  },
  {
    "startMs": 1176600,
    "text": "they or they they have a degree of"
  },
  {
    "startMs": 1178960,
    "text": "uncertainty where they're not great"
  },
  {
    "startMs": 1181000,
    "text": "they're not just a a script or they're"
  },
  {
    "startMs": 1183360,
    "text": "not out of distribution for training,"
  },
  {
    "startMs": 1185440,
    "text": "but they have enough uncertainty to be"
  },
  {
    "startMs": 1186840,
    "text": "interesting, but at the at the same time"
  },
  {
    "startMs": 1189280,
    "text": "they also have a property of being"
  },
  {
    "startMs": 1190920,
    "text": "relatively easy to"
  },
  {
    "startMs": 1192760,
    "text": "validate the"
  },
  {
    "startMs": 1194520,
    "text": "to check the value of them. And so they"
  },
  {
    "startMs": 1196760,
    "text": "become in this place where they kind of"
  },
  {
    "startMs": 1198600,
    "text": "become the shape of an NP-style problem,"
  },
  {
    "startMs": 1200360,
    "text": "which means they're easier to verify"
  },
  {
    "startMs": 1201640,
    "text": "than to execute. And the reason I say"
  },
  {
    "startMs": 1203600,
    "text": "that is because if you can figure out a"
  },
  {
    "startMs": 1205040,
    "text": "pretty repeatable pattern for verifying"
  },
  {
    "startMs": 1207720,
    "text": "their their work, you can actually just"
  },
  {
    "startMs": 1209520,
    "text": "throw agents to that problem as well."
  },
  {
    "startMs": 1211880,
    "text": "And so of course you don't just build"
  },
  {
    "startMs": 1213840,
    "text": "the agent, you perhaps build the agent"
  },
  {
    "startMs": 1215640,
    "text": "that verifies the work of the agent."
  },
  {
    "startMs": 1217880,
    "text": "Um and so yeah, this is perhaps this is"
  },
  {
    "startMs": 1221120,
    "text": "a thought starter mostly kind of have"
  },
  {
    "startMs": 1223120,
    "text": "kind of still in the works, so"
  },
  {
    "startMs": 1225560,
    "text": "I'm happy to hear any thoughts on it,"
  },
  {
    "startMs": 1226840,
    "text": "but if with this guidance I hope when"
  },
  {
    "startMs": 1229200,
    "text": "you're building your next agent you can"
  },
  {
    "startMs": 1230520,
    "text": "also figure out how to also build its"
  },
  {
    "startMs": 1232720,
    "text": "mouse power."
  },
  {
    "startMs": 1234080,
    "text": "And thanks very much."
  }
]