[
  {
    "startMs": 19760,
    "text": "Good morning. We're super excited to be"
  },
  {
    "startMs": 22080,
    "text": "here at AI Engineer with all of you. I'm"
  },
  {
    "startMs": 25039,
    "text": "Caitlyn and I lead platform engineering"
  },
  {
    "startMs": 26960,
    "text": "at Anthropic."
  },
  {
    "startMs": 28080,
    "text": ">> And I'm Angela. I lead platform product"
  },
  {
    "startMs": 29760,
    "text": "at Anthropic. And today we want to talk"
  },
  {
    "startMs": 31840,
    "text": "to you about a concept that we've been"
  },
  {
    "startMs": 33520,
    "text": "spending a lot of time thinking about"
  },
  {
    "startMs": 35280,
    "text": "and working on with our team, which is"
  },
  {
    "startMs": 37280,
    "text": "this idea that we think that tokens"
  },
  {
    "startMs": 39120,
    "text": "should have jobs."
  },
  {
    "startMs": 41760,
    "text": "So if you're building an agentic system"
  },
  {
    "startMs": 43600,
    "text": "and you're trying to accomplish some"
  },
  {
    "startMs": 45280,
    "text": "specific outcome, you're trying to get"
  },
  {
    "startMs": 46800,
    "text": "something done with agents, there's one"
  },
  {
    "startMs": 49039,
    "text": "lever that everybody pulls in order to"
  },
  {
    "startMs": 50879,
    "text": "get a better outcome, and that's usually"
  },
  {
    "startMs": 53039,
    "text": "increasing your budget, which means you"
  },
  {
    "startMs": 54879,
    "text": "spend more tokens or you spend more"
  },
  {
    "startMs": 56719,
    "text": "expensive tokens."
  },
  {
    "startMs": 59760,
    "text": "But we've been wondering is that all"
  },
  {
    "startMs": 61600,
    "text": "there is underlying this assumption of"
  },
  {
    "startMs": 65119,
    "text": "uh using the budget is this kind of"
  },
  {
    "startMs": 66880,
    "text": "implicit perspective that every single"
  },
  {
    "startMs": 69119,
    "text": "token is basically fungeable. And we've"
  },
  {
    "startMs": 71600,
    "text": "been wondering is that actually true?"
  },
  {
    "startMs": 73520,
    "text": "Are all these tokens actually fungeible?"
  },
  {
    "startMs": 75840,
    "text": "And to test that, we've been thinking,"
  },
  {
    "startMs": 77119,
    "text": "what if we gave tokens jobs?"
  },
  {
    "startMs": 80640,
    "text": "So, if you think about the way that you"
  },
  {
    "startMs": 81840,
    "text": "would normally set up an agent to go"
  },
  {
    "startMs": 83040,
    "text": "accomplish a task, you give it that"
  },
  {
    "startMs": 84560,
    "text": "task, you give it this token budget, and"
  },
  {
    "startMs": 86640,
    "text": "then all the tokens that are being spent"
  },
  {
    "startMs": 88400,
    "text": "are basically indiscriminate in the"
  },
  {
    "startMs": 90080,
    "text": "sense that they're all doing one job."
  },
  {
    "startMs": 91680,
    "text": "They're just executing."
  },
  {
    "startMs": 94240,
    "text": "But what if you take some of those"
  },
  {
    "startMs": 95520,
    "text": "tokens and they're not just executing,"
  },
  {
    "startMs": 97520,
    "text": "they're doing some other job. So, for"
  },
  {
    "startMs": 100560,
    "text": "example, maybe you take some of your"
  },
  {
    "startMs": 102240,
    "text": "tokens and they're advising the tokens"
  },
  {
    "startMs": 104560,
    "text": "that are executing. Or maybe the tokens"
  },
  {
    "startMs": 107200,
    "text": "that are executing try to get something"
  },
  {
    "startMs": 109040,
    "text": "done well and you take some other tokens"
  },
  {
    "startMs": 111119,
    "text": "and you actually grade how well the"
  },
  {
    "startMs": 112960,
    "text": "executor is doing so that it can iterate"
  },
  {
    "startMs": 114960,
    "text": "and try again. Or maybe you have tokens"
  },
  {
    "startMs": 118240,
    "text": "that are dreaming. They're reflecting"
  },
  {
    "startMs": 119759,
    "text": "back on the job that other executors"
  },
  {
    "startMs": 122640,
    "text": "have done and writing learnings to"
  },
  {
    "startMs": 124479,
    "text": "memory so that they can do it again. And"
  },
  {
    "startMs": 127119,
    "text": "what we call each of these if you take"
  },
  {
    "startMs": 129039,
    "text": "some tokens that are executing and some"
  },
  {
    "startMs": 130879,
    "text": "tokens that are doing some other job."
  },
  {
    "startMs": 132959,
    "text": "Let's call this a strategy."
  },
  {
    "startMs": 136160,
    "text": "So let's go take a look at the first"
  },
  {
    "startMs": 137520,
    "text": "strategy, the advising strategy. Here"
  },
  {
    "startMs": 139840,
    "text": "we're splitting up an executor and an"
  },
  {
    "startMs": 141840,
    "text": "adviser. The executor obviously"
  },
  {
    "startMs": 143599,
    "text": "executes, but crucially they can call"
  },
  {
    "startMs": 145120,
    "text": "out to an adviser for advice. And then"
  },
  {
    "startMs": 147200,
    "text": "they can take this advice and figure out"
  },
  {
    "startMs": 148480,
    "text": "if they're doing the next step"
  },
  {
    "startMs": 149440,
    "text": "correctly."
  },
  {
    "startMs": 151760,
    "text": "This is really helpful in use cases. For"
  },
  {
    "startMs": 153280,
    "text": "example, if you're building a sales"
  },
  {
    "startMs": 154480,
    "text": "agent, in an ideal world, you'd have"
  },
  {
    "startMs": 156319,
    "text": "that sales agent be able to actually"
  },
  {
    "startMs": 157840,
    "text": "help the sales rep flag when a follow-up"
  },
  {
    "startMs": 159760,
    "text": "is overdue or deal is stalling. In this"
  },
  {
    "startMs": 162239,
    "text": "construct, having an adviser to be able"
  },
  {
    "startMs": 164000,
    "text": "to kind of make sure that all the"
  },
  {
    "startMs": 165599,
    "text": "different pieces are actually working is"
  },
  {
    "startMs": 167200,
    "text": "really helpful."
  },
  {
    "startMs": 169360,
    "text": "So another example is grading. Let's say"
  },
  {
    "startMs": 171760,
    "text": "you're executing and you kind of know"
  },
  {
    "startMs": 174319,
    "text": "exactly what good really does look like."
  },
  {
    "startMs": 176480,
    "text": "You can define this in a rubric and then"
  },
  {
    "startMs": 178800,
    "text": "each time an executor tries to"
  },
  {
    "startMs": 181040,
    "text": "accomplish that outcome, you can have a"
  },
  {
    "startMs": 183200,
    "text": "grader provisioned that grades how well"
  },
  {
    "startMs": 185519,
    "text": "the executor did while looking at that"
  },
  {
    "startMs": 187680,
    "text": "rubric. And if the executor did a good"
  },
  {
    "startMs": 190000,
    "text": "job, then great, it can be done. But if"
  },
  {
    "startMs": 192000,
    "text": "it didn't do such a great job, you can"
  },
  {
    "startMs": 193680,
    "text": "iterate again until you get that good"
  },
  {
    "startMs": 195680,
    "text": "outcome."
  },
  {
    "startMs": 197360,
    "text": "So an example in practice of when you"
  },
  {
    "startMs": 199120,
    "text": "might want to use this is let's say you"
  },
  {
    "startMs": 200640,
    "text": "have a customer service agent and you're"
  },
  {
    "startMs": 202879,
    "text": "running a store and your customers are"
  },
  {
    "startMs": 204560,
    "text": "writing in and they're saying, \"H, I"
  },
  {
    "startMs": 206000,
    "text": "should get a refund for this thing.\" And"
  },
  {
    "startMs": 207680,
    "text": "your customer service agent needs to be"
  },
  {
    "startMs": 209280,
    "text": "able to respond. You probably have some"
  },
  {
    "startMs": 211519,
    "text": "like pretty specific criteria on when"
  },
  {
    "startMs": 213599,
    "text": "you would give somebody a refund. And so"
  },
  {
    "startMs": 216640,
    "text": "what you can do is define a rubric that"
  },
  {
    "startMs": 218560,
    "text": "uses that criteria. You can have a"
  },
  {
    "startMs": 220480,
    "text": "grader that goes and looks at the work"
  },
  {
    "startMs": 222239,
    "text": "that the customer service agent is doing"
  },
  {
    "startMs": 224560,
    "text": "and decide is it getting it right and is"
  },
  {
    "startMs": 226640,
    "text": "it coming to the right outcome."
  },
  {
    "startMs": 229040,
    "text": "And the last strategy we have is"
  },
  {
    "startMs": 230720,
    "text": "dreaming. So in dreaming there's an"
  },
  {
    "startMs": 232640,
    "text": "executor who naturally executes and then"
  },
  {
    "startMs": 234879,
    "text": "there's a dreamer. The dreamer is"
  },
  {
    "startMs": 236319,
    "text": "actually able to inspect the work and"
  },
  {
    "startMs": 238319,
    "text": "the transcripts of the executor and then"
  },
  {
    "startMs": 240560,
    "text": "it takes any of the findings that it has"
  },
  {
    "startMs": 242159,
    "text": "and it writes them to memory. This"
  },
  {
    "startMs": 243840,
    "text": "memory is repicked up by the executor"
  },
  {
    "startMs": 245680,
    "text": "for the next round. So ideally would"
  },
  {
    "startMs": 247360,
    "text": "have improved."
  },
  {
    "startMs": 249280,
    "text": "A great use case for this is if you're"
  },
  {
    "startMs": 250720,
    "text": "building a recruiting agent. Now"
  },
  {
    "startMs": 252319,
    "text": "recruiting requires a lot of interaction"
  },
  {
    "startMs": 254400,
    "text": "with feedback on whether or not a"
  },
  {
    "startMs": 255760,
    "text": "candidate does or doesn't make sense and"
  },
  {
    "startMs": 257280,
    "text": "if it's a good fit between both parties."
  },
  {
    "startMs": 259040,
    "text": "And so by taking all this type of data,"
  },
  {
    "startMs": 260720,
    "text": "if you build a dreaming type of strategy"
  },
  {
    "startMs": 262560,
    "text": "on this agent, it's actually able to"
  },
  {
    "startMs": 264000,
    "text": "kind of sharpen the next round so that"
  },
  {
    "startMs": 265520,
    "text": "it's more and more increasingly useful."
  },
  {
    "startMs": 268800,
    "text": "So let's make this concrete with some"
  },
  {
    "startMs": 270880,
    "text": "experiments. So what we did was we"
  },
  {
    "startMs": 273120,
    "text": "created a bench of a bunch of tasks"
  },
  {
    "startMs": 275199,
    "text": "related to financial an analysis. And"
  },
  {
    "startMs": 277680,
    "text": "what we were doing with each of these"
  },
  {
    "startMs": 278880,
    "text": "tasks is trying to replicate in the real"
  },
  {
    "startMs": 280960,
    "text": "world a expert human financial analyst."
  },
  {
    "startMs": 283840,
    "text": "How well would they do on each of these"
  },
  {
    "startMs": 285600,
    "text": "various tasks? And so what we did was we"
  },
  {
    "startMs": 287759,
    "text": "start with a control that's just"
  },
  {
    "startMs": 289280,
    "text": "executing. Let's try each of these tasks"
  },
  {
    "startMs": 291120,
    "text": "and we'll eval them when we're literally"
  },
  {
    "startMs": 292720,
    "text": "just executing. But then we can"
  },
  {
    "startMs": 294479,
    "text": "experiment with each of our strategies"
  },
  {
    "startMs": 296080,
    "text": "and see how well we perform."
  },
  {
    "startMs": 300400,
    "text": "So, we start with a super basic"
  },
  {
    "startMs": 301840,
    "text": "experiment. Let's just oneshot it. Let's"
  },
  {
    "startMs": 303919,
    "text": "take each of our strategies and we'll go"
  },
  {
    "startMs": 305759,
    "text": "and just make an attempt to accomplish"
  },
  {
    "startMs": 307600,
    "text": "these tasks and we'll see how accurate"
  },
  {
    "startMs": 309360,
    "text": "we are. And so, you can see here with"
  },
  {
    "startMs": 311360,
    "text": "executing um it didn't do so well. 15%"
  },
  {
    "startMs": 314400,
    "text": "accuracy, but because it was just a"
  },
  {
    "startMs": 316639,
    "text": "oneshot, the strategy got to choose how"
  },
  {
    "startMs": 318880,
    "text": "many tokens it would actually spend on"
  },
  {
    "startMs": 320560,
    "text": "its own. And so, you can actually see"
  },
  {
    "startMs": 322000,
    "text": "that execute decided not to spend that"
  },
  {
    "startMs": 324080,
    "text": "many tokens, only 39,000. And as we go"
  },
  {
    "startMs": 327360,
    "text": "into our larger strategies, our more"
  },
  {
    "startMs": 329520,
    "text": "complex strategies, we did choose to"
  },
  {
    "startMs": 331280,
    "text": "spend more tokens, but we did a better"
  },
  {
    "startMs": 332800,
    "text": "job. So, this isn't really telling us"
  },
  {
    "startMs": 334479,
    "text": "much because sure, Drain did really,"
  },
  {
    "startMs": 336240,
    "text": "really well, but it used a whopping"
  },
  {
    "startMs": 338000,
    "text": "600,000 tokens to get there. That's"
  },
  {
    "startMs": 340160,
    "text": "right. So, in order to actually figure"
  },
  {
    "startMs": 342800,
    "text": "out if varying the jobs produces any"
  },
  {
    "startMs": 345039,
    "text": "alpha, what we need to do is hold the"
  },
  {
    "startMs": 346800,
    "text": "budget constant. And to do this, we're"
  },
  {
    "startMs": 348560,
    "text": "going to take Dreaming's budget, that"
  },
  {
    "startMs": 349919,
    "text": "600,000 or so, as the maximum budget"
  },
  {
    "startMs": 351919,
    "text": "that is fixed across the board. And we"
  },
  {
    "startMs": 353759,
    "text": "give every single strategy this budget"
  },
  {
    "startMs": 355520,
    "text": "in order to analyze how well it's"
  },
  {
    "startMs": 357440,
    "text": "performing. And as expected again, if"
  },
  {
    "startMs": 360080,
    "text": "you give a lot of strategies more"
  },
  {
    "startMs": 361520,
    "text": "budget, you are going to see performance"
  },
  {
    "startMs": 363199,
    "text": "increase across the board. So execute"
  },
  {
    "startMs": 364960,
    "text": "went from 0.15 to 76. Advise and grade"
  },
  {
    "startMs": 368400,
    "text": "went from the 60s to closer to the 90s."
  },
  {
    "startMs": 370960,
    "text": "And that's again expected given the fact"
  },
  {
    "startMs": 372720,
    "text": "that if you give things more test time"
  },
  {
    "startMs": 374319,
    "text": "compute, they should generally perform"
  },
  {
    "startMs": 376080,
    "text": "better. But if that was the only thing"
  },
  {
    "startMs": 378479,
    "text": "that mattered, we should actually expect"
  },
  {
    "startMs": 380000,
    "text": "to see execute, advise, grade, dream"
  },
  {
    "startMs": 382560,
    "text": "actually all be at the exact same level"
  },
  {
    "startMs": 384319,
    "text": "given the exact same token budget. But"
  },
  {
    "startMs": 386800,
    "text": "what we're actually seeing is that there"
  },
  {
    "startMs": 388319,
    "text": "is an alpha or there is a difference and"
  },
  {
    "startMs": 390560,
    "text": "therefore an alpha for us to exploit. If"
  },
  {
    "startMs": 392560,
    "text": "you look at execute at this exact same"
  },
  {
    "startMs": 394400,
    "text": "budget level, it gets to 76 but advise"
  },
  {
    "startMs": 397520,
    "text": "is at 89. So while a minimal, it does"
  },
  {
    "startMs": 400160,
    "text": "exist and so there is alpha for us to"
  },
  {
    "startMs": 401759,
    "text": "take a look at."
  },
  {
    "startMs": 404479,
    "text": "Now, we decided to take a look at this"
  },
  {
    "startMs": 405919,
    "text": "analysis from a completely different"
  },
  {
    "startMs": 407280,
    "text": "lens. And as Kayla mentioned, you know,"
  },
  {
    "startMs": 409759,
    "text": "we're doing this bench for a very"
  },
  {
    "startMs": 411919,
    "text": "complex set of financial tasks in the"
  },
  {
    "startMs": 413680,
    "text": "real world. And we wanted to analyze the"
  },
  {
    "startMs": 416720,
    "text": "usage of agents with actual experts. So,"
  },
  {
    "startMs": 420160,
    "text": "if we look at a financial analyst"
  },
  {
    "startMs": 422000,
    "text": "expert, right, the kind of task that"
  },
  {
    "startMs": 423759,
    "text": "they need to do with an agent is that"
  },
  {
    "startMs": 425919,
    "text": "they're giving it something very"
  },
  {
    "startMs": 427440,
    "text": "concrete like let's say make a P&L and"
  },
  {
    "startMs": 429759,
    "text": "then they're getting the result back."
  },
  {
    "startMs": 431360,
    "text": "Now if that result is 80% accurate on a"
  },
  {
    "startMs": 434400,
    "text": "bench that sounds great but in reality"
  },
  {
    "startMs": 436319,
    "text": "what that means for that expert is they"
  },
  {
    "startMs": 438240,
    "text": "have to go back and recomputee that P&L"
  },
  {
    "startMs": 440479,
    "text": "themselves or alter or alternatively"
  },
  {
    "startMs": 442560,
    "text": "send it through another run and that's"
  },
  {
    "startMs": 444240,
    "text": "because in this kind of domain for this"
  },
  {
    "startMs": 445840,
    "text": "kind of task if you're not 100% accurate"
  },
  {
    "startMs": 448639,
    "text": "it's actually not useful."
  },
  {
    "startMs": 451199,
    "text": "You cannot make up an income number or"
  },
  {
    "startMs": 452720,
    "text": "you can't make up a cost number right"
  },
  {
    "startMs": 454000,
    "text": "you have to make sure that it's 100%"
  },
  {
    "startMs": 455759,
    "text": "accurate. So with this lens of the real"
  },
  {
    "startMs": 457759,
    "text": "world consequence associated with this"
  },
  {
    "startMs": 459199,
    "text": "domain, we needed to recomputee our"
  },
  {
    "startMs": 460960,
    "text": "experiments and score them a bit"
  },
  {
    "startMs": 462560,
    "text": "differently. Crucially, we needed to"
  },
  {
    "startMs": 464479,
    "text": "make sure that our experiment had this"
  },
  {
    "startMs": 466240,
    "text": "kind of construct where if it was scored"
  },
  {
    "startMs": 468319,
    "text": "perfectly, we'd actually give it a pass."
  },
  {
    "startMs": 470080,
    "text": "And if it scored anything less than 100%"
  },
  {
    "startMs": 472240,
    "text": "on that kind of task, we would actually"
  },
  {
    "startMs": 473840,
    "text": "mark it as a failure."
  },
  {
    "startMs": 476560,
    "text": "So let's look at a different cut of our"
  },
  {
    "startMs": 478319,
    "text": "data from our experiments with this lens"
  },
  {
    "startMs": 480160,
    "text": "where we're looking for this perfect run"
  },
  {
    "startMs": 482400,
    "text": "100% accuracy pass. And let's look at"
  },
  {
    "startMs": 485039,
    "text": "what percent of the time each of these"
  },
  {
    "startMs": 487039,
    "text": "strategies was able to achieve a pass."
  },
  {
    "startMs": 489840,
    "text": "Um so we we've got executes um down at"
  },
  {
    "startMs": 492479,
    "text": "42% and we've got our more complex"
  },
  {
    "startMs": 494960,
    "text": "strategies doing a bit better up to 75%"
  },
  {
    "startMs": 497440,
    "text": "accuracy. Um and again this doesn't"
  },
  {
    "startMs": 500000,
    "text": "necessarily tell us a ton because um you"
  },
  {
    "startMs": 502879,
    "text": "know each of these strategies might um"
  },
  {
    "startMs": 505360,
    "text": "choose to use different budgets over"
  },
  {
    "startMs": 506800,
    "text": "time, right? So what we did here was we"
  },
  {
    "startMs": 508800,
    "text": "fixed the budget and we said within a"
  },
  {
    "startMs": 510400,
    "text": "fixed budget, how well do each of these"
  },
  {
    "startMs": 512320,
    "text": "strategies perform?"
  },
  {
    "startMs": 515599,
    "text": "And so what really matters to us"
  },
  {
    "startMs": 517039,
    "text": "actually is if you're trying to get this"
  },
  {
    "startMs": 519120,
    "text": "perfect answer and you're in the real"
  },
  {
    "startMs": 521120,
    "text": "world, you're running a business, what"
  },
  {
    "startMs": 522719,
    "text": "matters to you is the cost to you to get"
  },
  {
    "startMs": 525200,
    "text": "to that perfect answer. And so one way"
  },
  {
    "startMs": 527279,
    "text": "we can think about this is we had our"
  },
  {
    "startMs": 529040,
    "text": "execute strategy for example. The"
  },
  {
    "startMs": 530880,
    "text": "execute strategy around 40% of the time"
  },
  {
    "startMs": 533519,
    "text": "will give you that perfect answer. So on"
  },
  {
    "startMs": 535360,
    "text": "average, you can expect to have to run"
  },
  {
    "startMs": 536880,
    "text": "it three times and you should hopefully"
  },
  {
    "startMs": 538480,
    "text": "sometime in those three runs get a"
  },
  {
    "startMs": 540480,
    "text": "perfect answer. And as we talked about"
  },
  {
    "startMs": 542640,
    "text": "earlier, we fixed our budget to that"
  },
  {
    "startMs": 544560,
    "text": "highest token budget strategy, which was"
  },
  {
    "startMs": 546640,
    "text": "600,000 tokens. So if you spend 600,000"
  },
  {
    "startMs": 549120,
    "text": "tokens in each individual run, you have"
  },
  {
    "startMs": 551519,
    "text": "to run approximately three times. You"
  },
  {
    "startMs": 553360,
    "text": "can expect on average to have to spend"
  },
  {
    "startMs": 555200,
    "text": "1.8 million tokens with the execution"
  },
  {
    "startMs": 557920,
    "text": "strategy to get to your perfect answer."
  },
  {
    "startMs": 561279,
    "text": "And so if we take this analysis and run"
  },
  {
    "startMs": 563120,
    "text": "it across the board against all these"
  },
  {
    "startMs": 564720,
    "text": "strategies, this is actually the true"
  },
  {
    "startMs": 566959,
    "text": "cost it took in this domain for that"
  },
  {
    "startMs": 569040,
    "text": "agent to be useful for that strategy. So"
  },
  {
    "startMs": 571440,
    "text": "as Caitlyn mentioned for execute, which"
  },
  {
    "startMs": 573440,
    "text": "is our baseline, this is going to be 1.8"
  },
  {
    "startMs": 575760,
    "text": "million true total token cost for you."
  },
  {
    "startMs": 578959,
    "text": "But advise, grade, and dream are showing"
  },
  {
    "startMs": 580800,
    "text": "us a bit of difference. Crucially,"
  },
  {
    "startMs": 582560,
    "text": "advise and grade are actually quite"
  },
  {
    "startMs": 584399,
    "text": "token efficient when you think about the"
  },
  {
    "startMs": 586000,
    "text": "actual usage of the end output of each"
  },
  {
    "startMs": 588399,
    "text": "of these agents."
  },
  {
    "startMs": 591200,
    "text": "So what does this mean for you as a"
  },
  {
    "startMs": 592560,
    "text": "business? Well, it actually really"
  },
  {
    "startMs": 594560,
    "text": "depends on what kind of thing you want"
  },
  {
    "startMs": 596800,
    "text": "to optimize for and it's going to vary,"
  },
  {
    "startMs": 598720,
    "text": "right? There's going to be businesses"
  },
  {
    "startMs": 600000,
    "text": "who say, \"Actually, for me, the most"
  },
  {
    "startMs": 601680,
    "text": "important thing is to be really token"
  },
  {
    "startMs": 603360,
    "text": "efficient. In that case, you should"
  },
  {
    "startMs": 605040,
    "text": "probably pick the advised type of"
  },
  {
    "startMs": 606480,
    "text": "strategy in order to solve for that"
  },
  {
    "startMs": 608000,
    "text": "particular domain in which you want to"
  },
  {
    "startMs": 609360,
    "text": "optimize that.\" There's going to be"
  },
  {
    "startMs": 610959,
    "text": "other areas or other businesses where"
  },
  {
    "startMs": 612399,
    "text": "you're going to say, I'm not going to"
  },
  {
    "startMs": 614079,
    "text": "care so much about token efficiency"
  },
  {
    "startMs": 615519,
    "text": "because what I really care about is"
  },
  {
    "startMs": 616880,
    "text": "reliability of that answer and so I need"
  },
  {
    "startMs": 619200,
    "text": "to maximize the percentage of runs in"
  },
  {
    "startMs": 620880,
    "text": "which I get that perfect answer. In"
  },
  {
    "startMs": 622640,
    "text": "which case, you would actually pick"
  },
  {
    "startMs": 623839,
    "text": "completely different strategies. You"
  },
  {
    "startMs": 624959,
    "text": "probably lean towards grade or dream."
  },
  {
    "startMs": 628880,
    "text": "So if you take away one thing, the thing"
  },
  {
    "startMs": 630399,
    "text": "we want everyone to think about is this"
  },
  {
    "startMs": 631920,
    "text": "idea that tokens are not fungeible. You"
  },
  {
    "startMs": 634399,
    "text": "can use your tokens to execute. You can"
  },
  {
    "startMs": 636160,
    "text": "brute force your way through your task"
  },
  {
    "startMs": 637680,
    "text": "and you can throw more budget at it. But"
  },
  {
    "startMs": 639680,
    "text": "if you get really smart about having"
  },
  {
    "startMs": 641360,
    "text": "your tokens do these different jobs and"
  },
  {
    "startMs": 643519,
    "text": "try these different strategies, you're"
  },
  {
    "startMs": 645279,
    "text": "very very likely to be able to get a"
  },
  {
    "startMs": 646959,
    "text": "better outcome for the task at hand"
  },
  {
    "startMs": 649279,
    "text": "within a fixed budget."
  },
  {
    "startMs": 652079,
    "text": "And so let's talk a little bit about how"
  },
  {
    "startMs": 653440,
    "text": "we actually build strategies and how we"
  },
  {
    "startMs": 655040,
    "text": "bring this to life. Um so we've done a"
  },
  {
    "startMs": 657040,
    "text": "lot of work to create a really excellent"
  },
  {
    "startMs": 658959,
    "text": "harness for individual agents. Um and if"
  },
  {
    "startMs": 661519,
    "text": "you see this uh picture at the bottom"
  },
  {
    "startMs": 663519,
    "text": "here, this is actually the architecture"
  },
  {
    "startMs": 665279,
    "text": "that we've used for cloud managed agents"
  },
  {
    "startMs": 667760,
    "text": "um which is our Aentic solution that we"
  },
  {
    "startMs": 669680,
    "text": "give to you within the cloud platform."
  },
  {
    "startMs": 672240,
    "text": "And what we do on top of this is we"
  },
  {
    "startMs": 674000,
    "text": "start to get into the meta harness level"
  },
  {
    "startMs": 675839,
    "text": "like the multi- aent orchestration and"
  },
  {
    "startMs": 677920,
    "text": "execution level where this strategy can"
  },
  {
    "startMs": 680880,
    "text": "go and be um coordinated between our"
  },
  {
    "startMs": 683120,
    "text": "executor and our adviser or the other"
  },
  {
    "startMs": 685360,
    "text": "agents within our strategy. And some of"
  },
  {
    "startMs": 688000,
    "text": "these um like dreaming and outcomes we"
  },
  {
    "startMs": 690320,
    "text": "actually give to you out of the box"
  },
  {
    "startMs": 691519,
    "text": "within cloud manage agents."
  },
  {
    "startMs": 694399,
    "text": "So with those set of primitives it's"
  },
  {
    "startMs": 696320,
    "text": "actually relatively trivial for us to"
  },
  {
    "startMs": 698240,
    "text": "construct this kind of you know"
  },
  {
    "startMs": 700079,
    "text": "architecture where we're able to combine"
  },
  {
    "startMs": 701920,
    "text": "these different types of strategies and"
  },
  {
    "startMs": 703760,
    "text": "figure out how to they should work"
  },
  {
    "startMs": 705200,
    "text": "together. So for example it's relatively"
  },
  {
    "startMs": 707360,
    "text": "trivial for us to say okay now with this"
  },
  {
    "startMs": 709200,
    "text": "I can take a task and I should be able"
  },
  {
    "startMs": 711279,
    "text": "to execute it but also allow it to"
  },
  {
    "startMs": 713200,
    "text": "advise and fable is back online. So we"
  },
  {
    "startMs": 715680,
    "text": "could actually say Fable is the one"
  },
  {
    "startMs": 717040,
    "text": "that's actually advising uh the"
  },
  {
    "startMs": 718720,
    "text": "executor. And then I can take all these"
  },
  {
    "startMs": 720480,
    "text": "results and say send them to a greater"
  },
  {
    "startMs": 722480,
    "text": "so that I can make sure that this is"
  },
  {
    "startMs": 723839,
    "text": "verifying in a loop that makes sense."
  },
  {
    "startMs": 725760,
    "text": "And if it passes, that's awesome. I want"
  },
  {
    "startMs": 727680,
    "text": "to send all of that stuff to Dreaming"
  },
  {
    "startMs": 729120,
    "text": "and make sure that my next run is better"
  },
  {
    "startMs": 730880,
    "text": "than ever."
  },
  {
    "startMs": 732959,
    "text": "And of course, you don't have to stop"
  },
  {
    "startMs": 734240,
    "text": "there, right? If the right primitives"
  },
  {
    "startMs": 735920,
    "text": "are there and the right coordination is"
  },
  {
    "startMs": 737360,
    "text": "there, then you can actually construct"
  },
  {
    "startMs": 739279,
    "text": "really complex setups that fit for all"
  },
  {
    "startMs": 741360,
    "text": "the different types of dynamic problems"
  },
  {
    "startMs": 743279,
    "text": "that you have. You can invent these"
  },
  {
    "startMs": 745279,
    "text": "kinds of large-scale architectures,"
  },
  {
    "startMs": 747200,
    "text": "again, very triv. And you could also"
  },
  {
    "startMs": 749200,
    "text": "invent completely new jobs, not just the"
  },
  {
    "startMs": 751120,
    "text": "ones of the pieces that Caitlyn and I"
  },
  {
    "startMs": 753200,
    "text": "have presented in this conversation."
  },
  {
    "startMs": 756240,
    "text": "So, a big goal that we have over time is"
  },
  {
    "startMs": 758320,
    "text": "to get our models better and better and"
  },
  {
    "startMs": 760320,
    "text": "our platform better and better at"
  },
  {
    "startMs": 762000,
    "text": "dynamically constructing these"
  },
  {
    "startMs": 763360,
    "text": "strategies for you as you're doing work."
  },
  {
    "startMs": 765839,
    "text": "But in the meantime, as we're working"
  },
  {
    "startMs": 767360,
    "text": "our way there, we would love for you to"
  },
  {
    "startMs": 769279,
    "text": "continue to think about this idea that"
  },
  {
    "startMs": 770880,
    "text": "you should give your tokens jobs and you"
  },
  {
    "startMs": 772800,
    "text": "should use different novel strategies by"
  },
  {
    "startMs": 774800,
    "text": "combining these primitives in order to"
  },
  {
    "startMs": 776800,
    "text": "get the outcomes that you want for your"
  },
  {
    "startMs": 778160,
    "text": "tasks."
  },
  {
    "startMs": 779920,
    "text": ">> Thanks for joining us."
  }
]