[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 13519,
    "text": "Okay. Um, good morning everyone. So, um,"
  },
  {
    "startMs": 16320,
    "text": "I'm Tisha and I have Sushim with me as"
  },
  {
    "startMs": 18960,
    "text": "my co-presenter. All right. So, we'll be"
  },
  {
    "startMs": 21439,
    "text": "talking about the most expensive"
  },
  {
    "startMs": 23920,
    "text": "question in AI today. I think a lot of"
  },
  {
    "startMs": 27359,
    "text": "you would have come across the scenario"
  },
  {
    "startMs": 29119,
    "text": "that um you know when you opened an AI"
  },
  {
    "startMs": 32399,
    "text": "bill like through your agent workflows"
  },
  {
    "startMs": 34960,
    "text": "um you couldn't actually trace back"
  },
  {
    "startMs": 37600,
    "text": "where that bill was actually coming from"
  },
  {
    "startMs": 39760,
    "text": "right and um and I don't think that's a"
  },
  {
    "startMs": 42719,
    "text": "problem right now because right now the"
  },
  {
    "startMs": 45760,
    "text": "industry is valuing you know um token"
  },
  {
    "startMs": 48640,
    "text": "maxing that is like spending the most"
  },
  {
    "startMs": 51760,
    "text": "amount of tokens for exploration for all"
  },
  {
    "startMs": 54239,
    "text": "of those purposes"
  },
  {
    "startMs": 55760,
    "text": "And um people are proud to call"
  },
  {
    "startMs": 57520,
    "text": "themselves token billionaires and um I"
  },
  {
    "startMs": 59520,
    "text": "think that's all right but this talk is"
  },
  {
    "startMs": 62320,
    "text": "you know the shift from token maxing to"
  },
  {
    "startMs": 65920,
    "text": "value maxing you know how do we get"
  },
  {
    "startMs": 68159,
    "text": "there and um we'll talk about it from"
  },
  {
    "startMs": 72479,
    "text": "this question um who spent all the"
  },
  {
    "startMs": 75520,
    "text": "tokens and um if anyone spent all the"
  },
  {
    "startMs": 79680,
    "text": "tokens there has to be value associated"
  },
  {
    "startMs": 81759,
    "text": "with this right and that is um the talk"
  },
  {
    "startMs": 86400,
    "text": "about."
  },
  {
    "startMs": 90400,
    "text": "All right. Now in order to minimize the"
  },
  {
    "startMs": 93439,
    "text": "gap you know from token maxing to value"
  },
  {
    "startMs": 96880,
    "text": "maxing we'll kind of see we'll observe"
  },
  {
    "startMs": 99920,
    "text": "the patterns which the like the existing"
  },
  {
    "startMs": 103680,
    "text": "um u like the past software evolution"
  },
  {
    "startMs": 106079,
    "text": "eras had like for instance when we talk"
  },
  {
    "startMs": 109520,
    "text": "about the SAS era the interface was UI"
  },
  {
    "startMs": 113360,
    "text": "and the control was in the form of usage"
  },
  {
    "startMs": 116479,
    "text": "caps right like or the seat limits or"
  },
  {
    "startMs": 118640,
    "text": "tier based policies"
  },
  {
    "startMs": 120799,
    "text": "Now when we moved on to the cloud era,"
  },
  {
    "startMs": 123840,
    "text": "the control surface again changed. The"
  },
  {
    "startMs": 126479,
    "text": "model became pay as you go and the"
  },
  {
    "startMs": 128239,
    "text": "control moved like in the form of"
  },
  {
    "startMs": 130879,
    "text": "autoprovisioning and you know"
  },
  {
    "startMs": 132239,
    "text": "autoscaling policies."
  },
  {
    "startMs": 134640,
    "text": "Now we are in the agentic era right and"
  },
  {
    "startMs": 138640,
    "text": "um now how the cost is calculated here"
  },
  {
    "startMs": 142480,
    "text": "is in the form of model calls right like"
  },
  {
    "startMs": 146239,
    "text": "u how like the code calls your model but"
  },
  {
    "startMs": 151520,
    "text": "what we've observed is that there isn't"
  },
  {
    "startMs": 154319,
    "text": "a proper control plane in place for that"
  },
  {
    "startMs": 157920,
    "text": "like we do have control plane in place"
  },
  {
    "startMs": 160239,
    "text": "for you in place as model gateways where"
  },
  {
    "startMs": 163120,
    "text": "they're um are hard caps or there is"
  },
  {
    "startMs": 166959,
    "text": "model routing to downgrade the model but"
  },
  {
    "startMs": 169840,
    "text": "the part like where the code you know"
  },
  {
    "startMs": 173840,
    "text": "calls the model that um is what we'll be"
  },
  {
    "startMs": 177360,
    "text": "talking about uh today"
  },
  {
    "startMs": 180480,
    "text": "and um we also you know see um like in"
  },
  {
    "startMs": 185840,
    "text": "the last year we've seen a lot of"
  },
  {
    "startMs": 187760,
    "text": "unbounded consumption happening like um"
  },
  {
    "startMs": 192000,
    "text": "if you've read the news. There was news"
  },
  {
    "startMs": 194959,
    "text": "about the like the uh AI budget for Uber"
  },
  {
    "startMs": 199040,
    "text": "getting exhausted within 4 months and um"
  },
  {
    "startMs": 201840,
    "text": "there were companies who like who ran"
  },
  {
    "startMs": 204480,
    "text": "into you know"
  },
  {
    "startMs": 206720,
    "text": "like hundreds of millions of dollars"
  },
  {
    "startMs": 208319,
    "text": "within just months or days and like"
  },
  {
    "startMs": 210239,
    "text": "there were a lot of um like other news"
  },
  {
    "startMs": 212959,
    "text": "in place as well where like these"
  },
  {
    "startMs": 215440,
    "text": "runaway loops um led to a very like"
  },
  {
    "startMs": 219840,
    "text": "massive increase in the cost and there"
  },
  {
    "startMs": 222959,
    "text": "wasn't proper mechanisms to control it."
  },
  {
    "startMs": 227519,
    "text": "Um so when we see all of this the first"
  },
  {
    "startMs": 230560,
    "text": "thing that comes to our mind is is there"
  },
  {
    "startMs": 232560,
    "text": "a tool or is there a product to save us?"
  },
  {
    "startMs": 234879,
    "text": "But uh we'll instead talk about the"
  },
  {
    "startMs": 238640,
    "text": "first principles of how you know we can"
  },
  {
    "startMs": 242159,
    "text": "design a system which is actually true"
  },
  {
    "startMs": 245040,
    "text": "enough to solve the problem from the"
  },
  {
    "startMs": 247040,
    "text": "very root. So for that let's um like"
  },
  {
    "startMs": 250720,
    "text": "dive onto the principles. First of all,"
  },
  {
    "startMs": 253040,
    "text": "let's talk about token being the unit of"
  },
  {
    "startMs": 255680,
    "text": "cost. Right? We are charged in terms of"
  },
  {
    "startMs": 258720,
    "text": "token. So the now we have to see value"
  },
  {
    "startMs": 263040,
    "text": "also in terms of token. Right? Next um"
  },
  {
    "startMs": 267440,
    "text": "we all know that cost is created at the"
  },
  {
    "startMs": 270240,
    "text": "LLM like the model call boundary. Um so"
  },
  {
    "startMs": 273840,
    "text": "that is what we'll have to track and if"
  },
  {
    "startMs": 277600,
    "text": "we don't have proper attribution like if"
  },
  {
    "startMs": 279520,
    "text": "we don't know what agent want run made"
  },
  {
    "startMs": 282639,
    "text": "that particular call we we can't you"
  },
  {
    "startMs": 285440,
    "text": "know control it right we we just know"
  },
  {
    "startMs": 288479,
    "text": "the like the broad uh picture of what"
  },
  {
    "startMs": 291759,
    "text": "went wrong but we don't we can't you"
  },
  {
    "startMs": 293360,
    "text": "know trace it back or narrow it down. So"
  },
  {
    "startMs": 295600,
    "text": "that is why attribution is a very"
  },
  {
    "startMs": 297759,
    "text": "important element to have and um like"
  },
  {
    "startMs": 301040,
    "text": "once you know which particular run or"
  },
  {
    "startMs": 304240,
    "text": "which particular agent is actually you"
  },
  {
    "startMs": 307280,
    "text": "know attributing to the cost you should"
  },
  {
    "startMs": 309520,
    "text": "have proper policies in place to"
  },
  {
    "startMs": 311600,
    "text": "actually stop it. Like um let's take"
  },
  {
    "startMs": 314320,
    "text": "example that um if you have a you know"
  },
  {
    "startMs": 317440,
    "text": "um a loop which is you know running um"
  },
  {
    "startMs": 322320,
    "text": "very excessively and which is not"
  },
  {
    "startMs": 324560,
    "text": "required or you know if your context is"
  },
  {
    "startMs": 327840,
    "text": "growing very out of range. you should"
  },
  {
    "startMs": 330320,
    "text": "have in place policies which can um like"
  },
  {
    "startMs": 333600,
    "text": "solve that particular thing there and"
  },
  {
    "startMs": 335759,
    "text": "there instead of halting that and if um"
  },
  {
    "startMs": 338639,
    "text": "and as the last resort only a like a"
  },
  {
    "startMs": 341440,
    "text": "halting or a should happen from a budget"
  },
  {
    "startMs": 344080,
    "text": "cap. So these are the first principles."
  },
  {
    "startMs": 346720,
    "text": "Now let's see how we can you know define"
  },
  {
    "startMs": 350240,
    "text": "an ideal uh platform on top of that from"
  },
  {
    "startMs": 353520,
    "text": "these principles which we talked about."
  },
  {
    "startMs": 356000,
    "text": "All right. Uh so one thing which is very"
  },
  {
    "startMs": 358880,
    "text": "important that which matters here is"
  },
  {
    "startMs": 360720,
    "text": "that u when we talk about um like the um"
  },
  {
    "startMs": 365039,
    "text": "existing frameworks for token ops or for"
  },
  {
    "startMs": 367680,
    "text": "token management most of them are at the"
  },
  {
    "startMs": 371039,
    "text": "um like u basically monitor the model uh"
  },
  {
    "startMs": 375759,
    "text": "request. they like they are like model"
  },
  {
    "startMs": 379360,
    "text": "gateways which will u you know u"
  },
  {
    "startMs": 382240,
    "text": "basically um do like model routing or"
  },
  {
    "startMs": 386160,
    "text": "hard budget capping. But what we need"
  },
  {
    "startMs": 388880,
    "text": "right now is something which you know um"
  },
  {
    "startMs": 393919,
    "text": "like monitors you at the run instead."
  },
  {
    "startMs": 395759,
    "text": "Like um if you see we need something uh"
  },
  {
    "startMs": 399440,
    "text": "which can control the loop between like"
  },
  {
    "startMs": 403759,
    "text": "the agent call between the tool um and"
  },
  {
    "startMs": 407680,
    "text": "the agent. something you know which can"
  },
  {
    "startMs": 410240,
    "text": "um uh see or control the the spawning of"
  },
  {
    "startMs": 414240,
    "text": "multiple sub aents happening from a one"
  },
  {
    "startMs": 416960,
    "text": "main agent or um like something which"
  },
  {
    "startMs": 420240,
    "text": "can control the growing of context. So"
  },
  {
    "startMs": 422560,
    "text": "like that is the need of the right and"
  },
  {
    "startMs": 425599,
    "text": "that is what we need. So for all of this"
  },
  {
    "startMs": 428800,
    "text": "um we like uh kind of are proposing a"
  },
  {
    "startMs": 432560,
    "text": "platform which first of all um has a"
  },
  {
    "startMs": 435680,
    "text": "cumulative budget across like the uh"
  },
  {
    "startMs": 439840,
    "text": "like the attribution runs which happened"
  },
  {
    "startMs": 444000,
    "text": "and then where enforcement actually"
  },
  {
    "startMs": 446560,
    "text": "happens in call path rather than um you"
  },
  {
    "startMs": 450960,
    "text": "know a separate thing like for example"
  },
  {
    "startMs": 453360,
    "text": "if something goes wrong if your like if"
  },
  {
    "startMs": 455599,
    "text": "your context is just growing heavily."
  },
  {
    "startMs": 458080,
    "text": "Then like in place compaction should"
  },
  {
    "startMs": 463120,
    "text": "happen or like in place caching or"
  },
  {
    "startMs": 465520,
    "text": "something like that should happen. And"
  },
  {
    "startMs": 467599,
    "text": "um after that if like after basically"
  },
  {
    "startMs": 470639,
    "text": "exhausting the list of all in place"
  },
  {
    "startMs": 473280,
    "text": "policies only like uh the budget cap"
  },
  {
    "startMs": 477360,
    "text": "should happen at the very last. Um so"
  },
  {
    "startMs": 479840,
    "text": "that is something which we are"
  },
  {
    "startMs": 481680,
    "text": "proposing. But um if you look at the"
  },
  {
    "startMs": 484240,
    "text": "landscape today, if you see the uh like"
  },
  {
    "startMs": 487039,
    "text": "the uh tools like um this light LLM,"
  },
  {
    "startMs": 490400,
    "text": "port key, cloudflare, all of those they"
  },
  {
    "startMs": 492479,
    "text": "happen at again the request level right"
  },
  {
    "startMs": 496960,
    "text": "um like if you see like halting is"
  },
  {
    "startMs": 500080,
    "text": "there, routing is there for some of"
  },
  {
    "startMs": 502639,
    "text": "those but all of this again is at a"
  },
  {
    "startMs": 506080,
    "text": "request and you can't control the cost"
  },
  {
    "startMs": 508160,
    "text": "at the uh request layer uh at the model"
  },
  {
    "startMs": 511919,
    "text": "layer, Right."
  },
  {
    "startMs": 514080,
    "text": "So this is the missing piece which is"
  },
  {
    "startMs": 516560,
    "text": "you know the"
  },
  {
    "startMs": 518959,
    "text": "u basically navigating it at the um you"
  },
  {
    "startMs": 522800,
    "text": "know the model the agent run layer."
  },
  {
    "startMs": 527120,
    "text": "So for that we have token ops which is"
  },
  {
    "startMs": 530320,
    "text": "uh you know a runaway token governance"
  },
  {
    "startMs": 532880,
    "text": "for AI agents and u this is the uh"
  },
  {
    "startMs": 537519,
    "text": "architecture for that. So first of all"
  },
  {
    "startMs": 540080,
    "text": "one thing I would want to highlight is"
  },
  {
    "startMs": 542080,
    "text": "the like the intentional design decision"
  },
  {
    "startMs": 544959,
    "text": "we took here was an out ofbound plane."
  },
  {
    "startMs": 547920,
    "text": "So it doesn't interfere with your code"
  },
  {
    "startMs": 550320,
    "text": "at all. Um so if you see here that out"
  },
  {
    "startMs": 553440,
    "text": "of the bandound plane has three modules"
  },
  {
    "startMs": 555680,
    "text": "which I'll be talking about. The first"
  },
  {
    "startMs": 557360,
    "text": "one being instrumentation. It is a"
  },
  {
    "startMs": 559120,
    "text": "common observability layer where you"
  },
  {
    "startMs": 561279,
    "text": "know you'll u like uh have u like the"
  },
  {
    "startMs": 565279,
    "text": "basic telemetry the open telemetry the"
  },
  {
    "startMs": 567839,
    "text": "cost in microns and um like the um like"
  },
  {
    "startMs": 572320,
    "text": "enrichment layer and basically um the uh"
  },
  {
    "startMs": 577040,
    "text": "attribution like what caused that uh"
  },
  {
    "startMs": 580320,
    "text": "like particular run"
  },
  {
    "startMs": 583360,
    "text": "and then there is um obviously"
  },
  {
    "startMs": 585600,
    "text": "accounting where"
  },
  {
    "startMs": 588240,
    "text": "you'll basically accumulate it in a kind"
  },
  {
    "startMs": 591279,
    "text": "of a ledger like the total runs which"
  },
  {
    "startMs": 594720,
    "text": "are happening. And finally we have this"
  },
  {
    "startMs": 596880,
    "text": "enforced layer which has uh two main"
  },
  {
    "startMs": 600640,
    "text": "purposes. one is steering it um through"
  },
  {
    "startMs": 604800,
    "text": "the policies which we've defined which I"
  },
  {
    "startMs": 606880,
    "text": "think will cover later and um then we"
  },
  {
    "startMs": 610240,
    "text": "have halt in place as the you know final"
  },
  {
    "startMs": 613360,
    "text": "um like u final thing if um you know"
  },
  {
    "startMs": 616560,
    "text": "your budget is getting exhausted"
  },
  {
    "startMs": 619120,
    "text": "so yeah that is there now when we again"
  },
  {
    "startMs": 622880,
    "text": "look at the landscape this kind of will"
  },
  {
    "startMs": 626800,
    "text": "solve a lot of problems"
  },
  {
    "startMs": 629839,
    "text": "um which kind of happened uh when we"
  },
  {
    "startMs": 632640,
    "text": "like look at the previous um tools or"
  },
  {
    "startMs": 635279,
    "text": "products there because uh it is at"
  },
  {
    "startMs": 638320,
    "text": "happening at run and it is you know uh"
  },
  {
    "startMs": 641120,
    "text": "helping you solve the problem from the"
  },
  {
    "startMs": 644079,
    "text": "very root by steering it in place."
  },
  {
    "startMs": 648320,
    "text": "All right. So uh with this I would like"
  },
  {
    "startMs": 651519,
    "text": "to hand it over to Sashim for the demo."
  },
  {
    "startMs": 653920,
    "text": ">> Yeah."
  },
  {
    "startMs": 657120,
    "text": ">> Oh yeah. Now I think I should be able to"
  },
  {
    "startMs": 659200,
    "text": "everyone in the back can hear me. All"
  },
  {
    "startMs": 660800,
    "text": "right, perfect. So yeah, we have"
  },
  {
    "startMs": 663600,
    "text": "established the principles behind token"
  },
  {
    "startMs": 665279,
    "text": "ops till now. Right. Now let's shift"
  },
  {
    "startMs": 668079,
    "text": "gears, talk about the design part of it"
  },
  {
    "startMs": 669920,
    "text": "and uh maybe get into the code and the"
  },
  {
    "startMs": 672240,
    "text": "eventual demo. Right? So what I have"
  },
  {
    "startMs": 674640,
    "text": "behind me on the screen is the like"
  },
  {
    "startMs": 676720,
    "text": "bird's eye view of what token ops looks"
  },
  {
    "startMs": 679200,
    "text": "like today. It's it's three layers."
  },
  {
    "startMs": 681760,
    "text": "We'll go left to right and top to"
  },
  {
    "startMs": 683040,
    "text": "bottom. So on the left most you have"
  },
  {
    "startMs": 684800,
    "text": "your own agent runtime which you're"
  },
  {
    "startMs": 686240,
    "text": "trying to instrument and kind of manage"
  },
  {
    "startMs": 687760,
    "text": "the cost for right the middle layer is"
  },
  {
    "startMs": 690079,
    "text": "what we're calling the bridge that"
  },
  {
    "startMs": 692160,
    "text": "basically shuffles data between your"
  },
  {
    "startMs": 693760,
    "text": "agent and the control plane and the"
  },
  {
    "startMs": 696160,
    "text": "control plane is where the mind of the"
  },
  {
    "startMs": 698160,
    "text": "system lies right so let's talk about"
  },
  {
    "startMs": 700160,
    "text": "the bridge layer very briefly if we uh"
  },
  {
    "startMs": 702480,
    "text": "go from top to bottom you have the"
  },
  {
    "startMs": 703839,
    "text": "attribution on top so what we're trying"
  },
  {
    "startMs": 705920,
    "text": "to do here is every agent run that you"
  },
  {
    "startMs": 708480,
    "text": "do it's attributed to some user"
  },
  {
    "startMs": 710880,
    "text": "dimensions so the idea is everything"
  },
  {
    "startMs": 713040,
    "text": "that you do every run of the agent is"
  },
  {
    "startMs": 715440,
    "text": "accounted to some usability or some"
  },
  {
    "startMs": 717200,
    "text": "usage. This comes in handy later. We'll"
  },
  {
    "startMs": 719040,
    "text": "talk about it. Uh the second part which"
  },
  {
    "startMs": 721120,
    "text": "is the boundary annotation that you see"
  },
  {
    "startMs": 723040,
    "text": "this is pretty much the heart and soul"
  },
  {
    "startMs": 724560,
    "text": "of this middle layer. So the idea behind"
  },
  {
    "startMs": 727200,
    "text": "the boundary annotation is that you take"
  },
  {
    "startMs": 728959,
    "text": "any method. It doesn't matter what"
  },
  {
    "startMs": 730560,
    "text": "framework you're using. You might be"
  },
  {
    "startMs": 731760,
    "text": "using uh let's say lang chain lang"
  },
  {
    "startMs": 733760,
    "text": "whatever. If you have a method you can"
  },
  {
    "startMs": 735600,
    "text": "annotate it with boundary. What this"
  },
  {
    "startMs": 737600,
    "text": "annotation is going to do is it's going"
  },
  {
    "startMs": 739120,
    "text": "to do two things. First it's going to"
  },
  {
    "startMs": 740720,
    "text": "track the input and the output and it's"
  },
  {
    "startMs": 742720,
    "text": "going to flight that up to the control"
  },
  {
    "startMs": 744399,
    "text": "layer and record it there as a ledger"
  },
  {
    "startMs": 746399,
    "text": "entry. Now this will be annotated with"
  },
  {
    "startMs": 748720,
    "text": "the further agent run ID and the other"
  },
  {
    "startMs": 750480,
    "text": "attributes and so on. The second thing"
  },
  {
    "startMs": 752240,
    "text": "the boundary annotation does is it acts"
  },
  {
    "startMs": 754079,
    "text": "as a channel through which the control"
  },
  {
    "startMs": 755600,
    "text": "plane can push actions down to the"
  },
  {
    "startMs": 757680,
    "text": "agent. This is where the entire"
  },
  {
    "startMs": 759120,
    "text": "intelligence lies. So we do not have a"
  },
  {
    "startMs": 761519,
    "text": "single directional highway. We want the"
  },
  {
    "startMs": 763600,
    "text": "control plane to be able to tweak the"
  },
  {
    "startMs": 765839,
    "text": "behavior of the agent on the fly to"
  },
  {
    "startMs": 768000,
    "text": "ensure that we are able to squeeze in"
  },
  {
    "startMs": 769519,
    "text": "more runs inside our budget cap. Right"
  },
  {
    "startMs": 772399,
    "text": "now let's say the control plane pushes"
  },
  {
    "startMs": 774079,
    "text": "down an action. Let's take a small"
  },
  {
    "startMs": 775760,
    "text": "example. Let's say you have a rag"
  },
  {
    "startMs": 776959,
    "text": "retrieval tool which is generating like"
  },
  {
    "startMs": 778480,
    "text": "20 chunks every retrieval for every call"
  },
  {
    "startMs": 781040,
    "text": "and that's eating up eating up your"
  },
  {
    "startMs": 782560,
    "text": "budget. And let's say the LLM is not"
  },
  {
    "startMs": 784720,
    "text": "even using the chunks that are after"
  },
  {
    "startMs": 786079,
    "text": "five because they are just not relevant,"
  },
  {
    "startMs": 787360,
    "text": "right? They're sorted by relevance. So"
  },
  {
    "startMs": 789600,
    "text": "let's say the control plane observes"
  },
  {
    "startMs": 790880,
    "text": "this and it wants to limit the output to"
  },
  {
    "startMs": 793279,
    "text": "just five chunks. So it can push down an"
  },
  {
    "startMs": 795120,
    "text": "action but that action has to be"
  },
  {
    "startMs": 797279,
    "text": "received by boundary and then has to be"
  },
  {
    "startMs": 799200,
    "text": "executed by something. That is where the"
  },
  {
    "startMs": 801040,
    "text": "third node, the governor node comes in."
  },
  {
    "startMs": 802880,
    "text": "The governor knows what actions are"
  },
  {
    "startMs": 806160,
    "text": "allowed on your agent by you as a"
  },
  {
    "startMs": 808160,
    "text": "developer and it receives those actions"
  },
  {
    "startMs": 810240,
    "text": "from the control plane and knows how to"
  },
  {
    "startMs": 811839,
    "text": "apply it in a non-destructive way. So"
  },
  {
    "startMs": 814160,
    "text": "that's the first three. The fourth one"
  },
  {
    "startMs": 815760,
    "text": "wrap uh the wrap complete is essentially"
  },
  {
    "startMs": 817920,
    "text": "just a helper method. So as we know most"
  },
  {
    "startMs": 820399,
    "text": "of the agent providers or the model"
  },
  {
    "startMs": 821839,
    "text": "providers they provide objects rather"
  },
  {
    "startMs": 823200,
    "text": "than methods for their LMS right. So"
  },
  {
    "startMs": 825360,
    "text": "wrap complete is just another way of"
  },
  {
    "startMs": 826800,
    "text": "applying boundary on objects rather than"
  },
  {
    "startMs": 828399,
    "text": "methods. Let's shift right to the"
  },
  {
    "startMs": 831279,
    "text": "control plane. On the control plane the"
  },
  {
    "startMs": 832959,
    "text": "first layer is the segment. Now this is"
  },
  {
    "startMs": 835279,
    "text": "where the attribution that we talked"
  },
  {
    "startMs": 836800,
    "text": "about earlier comes into picture. So any"
  },
  {
    "startMs": 839360,
    "text": "dimensions that you float from the"
  },
  {
    "startMs": 841199,
    "text": "attribution layer. Let's say you have a"
  },
  {
    "startMs": 843279,
    "text": "preview agent that you share with"
  },
  {
    "startMs": 844560,
    "text": "everyone in this room and your agent is"
  },
  {
    "startMs": 846880,
    "text": "floating a dimension saying that cohort"
  },
  {
    "startMs": 849040,
    "text": "is AIE 2026 right so you can create a"
  },
  {
    "startMs": 852160,
    "text": "segment which is a cohort of users which"
  },
  {
    "startMs": 855040,
    "text": "is based on this tag like dimension"
  },
  {
    "startMs": 856959,
    "text": "being AI 2026 right and you can apply"
  },
  {
    "startMs": 859680,
    "text": "your budgets at this cohort level so you"
  },
  {
    "startMs": 861680,
    "text": "don't necessarily have to restrict"
  },
  {
    "startMs": 863040,
    "text": "everything at an agent level or a run"
  },
  {
    "startMs": 864959,
    "text": "level you can do you can do rollups you"
  },
  {
    "startMs": 867440,
    "text": "can do fine grain or coarse grain"
  },
  {
    "startMs": 868959,
    "text": "control right so that's the segmentation"
  },
  {
    "startMs": 871120,
    "text": "part of Ledger as I mentioned is just"
  },
  {
    "startMs": 873199,
    "text": "one agent run all the traces in one"
  },
  {
    "startMs": 875600,
    "text": "place. Then you have budgets. Budgets"
  },
  {
    "startMs": 877680,
    "text": "are basically just the static thresholds"
  },
  {
    "startMs": 879519,
    "text": "that work across a time window against a"
  },
  {
    "startMs": 881920,
    "text": "particular segment or an agent run. And"
  },
  {
    "startMs": 883839,
    "text": "then you have actions. So on the actions"
  },
  {
    "startMs": 885839,
    "text": "part we have broadly two flavors. First"
  },
  {
    "startMs": 888000,
    "text": "is the halt type actions which basically"
  },
  {
    "startMs": 889839,
    "text": "just kill your agent if it exceeds a"
  },
  {
    "startMs": 891839,
    "text": "budget. The second part where we are"
  },
  {
    "startMs": 893680,
    "text": "adding value is the steer type actions."
  },
  {
    "startMs": 896399,
    "text": "So here we do not kill the agent."
  },
  {
    "startMs": 898240,
    "text": "Instead we try to steer the behavior of"
  },
  {
    "startMs": 900079,
    "text": "the agent or the components of the agent"
  },
  {
    "startMs": 902720,
    "text": "to try and fit that particular run"
  },
  {
    "startMs": 905199,
    "text": "within the alerted budget. Right? And"
  },
  {
    "startMs": 907680,
    "text": "then the policies layer is where it all"
  },
  {
    "startMs": 909440,
    "text": "comes together. You basically uh group"
  },
  {
    "startMs": 911600,
    "text": "the budgets the actions and then set"
  },
  {
    "startMs": 913600,
    "text": "your policies against certain segments"
  },
  {
    "startMs": 915279,
    "text": "or agent runs and that is where it"
  },
  {
    "startMs": 916959,
    "text": "executes. Right? So moving on uh what"
  },
  {
    "startMs": 920079,
    "text": "changes in your code that is the"
  },
  {
    "startMs": 921600,
    "text": "boundary annotation that we just talked"
  },
  {
    "startMs": 922880,
    "text": "about. As Disha mentioned earlier this"
  },
  {
    "startMs": 924800,
    "text": "is all out of band. So you do not have"
  },
  {
    "startMs": 926480,
    "text": "to change your code. You just have to"
  },
  {
    "startMs": 927680,
    "text": "apply the annotation on the methods that"
  },
  {
    "startMs": 929279,
    "text": "you have. This boundary annotation will"
  },
  {
    "startMs": 931040,
    "text": "take care of floating all the"
  },
  {
    "startMs": 932320,
    "text": "information up to the control plane. And"
  },
  {
    "startMs": 934079,
    "text": "uh the control plane lies in your own"
  },
  {
    "startMs": 935600,
    "text": "tenant. So you do not need to worry"
  },
  {
    "startMs": 937040,
    "text": "about any data leaks or anything. Then"
  },
  {
    "startMs": 939760,
    "text": "if I talk about the governor, so for the"
  },
  {
    "startMs": 942000,
    "text": "governor, you just have to create an"
  },
  {
    "startMs": 943360,
    "text": "instance. You just have to pass it your"
  },
  {
    "startMs": 944720,
    "text": "own configs. These configs will"
  },
  {
    "startMs": 946399,
    "text": "basically declare what sort of actions"
  },
  {
    "startMs": 948800,
    "text": "are allowed for those agents, right? so"
  },
  {
    "startMs": 950959,
    "text": "that your control plane cannot just"
  },
  {
    "startMs": 953279,
    "text": "willingly do any random things on your"
  },
  {
    "startMs": 955360,
    "text": "on your agents. So before we move on to"
  },
  {
    "startMs": 958079,
    "text": "the demo, I'll just briefly touch upon"
  },
  {
    "startMs": 960160,
    "text": "the uh test that we're going to use"
  },
  {
    "startMs": 961680,
    "text": "today. So it's a simple two agent"
  },
  {
    "startMs": 963839,
    "text": "workflow. We have a research agent which"
  },
  {
    "startMs": 966000,
    "text": "has access to a search tool. Uh you give"
  },
  {
    "startMs": 968000,
    "text": "it a question. It's allowed to look up"
  },
  {
    "startMs": 970560,
    "text": "on the web as many times as it wants."
  },
  {
    "startMs": 972720,
    "text": "And once it knows that it has all the"
  },
  {
    "startMs": 974399,
    "text": "data, it passes the findings on to the"
  },
  {
    "startMs": 976320,
    "text": "second agent which is a summarizer which"
  },
  {
    "startMs": 977839,
    "text": "creates creates a research report."
  },
  {
    "startMs": 979920,
    "text": "Right? So with that out of the way,"
  },
  {
    "startMs": 981839,
    "text": "let's just quickly walk over to the"
  },
  {
    "startMs": 983680,
    "text": "demo. So for the demo, we have three"
  },
  {
    "startMs": 985600,
    "text": "different scenarios that we're going to"
  },
  {
    "startMs": 986800,
    "text": "talk about. For the first one, we're"
  },
  {
    "startMs": 988560,
    "text": "going to run the token ops in what we"
  },
  {
    "startMs": 990399,
    "text": "call preview mode. So in preview mode,"
  },
  {
    "startMs": 992720,
    "text": "what happens is that all the policies"
  },
  {
    "startMs": 994320,
    "text": "run as is, but the enforcement doesn't"
  },
  {
    "startMs": 996480,
    "text": "happen. So if you see we ran a"
  },
  {
    "startMs": 999120,
    "text": "particular run over here which completed"
  },
  {
    "startMs": 1002800,
    "text": "but we did not see any sort of failures"
  },
  {
    "startMs": 1005839,
    "text": "there. The policies executed but the"
  },
  {
    "startMs": 1008800,
    "text": "actions that were associated with those"
  },
  {
    "startMs": 1010560,
    "text": "policies were not allowed to be"
  },
  {
    "startMs": 1011759,
    "text": "executed. So we're just going to load"
  },
  {
    "startMs": 1013839,
    "text": "the dashboard screen here."
  },
  {
    "startMs": 1017199,
    "text": "Yeah. So this is the governance output."
  },
  {
    "startMs": 1020000,
    "text": "Governance is off. The run completed."
  },
  {
    "startMs": 1022240,
    "text": "But in the dashboard you can see the"
  },
  {
    "startMs": 1023600,
    "text": "policies have executed. So you can see"
  },
  {
    "startMs": 1024959,
    "text": "the cost budget, the cost guard and so"
  },
  {
    "startMs": 1026959,
    "text": "on. Right? So this was the first"
  },
  {
    "startMs": 1028799,
    "text": "scenario. For the second scenario, what"
  },
  {
    "startMs": 1030480,
    "text": "we're going to do is we're going to turn"
  },
  {
    "startMs": 1031839,
    "text": "on the governance. Now while that is"
  },
  {
    "startMs": 1034079,
    "text": "happening, I just want to touch upon why"
  },
  {
    "startMs": 1035439,
    "text": "this is important. So if you want to"
  },
  {
    "startMs": 1037679,
    "text": "like include this product into your"
  },
  {
    "startMs": 1039199,
    "text": "production agents, you want to have a"
  },
  {
    "startMs": 1041039,
    "text": "safe environment or a safe way to"
  },
  {
    "startMs": 1042559,
    "text": "firstly put it in your production"
  },
  {
    "startMs": 1044079,
    "text": "environment, test the guardrails, tweak"
  },
  {
    "startMs": 1045839,
    "text": "the guardrail, see what's the policies"
  },
  {
    "startMs": 1047280,
    "text": "are doing and then finalize the"
  },
  {
    "startMs": 1050320,
    "text": "thresholds. Right? So this is the second"
  },
  {
    "startMs": 1052400,
    "text": "one where we have now enforced the"
  },
  {
    "startMs": 1054160,
    "text": "governance and you can see in the"
  },
  {
    "startMs": 1056080,
    "text": "dashboard that the pre-all cost cap has"
  },
  {
    "startMs": 1059200,
    "text": "exceeded. So you had a budget allotted"
  },
  {
    "startMs": 1061360,
    "text": "for this run but the agent exceeded the"
  },
  {
    "startMs": 1064160,
    "text": "budget and it was killed immediately. So"
  },
  {
    "startMs": 1065840,
    "text": "that's the simple circuit breaker sort"
  },
  {
    "startMs": 1067840,
    "text": "of a methodology. So this is the halt"
  },
  {
    "startMs": 1069919,
    "text": "behavior. And now let's see the steer"
  },
  {
    "startMs": 1072799,
    "text": "behavior which is the which is where we"
  },
  {
    "startMs": 1075200,
    "text": "are trying to add value to this entire"
  },
  {
    "startMs": 1077120,
    "text": "cost management scenario. So this time"
  },
  {
    "startMs": 1079280,
    "text": "we're going to run the third the second"
  },
  {
    "startMs": 1080640,
    "text": "prompt. The budget allotted for this one"
  },
  {
    "startMs": 1083200,
    "text": "is slightly higher but it's still not"
  },
  {
    "startMs": 1084799,
    "text": "high enough for the agent to complete in"
  },
  {
    "startMs": 1087200,
    "text": "time. So what instead happens is there"
  },
  {
    "startMs": 1089760,
    "text": "is something called cost guard which"
  },
  {
    "startMs": 1091360,
    "text": "kicks in. This cost guard it takes into"
  },
  {
    "startMs": 1093679,
    "text": "account two things. First how much of"
  },
  {
    "startMs": 1095919,
    "text": "your allotted budget have you consumed?"
  },
  {
    "startMs": 1097919,
    "text": "Second what is the velocity at which"
  },
  {
    "startMs": 1099360,
    "text": "you're consuming tokens. [music] Now"
  },
  {
    "startMs": 1100880,
    "text": "based on these two things if it predicts"
  },
  {
    "startMs": 1102799,
    "text": "that you're going to run out of your"
  },
  {
    "startMs": 1104320,
    "text": "tokens or your allotted budget by the"
  },
  {
    "startMs": 1106240,
    "text": "end of the run it's going to inject"
  },
  {
    "startMs": 1108880,
    "text": "something into your system instructions"
  },
  {
    "startMs": 1110720,
    "text": "that something could be as simple as hey"
  },
  {
    "startMs": 1113200,
    "text": "you're running out of budget so make"
  },
  {
    "startMs": 1115360,
    "text": "sure that the LM outputs are more"
  },
  {
    "startMs": 1116799,
    "text": "succinct or more summarized right so"
  },
  {
    "startMs": 1119679,
    "text": "that is the way we are doing the"
  },
  {
    "startMs": 1121840,
    "text": "steering now the this was a very simple"
  },
  {
    "startMs": 1124400,
    "text": "test bench to show you like how this"
  },
  {
    "startMs": 1126799,
    "text": "works on a like working code we have"
  },
  {
    "startMs": 1129600,
    "text": "also benchmarked it on a couple of open"
  },
  {
    "startMs": 1131679,
    "text": "source repos. So we have benchmarked it"
  },
  {
    "startMs": 1133520,
    "text": "on browser use as well as metagp. Uh we"
  },
  {
    "startMs": 1136320,
    "text": "ran it across multiple iterations across"
  },
  {
    "startMs": 1138400,
    "text": "stress tests across simple scenarios"
  },
  {
    "startMs": 1140080,
    "text": "hard scenarios and everything. And the"
  },
  {
    "startMs": 1141760,
    "text": "results we see are the average spend"
  },
  {
    "startMs": 1143840,
    "text": "goes down by almost 78% with token ops"
  },
  {
    "startMs": 1146080,
    "text": "enabled with the full policy suit that"
  },
  {
    "startMs": 1147600,
    "text": "we have today. On the completion part"
  },
  {
    "startMs": 1150400,
    "text": "when we compare it with throttling just"
  },
  {
    "startMs": 1152799,
    "text": "simple throttling your simple throttling"
  },
  {
    "startMs": 1154799,
    "text": "is going to kill your agent runs no"
  },
  {
    "startMs": 1156160,
    "text": "matter what. Right? So with the reduced"
  },
  {
    "startMs": 1158320,
    "text": "average spend what you get is you get an"
  },
  {
    "startMs": 1160400,
    "text": "uplift in that completion percentage"
  },
  {
    "startMs": 1162160,
    "text": "from 67% to roughly 96%. So that is the"
  },
  {
    "startMs": 1165520,
    "text": "value ad that token ops is doing here."
  },
  {
    "startMs": 1167520,
    "text": "Now this is the policy catalog that we"
  },
  {
    "startMs": 1169679,
    "text": "run this benchmark against. This is what"
  },
  {
    "startMs": 1171200,
    "text": "we support today. We kind of researched"
  },
  {
    "startMs": 1173600,
    "text": "what are the different failure modes"
  },
  {
    "startMs": 1174799,
    "text": "that are there today out in the wild and"
  },
  {
    "startMs": 1176799,
    "text": "tried to cover most of them here. So you"
  },
  {
    "startMs": 1178880,
    "text": "have things across spend management, you"
  },
  {
    "startMs": 1180480,
    "text": "have things across context management"
  },
  {
    "startMs": 1181840,
    "text": "like context compaction, tool output"
  },
  {
    "startMs": 1183360,
    "text": "reduction, you have things across loop"
  },
  {
    "startMs": 1185280,
    "text": "detection and progress detection and"
  },
  {
    "startMs": 1186799,
    "text": "stuff like that. So this is the entire"
  },
  {
    "startMs": 1188480,
    "text": "set of policies that we support. And at"
  },
  {
    "startMs": 1190320,
    "text": "the bottom you can see the actions. So"
  },
  {
    "startMs": 1191919,
    "text": "as I mentioned earlier, we have two"
  },
  {
    "startMs": 1193280,
    "text": "flavors. You have the uh the halt type"
  },
  {
    "startMs": 1196480,
    "text": "actions and then the steer type actions."
  },
  {
    "startMs": 1198320,
    "text": "So for the steer we can do allow,"
  },
  {
    "startMs": 1200240,
    "text": "mutate, inject and so on. And for the"
  },
  {
    "startMs": 1202000,
    "text": "halt, it can be a simple kill. But this"
  },
  {
    "startMs": 1204320,
    "text": "is not the end state that we envision"
  },
  {
    "startMs": 1205919,
    "text": "for this. The end state is we have a lot"
  },
  {
    "startMs": 1209200,
    "text": "of data right we have a ledger that is"
  },
  {
    "startMs": 1211200,
    "text": "continuously being updated. So what we"
  },
  {
    "startMs": 1213120,
    "text": "want to try is we want to try a"
  },
  {
    "startMs": 1215360,
    "text": "self-learning module within the token"
  },
  {
    "startMs": 1217520,
    "text": "ops plane within the control plane which"
  },
  {
    "startMs": 1219520,
    "text": "can look at this ledger and ask this"
  },
  {
    "startMs": 1221120,
    "text": "question hey why or what is the failure"
  },
  {
    "startMs": 1224559,
    "text": "mode that I'm still not able to catch"
  },
  {
    "startMs": 1226559,
    "text": "and then based on that it can do two"
  },
  {
    "startMs": 1228559,
    "text": "things one is it can enhance it can"
  },
  {
    "startMs": 1230720,
    "text": "generate new policies on the fly based"
  },
  {
    "startMs": 1233039,
    "text": "on the missing or the still uh runaway"
  },
  {
    "startMs": 1236320,
    "text": "costs or it can refine the existing"
  },
  {
    "startMs": 1239200,
    "text": "parameters for the existing policies"
  },
  {
    "startMs": 1240799,
    "text": "that are there so that the runaway costs"
  },
  {
    "startMs": 1243440,
    "text": "are managed more effectively in the"
  },
  {
    "startMs": 1244799,
    "text": "future. So with that I think uh that is"
  },
  {
    "startMs": 1247919,
    "text": "all we have for you guys today. Thank"
  },
  {
    "startMs": 1249520,
    "text": "you so much for your time and you can"
  },
  {
    "startMs": 1251120,
    "text": "scan this QR code that's the public"
  },
  {
    "startMs": 1252799,
    "text": "wiki. We are updating it almost"
  },
  {
    "startMs": 1254480,
    "text": "regularly. So you can scan this and stay"
  },
  {
    "startMs": 1257280,
    "text": "up to date and uh Tisha and I are around"
  },
  {
    "startMs": 1259360,
    "text": "so if you guys have any questions or if"
  },
  {
    "startMs": 1260880,
    "text": "you want to discuss more about it just"
  },
  {
    "startMs": 1262559,
    "text": "let us know. That's it. Thank you."
  },
  {
    "startMs": 1265474,
    "text": "[applause]"
  }
]