[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 13680,
    "text": ">> Right. Hello, everybody. Um welcome to"
  },
  {
    "startMs": 16320,
    "text": "our talk. My name is Roberto Milev. I am"
  },
  {
    "startMs": 19040,
    "text": "the chief architect at Navan."
  },
  {
    "startMs": 21320,
    "text": "And I have Uday here, who's also part of"
  },
  {
    "startMs": 23160,
    "text": "the architecture team."
  },
  {
    "startMs": 25360,
    "text": "Uh Navan is a travel and expense"
  },
  {
    "startMs": 28440,
    "text": "management company."
  },
  {
    "startMs": 30160,
    "text": "And we'll share with you some of our"
  },
  {
    "startMs": 31840,
    "text": "learnings around how you run an AI and"
  },
  {
    "startMs": 34360,
    "text": "what have we"
  },
  {
    "startMs": 35800,
    "text": "uh discovered."
  },
  {
    "startMs": 37920,
    "text": "So,"
  },
  {
    "startMs": 39120,
    "text": "uh if you've been long enough um in this"
  },
  {
    "startMs": 42200,
    "text": "industry, you remember that"
  },
  {
    "startMs": 44360,
    "text": "over time there are a few paradigm"
  },
  {
    "startMs": 46040,
    "text": "shifts."
  },
  {
    "startMs": 47320,
    "text": "And we all tend to jump on a bandwagon"
  },
  {
    "startMs": 49720,
    "text": "and try to uh kind of do things, all"
  },
  {
    "startMs": 52120,
    "text": "right?"
  },
  {
    "startMs": 53280,
    "text": "Last time was when we all jumped on the"
  },
  {
    "startMs": 56200,
    "text": "microservices bandwagon. And out of"
  },
  {
    "startMs": 58680,
    "text": "that, a lot of good things came out,"
  },
  {
    "startMs": 60560,
    "text": "like container orchestration,"
  },
  {
    "startMs": 62880,
    "text": "Kubernetes."
  },
  {
    "startMs": 64480,
    "text": "Then we had service mesh,"
  },
  {
    "startMs": 67640,
    "text": "uh circuit breakers, all of those good"
  },
  {
    "startMs": 70000,
    "text": "things."
  },
  {
    "startMs": 70960,
    "text": "But it didn't happen overnight. Like it"
  },
  {
    "startMs": 72960,
    "text": "took a long time. It took some time for"
  },
  {
    "startMs": 75720,
    "text": "us to learn how to do these things."
  },
  {
    "startMs": 78160,
    "text": "So, one of the quotes from there is, \"If"
  },
  {
    "startMs": 80280,
    "text": "you can't build a well-structured"
  },
  {
    "startMs": 81760,
    "text": "monolith, why even try to build"
  },
  {
    "startMs": 83440,
    "text": "microservices?\""
  },
  {
    "startMs": 85160,
    "text": "Uh it kind of translates today because"
  },
  {
    "startMs": 88200,
    "text": "if you can't build a single agentic"
  },
  {
    "startMs": 90320,
    "text": "loop, why go in and try to build a"
  },
  {
    "startMs": 93320,
    "text": "multi-agent orchestrated system?"
  },
  {
    "startMs": 97880,
    "text": "So, over time, just like previously,"
  },
  {
    "startMs": 101680,
    "text": "uh a reference architecture is emerging."
  },
  {
    "startMs": 104280,
    "text": "So, we"
  },
  {
    "startMs": 106600,
    "text": "have learned a few things by by doing in"
  },
  {
    "startMs": 109840,
    "text": "production. We have a"
  },
  {
    "startMs": 111840,
    "text": "lot of agents, a lot of tokens per day"
  },
  {
    "startMs": 113880,
    "text": "being used."
  },
  {
    "startMs": 115280,
    "text": "And as I said, there are few layers that"
  },
  {
    "startMs": 118720,
    "text": "have standardized, that have"
  },
  {
    "startMs": 119920,
    "text": "crystallized around what do we need to"
  },
  {
    "startMs": 122720,
    "text": "run agentic flows"
  },
  {
    "startMs": 125200,
    "text": "reliably in production."
  },
  {
    "startMs": 127920,
    "text": "Runtime memory, context management, all"
  },
  {
    "startMs": 131240,
    "text": "around operational"
  },
  {
    "startMs": 132840,
    "text": "cross-cutting concerns, and around"
  },
  {
    "startMs": 134640,
    "text": "orchestration as well. So, today we'll"
  },
  {
    "startMs": 136640,
    "text": "go over some of these layers, all of"
  },
  {
    "startMs": 138480,
    "text": "these layers actually, and we will show"
  },
  {
    "startMs": 142040,
    "text": "kind of where the industry is, what we"
  },
  {
    "startMs": 144480,
    "text": "have done, what we have learned, and and"
  },
  {
    "startMs": 146880,
    "text": "so on."
  },
  {
    "startMs": 148560,
    "text": "So, starting at the runtime layer,"
  },
  {
    "startMs": 152280,
    "text": "we've talked a lot and we've built a lot"
  },
  {
    "startMs": 154360,
    "text": "of services"
  },
  {
    "startMs": 155840,
    "text": "in order to scale them"
  },
  {
    "startMs": 158000,
    "text": "statelessly"
  },
  {
    "startMs": 159480,
    "text": "before."
  },
  {
    "startMs": 160680,
    "text": "And now we're in a new world where, you"
  },
  {
    "startMs": 162800,
    "text": "know, agents are stateful by nature."
  },
  {
    "startMs": 164880,
    "text": "They need to have persistent sessions."
  },
  {
    "startMs": 168440,
    "text": "They need to have isolation. Their life"
  },
  {
    "startMs": 171280,
    "text": "cycle is different than the life cycle"
  },
  {
    "startMs": 173160,
    "text": "of a traditional API service, and so on."
  },
  {
    "startMs": 177560,
    "text": "So,"
  },
  {
    "startMs": 178920,
    "text": "the cloud providers have jumped in and"
  },
  {
    "startMs": 181360,
    "text": "try to fill this gap."
  },
  {
    "startMs": 183680,
    "text": "Um, you know, AWS, GCP, Azure, they all"
  },
  {
    "startMs": 188560,
    "text": "have a"
  },
  {
    "startMs": 190040,
    "text": "some incarnation of a agentic runtime."
  },
  {
    "startMs": 193160,
    "text": "If you scan the QR code for this slide"
  },
  {
    "startMs": 195200,
    "text": "and for the following slides,"
  },
  {
    "startMs": 197040,
    "text": "you will see a comparison of some of the"
  },
  {
    "startMs": 199040,
    "text": "features and how different cloud"
  },
  {
    "startMs": 200600,
    "text": "providers try to try to"
  },
  {
    "startMs": 203720,
    "text": "approach this."
  },
  {
    "startMs": 205120,
    "text": "At Nvono, we run everything on AWS. AWS"
  },
  {
    "startMs": 209200,
    "text": "has an agent core runtime. We heavily"
  },
  {
    "startMs": 211760,
    "text": "use that, but we have filled some gaps"
  },
  {
    "startMs": 213840,
    "text": "around that, like the"
  },
  {
    "startMs": 216120,
    "text": "session persistence and rehydration"
  },
  {
    "startMs": 219440,
    "text": "is something that we have built."
  },
  {
    "startMs": 221480,
    "text": "And we also run a bunch of other"
  },
  {
    "startMs": 224600,
    "text": "bunch of SDKs for writing agents. And"
  },
  {
    "startMs": 227480,
    "text": "part of"
  },
  {
    "startMs": 228800,
    "text": "these runtimes is typically they are"
  },
  {
    "startMs": 230320,
    "text": "framework agnostic, although they all"
  },
  {
    "startMs": 232000,
    "text": "prefer their"
  },
  {
    "startMs": 233480,
    "text": "native framework in a way."
  },
  {
    "startMs": 237640,
    "text": "Um the next"
  },
  {
    "startMs": 239800,
    "text": "layer in the stack is around memory."
  },
  {
    "startMs": 243120,
    "text": "Um"
  },
  {
    "startMs": 244800,
    "text": "we started with rag. Rag was kind of a"
  },
  {
    "startMs": 246840,
    "text": "big thing for a while. We were"
  },
  {
    "startMs": 250320,
    "text": "kind of driven to that out of necessity"
  },
  {
    "startMs": 252000,
    "text": "because you cannot fit an unlimited"
  },
  {
    "startMs": 255000,
    "text": "amount of context into an agent."
  },
  {
    "startMs": 258120,
    "text": "And over time um all of these cloud"
  },
  {
    "startMs": 261560,
    "text": "providers and the industry has"
  },
  {
    "startMs": 263840,
    "text": "implemented a pipeline where memory is"
  },
  {
    "startMs": 266919,
    "text": "kind of automatically generated by"
  },
  {
    "startMs": 268720,
    "text": "following a workflow of ingestion,"
  },
  {
    "startMs": 271200,
    "text": "extraction, and then consolidation and"
  },
  {
    "startMs": 273600,
    "text": "retrieval."
  },
  {
    "startMs": 274919,
    "text": "And there are parts of rag that are"
  },
  {
    "startMs": 277440,
    "text": "built in things like a long-term memory"
  },
  {
    "startMs": 280000,
    "text": "that inherently has some semantic"
  },
  {
    "startMs": 282240,
    "text": "characteristics. But memory is built up"
  },
  {
    "startMs": 284720,
    "text": "over time from short-term conversational"
  },
  {
    "startMs": 287080,
    "text": "memory"
  },
  {
    "startMs": 288200,
    "text": "to long-term memory that you kind of"
  },
  {
    "startMs": 290000,
    "text": "manage yourself. Uh then episodic"
  },
  {
    "startMs": 292640,
    "text": "memories about kind of instances that"
  },
  {
    "startMs": 295600,
    "text": "worked well and didn't work well."
  },
  {
    "startMs": 297760,
    "text": "Uh and so on. We at Navan again being a"
  },
  {
    "startMs": 301600,
    "text": "AWS shop, um"
  },
  {
    "startMs": 303880,
    "text": "utilize their agent core memory."
  },
  {
    "startMs": 306360,
    "text": "But we are also kind of doing it in a"
  },
  {
    "startMs": 307840,
    "text": "way that"
  },
  {
    "startMs": 309400,
    "text": "uh matches matches our our use case."
  },
  {
    "startMs": 314000,
    "text": "And then the next thing is context"
  },
  {
    "startMs": 316360,
    "text": "management. You know, it's a hot topic."
  },
  {
    "startMs": 318040,
    "text": "It was a hot topic and it's still a hot"
  },
  {
    "startMs": 319560,
    "text": "topic. Context windows are growing"
  },
  {
    "startMs": 321760,
    "text": "bigger, but there's never enough context"
  },
  {
    "startMs": 324320,
    "text": "or if there is too much context again,"
  },
  {
    "startMs": 327160,
    "text": "agents struggle with that cuz you lose"
  },
  {
    "startMs": 328840,
    "text": "focus and so on. Um"
  },
  {
    "startMs": 332400,
    "text": "what we found working is that"
  },
  {
    "startMs": 335840,
    "text": "uh focusing on skills as a unit of"
  },
  {
    "startMs": 338360,
    "text": "context. And I'll explain what I mean by"
  },
  {
    "startMs": 339760,
    "text": "that."
  },
  {
    "startMs": 340800,
    "text": "Uh we look at skills as both having"
  },
  {
    "startMs": 343040,
    "text": "context, meaning instructions and uh"
  },
  {
    "startMs": 346320,
    "text": "setup about a certain domain or a task."
  },
  {
    "startMs": 349600,
    "text": "And there's also the"
  },
  {
    "startMs": 351640,
    "text": "the second part of the skill, which is"
  },
  {
    "startMs": 352960,
    "text": "the tool execution and you know, the"
  },
  {
    "startMs": 355080,
    "text": "agentic part."
  },
  {
    "startMs": 356600,
    "text": "And we compose context dynamically out"
  },
  {
    "startMs": 361120,
    "text": "of skills that we"
  },
  {
    "startMs": 363720,
    "text": "use as units of work that are"
  },
  {
    "startMs": 366400,
    "text": "pluggable, that we can test"
  },
  {
    "startMs": 367880,
    "text": "independently, and that we can reuse."
  },
  {
    "startMs": 371480,
    "text": "So, for example, when we are"
  },
  {
    "startMs": 374360,
    "text": "we have an agent, we have skills that"
  },
  {
    "startMs": 376200,
    "text": "are"
  },
  {
    "startMs": 377080,
    "text": "that are specific to a domain."
  },
  {
    "startMs": 379320,
    "text": "And based on that, we compose them. And"
  },
  {
    "startMs": 381800,
    "text": "we rely on the"
  },
  {
    "startMs": 385000,
    "text": "you know, the progressive disclosure,"
  },
  {
    "startMs": 386800,
    "text": "which is a feature of the skills itself"
  },
  {
    "startMs": 389120,
    "text": "to start with a limited scope of context"
  },
  {
    "startMs": 391760,
    "text": "and then"
  },
  {
    "startMs": 392960,
    "text": "expand by included metadata"
  },
  {
    "startMs": 396120,
    "text": "further down the the line."
  },
  {
    "startMs": 399200,
    "text": "I'll hand it over to Uday Uday now to"
  },
  {
    "startMs": 402080,
    "text": "kind of walk us through the rest of"
  },
  {
    "startMs": 403560,
    "text": "this."
  },
  {
    "startMs": 404919,
    "text": ">> Thanks, Rudra."
  },
  {
    "startMs": 406600,
    "text": "All right. Can I have a quick show of"
  },
  {
    "startMs": 408320,
    "text": "hands"
  },
  {
    "startMs": 409560,
    "text": "here who have who had built an agent uh"
  },
  {
    "startMs": 412640,
    "text": "which failed halfway through multi 20 20"
  },
  {
    "startMs": 416000,
    "text": "step or 30 step process and be able to"
  },
  {
    "startMs": 419080,
    "text": "figure out quickly or reason about why"
  },
  {
    "startMs": 421320,
    "text": "the agent failed."
  },
  {
    "startMs": 425280,
    "text": "So, again, logs we've generally been"
  },
  {
    "startMs": 428480,
    "text": "traditionally with microservices, we all"
  },
  {
    "startMs": 430240,
    "text": "are familiar with logs. There's logs out"
  },
  {
    "startMs": 432280,
    "text": "there and then we go check out the logs."
  },
  {
    "startMs": 433960,
    "text": "But this changes everything the moment"
  },
  {
    "startMs": 436280,
    "text": "we switch to agents."
  },
  {
    "startMs": 437800,
    "text": "Agents"
  },
  {
    "startMs": 439040,
    "text": "output a lot of thinking. There's too"
  },
  {
    "startMs": 440880,
    "text": "much to consume. So, that's not the"
  },
  {
    "startMs": 442560,
    "text": "right way to do it, right? So,"
  },
  {
    "startMs": 444040,
    "text": "traditionally, that was the way, but our"
  },
  {
    "startMs": 446600,
    "text": "thought has to be changed right now."
  },
  {
    "startMs": 448720,
    "text": "In the in the way they"
  },
  {
    "startMs": 450680,
    "text": "Claude as an example, when we take"
  },
  {
    "startMs": 452520,
    "text": "Claude as an example for an agent,"
  },
  {
    "startMs": 454600,
    "text": "there is hooks and we can intercept"
  },
  {
    "startMs": 457120,
    "text": "everything that Claude as an agent that"
  },
  {
    "startMs": 458919,
    "text": "does at that level. So, what kind of"
  },
  {
    "startMs": 461160,
    "text": "tool it calls, right? What kind of"
  },
  {
    "startMs": 463560,
    "text": "decision it's making? So, before"
  },
  {
    "startMs": 465720,
    "text": "pre-tool and post-tool call or a"
  },
  {
    "startMs": 467320,
    "text": "pre-decision or a post-decision, so all"
  },
  {
    "startMs": 468960,
    "text": "of that are a"
  },
  {
    "startMs": 470760,
    "text": "point point in time for us to intercept"
  },
  {
    "startMs": 473160,
    "text": "and make a decision"
  },
  {
    "startMs": 474800,
    "text": "and either block"
  },
  {
    "startMs": 476960,
    "text": "to do a"
  },
  {
    "startMs": 478000,
    "text": "blocking operation or to log in metric"
  },
  {
    "startMs": 480520,
    "text": "or emit a metric, right? So, this is a"
  },
  {
    "startMs": 482720,
    "text": "critical"
  },
  {
    "startMs": 484280,
    "text": "place where we can emit auto traces."
  },
  {
    "startMs": 486840,
    "text": "At Nvone, we use one of our provider to"
  },
  {
    "startMs": 490160,
    "text": "interest to emit these auto traces and"
  },
  {
    "startMs": 493040,
    "text": "through these traces we should be able"
  },
  {
    "startMs": 494400,
    "text": "to figure out the spans, the traces and"
  },
  {
    "startMs": 497680,
    "text": "at what point in time where the agent is"
  },
  {
    "startMs": 499680,
    "text": "stuck, which gives much more confidence"
  },
  {
    "startMs": 502440,
    "text": "into"
  },
  {
    "startMs": 503640,
    "text": "how we operate and build the agent. This"
  },
  {
    "startMs": 506480,
    "text": "is"
  },
  {
    "startMs": 507360,
    "text": "day-to-day operational challenge."
  },
  {
    "startMs": 509520,
    "text": "Building agent these days there's so"
  },
  {
    "startMs": 511400,
    "text": "many frameworks, but how do you navigate"
  },
  {
    "startMs": 514400,
    "text": "building and operating an agent later is"
  },
  {
    "startMs": 516840,
    "text": "primary concern. Um"
  },
  {
    "startMs": 519039,
    "text": "And moreover, the reasoning chain, the"
  },
  {
    "startMs": 521640,
    "text": "thought process and critical signals"
  },
  {
    "startMs": 523880,
    "text": "that we emit"
  },
  {
    "startMs": 525160,
    "text": "here as part of the trace captures, we"
  },
  {
    "startMs": 527600,
    "text": "emit a few primary signals here. What is"
  },
  {
    "startMs": 530880,
    "text": "the current goal the agent is going"
  },
  {
    "startMs": 532560,
    "text": "through, the reasons behind its"
  },
  {
    "startMs": 534480,
    "text": "operations and the belief status and"
  },
  {
    "startMs": 536880,
    "text": "the tool calls that it's making. So,"
  },
  {
    "startMs": 538320,
    "text": "this kind of gives us a judgment"
  },
  {
    "startMs": 539920,
    "text": "pointers. Um"
  },
  {
    "startMs": 541360,
    "text": "Um in the traces. And when we make when"
  },
  {
    "startMs": 544720,
    "text": "the agent makes a decision, there is a"
  },
  {
    "startMs": 547440,
    "text": "confidence score, how confident it is"
  },
  {
    "startMs": 550000,
    "text": "when it makes the judgment, right? So,"
  },
  {
    "startMs": 552240,
    "text": "whether there are multiple paths that it"
  },
  {
    "startMs": 554000,
    "text": "leads to this choice"
  },
  {
    "startMs": 556600,
    "text": "or whether this is an inferred answer."
  },
  {
    "startMs": 558360,
    "text": "So, basically these are signals that"
  },
  {
    "startMs": 560040,
    "text": "gives us confidence later to review. If"
  },
  {
    "startMs": 563080,
    "text": "this is an inferred answer, there could"
  },
  {
    "startMs": 564600,
    "text": "be a human in the loop to guide through"
  },
  {
    "startMs": 566520,
    "text": "and tweak the agent to perform a little"
  },
  {
    "startMs": 568800,
    "text": "better."
  },
  {
    "startMs": 572480,
    "text": "Again, um"
  },
  {
    "startMs": 574440,
    "text": "Can I have a raise of hands again to see"
  },
  {
    "startMs": 576720,
    "text": "how confident are you like 100%"
  },
  {
    "startMs": 579560,
    "text": "confident in testing pipelines with your"
  },
  {
    "startMs": 581640,
    "text": "agents?"
  },
  {
    "startMs": 583720,
    "text": "Right. So, this is one of the other um"
  },
  {
    "startMs": 587000,
    "text": "critical aspect today. Um"
  },
  {
    "startMs": 591400,
    "text": "Because agents are non-deterministic."
  },
  {
    "startMs": 593080,
    "text": "We've all been used to program and write"
  },
  {
    "startMs": 595600,
    "text": "much more deterministic flows."
  },
  {
    "startMs": 597760,
    "text": "And"
  },
  {
    "startMs": 598880,
    "text": "we know how it works. The Can I ask an"
  },
  {
    "startMs": 601440,
    "text": "engineer? Engineer can come and tell me"
  },
  {
    "startMs": 602920,
    "text": "how this the algorithm, the sequence of"
  },
  {
    "startMs": 604520,
    "text": "operations. Everything is programmed in"
  },
  {
    "startMs": 606040,
    "text": "our mind. Everything is expectations."
  },
  {
    "startMs": 608080,
    "text": "But now the agents come into a"
  },
  {
    "startMs": 609600,
    "text": "non-deterministic way. And how do we"
  },
  {
    "startMs": 611360,
    "text": "test them, right? So, that is very"
  },
  {
    "startMs": 612520,
    "text": "criticality here."
  },
  {
    "startMs": 614000,
    "text": "And yeah. We are also struggling. Um"
  },
  {
    "startMs": 617320,
    "text": "we've uh started"
  },
  {
    "startMs": 619560,
    "text": "doing building agents. We the day to"
  },
  {
    "startMs": 621400,
    "text": "operations was challenging and then we"
  },
  {
    "startMs": 622640,
    "text": "failed in a lot of steps. How do we"
  },
  {
    "startMs": 624880,
    "text": "course correct? The moment we change"
  },
  {
    "startMs": 626080,
    "text": "something, something else broke breaks,"
  },
  {
    "startMs": 628800,
    "text": "right? So, how do we do that? Um one one"
  },
  {
    "startMs": 631440,
    "text": "approach that we took uh this is from um"
  },
  {
    "startMs": 633960,
    "text": "research papers uh around the"
  },
  {
    "startMs": 638440,
    "text": "in a multi-step"
  },
  {
    "startMs": 640400,
    "text": "uh orchestration, when an agent makes uh"
  },
  {
    "startMs": 642560,
    "text": "30 steps or decisions to make to reach"
  },
  {
    "startMs": 645000,
    "text": "to a goal,"
  },
  {
    "startMs": 646680,
    "text": "if that is a program or that's a"
  },
  {
    "startMs": 649200,
    "text": "different story. But this is not a"
  },
  {
    "startMs": 650320,
    "text": "program. This is non-deterministic way"
  },
  {
    "startMs": 652280,
    "text": "of It makes up its own steps every time"
  },
  {
    "startMs": 656600,
    "text": "uh differently."
  },
  {
    "startMs": 658320,
    "text": "So, how can we"
  },
  {
    "startMs": 660520,
    "text": "chart a deterministic graph here?"
  },
  {
    "startMs": 663240,
    "text": "Is it possible? No."
  },
  {
    "startMs": 665280,
    "text": "Can we have a trajectory of its starting"
  },
  {
    "startMs": 668560,
    "text": "from an end to a goal and then see how"
  },
  {
    "startMs": 670960,
    "text": "much how far it went in the trajectory"
  },
  {
    "startMs": 674200,
    "text": "and how far it went from the source to"
  },
  {
    "startMs": 675960,
    "text": "the destination is what we can compute"
  },
  {
    "startMs": 678839,
    "text": "to evaluate the efficiency or the"
  },
  {
    "startMs": 682000,
    "text": "completeness of the"
  },
  {
    "startMs": 683400,
    "text": "agent"
  },
  {
    "startMs": 684640,
    "text": "agent evaluation."
  },
  {
    "startMs": 686680,
    "text": "So, we we heavily rely on um trajectory"
  },
  {
    "startMs": 689040,
    "text": "vals um"
  },
  {
    "startMs": 690640,
    "text": "and uh"
  },
  {
    "startMs": 692400,
    "text": "this"
  },
  {
    "startMs": 693360,
    "text": "There are few other signals uh as I"
  },
  {
    "startMs": 695160,
    "text": "briefly spoke around uh in the previous"
  },
  {
    "startMs": 696760,
    "text": "slide around the inferred signal. Um"
  },
  {
    "startMs": 699200,
    "text": "If the answer is from an in in inferred"
  },
  {
    "startMs": 701600,
    "text": "answer, uh how can we uh"
  },
  {
    "startMs": 704720,
    "text": "loop that into uh and make a"
  },
  {
    "startMs": 708320,
    "text": "signals around uh how can we classify"
  },
  {
    "startMs": 710400,
    "text": "that this is a regression and make fixes"
  },
  {
    "startMs": 712840,
    "text": "towards the agent?"
  },
  {
    "startMs": 719120,
    "text": "Uh"
  },
  {
    "startMs": 723040,
    "text": "So, the next is the uh guardrails. Um"
  },
  {
    "startMs": 727480,
    "text": "Where"
  },
  {
    "startMs": 731760,
    "text": "Is this the one? Yeah."
  },
  {
    "startMs": 735200,
    "text": "So, guardrails and authorization, um"
  },
  {
    "startMs": 738320,
    "text": "this is uh critical"
  },
  {
    "startMs": 741240,
    "text": "displays a critical role in enterprise"
  },
  {
    "startMs": 742800,
    "text": "AI."
  },
  {
    "startMs": 744960,
    "text": "A lot of information is being piped to"
  },
  {
    "startMs": 747600,
    "text": "models. Um there could be sensitive"
  },
  {
    "startMs": 749920,
    "text": "information that goes into it uh without"
  },
  {
    "startMs": 752400,
    "text": "our knowledge. And"
  },
  {
    "startMs": 754560,
    "text": "we as uh uh leaders, how can we put in"
  },
  {
    "startMs": 756920,
    "text": "this governance layer um"
  },
  {
    "startMs": 759080,
    "text": "to stop this um is very uh critical"
  },
  {
    "startMs": 762400,
    "text": "here."
  },
  {
    "startMs": 763400,
    "text": "And and the concept of uh authentication"
  },
  {
    "startMs": 765600,
    "text": "and authorization um is"
  },
  {
    "startMs": 769040,
    "text": "taking up a different approach here. Um"
  },
  {
    "startMs": 771280,
    "text": "traditionally, we've seen um a user or a"
  },
  {
    "startMs": 774120,
    "text": "service account, but now what is an"
  },
  {
    "startMs": 776480,
    "text": "agent? Agent can be"
  },
  {
    "startMs": 778880,
    "text": "acting as on behalf of users. There is"
  },
  {
    "startMs": 780600,
    "text": "so much of things uh so many of use"
  },
  {
    "startMs": 782440,
    "text": "cases there. Hey, book me a flight"
  },
  {
    "startMs": 785120,
    "text": "whenever it's cheaper than $200, right?"
  },
  {
    "startMs": 787400,
    "text": "So, we just tell this assertion and then"
  },
  {
    "startMs": 789480,
    "text": "agent go figures out and does this"
  },
  {
    "startMs": 791400,
    "text": "action on behalf of me. So, is it me"
  },
  {
    "startMs": 793720,
    "text": "making this"
  },
  {
    "startMs": 794960,
    "text": "purchase or is it agent me making on"
  },
  {
    "startMs": 797000,
    "text": "behalf of me? So, there is"
  },
  {
    "startMs": 800000,
    "text": "Agent acts as a on behalf of user or"
  },
  {
    "startMs": 802360,
    "text": "agent uses a service account as well."
  },
  {
    "startMs": 804680,
    "text": "So,"
  },
  {
    "startMs": 805520,
    "text": "the line is being blurred here and we"
  },
  {
    "startMs": 807360,
    "text": "need to make fine-grained authorization"
  },
  {
    "startMs": 809160,
    "text": "decisions here, and the policy layer"
  },
  {
    "startMs": 812040,
    "text": "that's where the guardrails and"
  },
  {
    "startMs": 813080,
    "text": "authentication authorization plays a"
  },
  {
    "startMs": 814520,
    "text": "critical role."
  },
  {
    "startMs": 816080,
    "text": "And in the one what we employ here is"
  },
  {
    "startMs": 819320,
    "text": "before every tool call"
  },
  {
    "startMs": 821680,
    "text": "pre-tool and post-tool, we have this"
  },
  {
    "startMs": 823520,
    "text": "guardrails to check and block"
  },
  {
    "startMs": 826400,
    "text": "and make a informed decisions."
  },
  {
    "startMs": 832000,
    "text": "And"
  },
  {
    "startMs": 833280,
    "text": "this single agent versus multi-agent,"
  },
  {
    "startMs": 836040,
    "text": "again, this is kind of a"
  },
  {
    "startMs": 838040,
    "text": "orchestration wars you can think of"
  },
  {
    "startMs": 840400,
    "text": "with it to build a single agent or a"
  },
  {
    "startMs": 841840,
    "text": "multi-agent."
  },
  {
    "startMs": 843320,
    "text": "Again, as Roberto briefly hinted"
  },
  {
    "startMs": 845920,
    "text": "if you can't perfect and build a"
  },
  {
    "startMs": 848920,
    "text": "single agent, why go towards"
  },
  {
    "startMs": 851080,
    "text": "multi-agent, right? So, learn from our"
  },
  {
    "startMs": 853720,
    "text": "uh"
  },
  {
    "startMs": 854480,
    "text": "failures, experiences, and build towards"
  },
  {
    "startMs": 857000,
    "text": "that."
  },
  {
    "startMs": 858200,
    "text": "At Navan, yeah."
  },
  {
    "startMs": 860200,
    "text": "What the approach that we have taken is"
  },
  {
    "startMs": 862360,
    "text": "single master, and then we adopted"
  },
  {
    "startMs": 865720,
    "text": "sub-skills. Um"
  },
  {
    "startMs": 867560,
    "text": "There are sub-agents within it."
  },
  {
    "startMs": 869480,
    "text": "So, it's a single agent that can"
  },
  {
    "startMs": 872800,
    "text": "progressively load the skills and"
  },
  {
    "startMs": 874360,
    "text": "understand decisively what needs to be"
  },
  {
    "startMs": 877040,
    "text": "loaded into the context, and then"
  },
  {
    "startMs": 879240,
    "text": "make this"
  },
  {
    "startMs": 880600,
    "text": "navigation"
  },
  {
    "startMs": 881839,
    "text": "through the use case."
  },
  {
    "startMs": 884000,
    "text": "But there are"
  },
  {
    "startMs": 885560,
    "text": "other patterns that are also emerging."
  },
  {
    "startMs": 888320,
    "text": "There are different class of use cases"
  },
  {
    "startMs": 889920,
    "text": "here. One is um"
  },
  {
    "startMs": 892000,
    "text": "agent-to-agent communication. So, there"
  },
  {
    "startMs": 893440,
    "text": "are If you take a large scale"
  },
  {
    "startMs": 895160,
    "text": "organization, and there are so many of"
  },
  {
    "startMs": 896839,
    "text": "these teams that are"
  },
  {
    "startMs": 899040,
    "text": "that are acting as the boundaries, and"
  },
  {
    "startMs": 900560,
    "text": "they don't talk to each other, let's"
  },
  {
    "startMs": 901760,
    "text": "say."
  },
  {
    "startMs": 902680,
    "text": "How do we communicate? There are two"
  },
  {
    "startMs": 904080,
    "text": "agents on either of the side, right? How"
  },
  {
    "startMs": 906560,
    "text": "do we do it? So, there is A2A protocol"
  },
  {
    "startMs": 909560,
    "text": "which can help us establish the"
  },
  {
    "startMs": 912160,
    "text": "contracts in terms of skills."
  },
  {
    "startMs": 914360,
    "text": "And we can use A2A as a protocol there,"
  },
  {
    "startMs": 917320,
    "text": "which kind of"
  },
  {
    "startMs": 919000,
    "text": "is a boundary between the teams."
  },
  {
    "startMs": 922680,
    "text": "Yeah, over to you, Uday."
  },
  {
    "startMs": 929080,
    "text": ">> All right. So, as we went through the"
  },
  {
    "startMs": 930840,
    "text": "stack,"
  },
  {
    "startMs": 932040,
    "text": "it's obvious that um some components of"
  },
  {
    "startMs": 934920,
    "text": "the stack are in a more mature state and"
  },
  {
    "startMs": 938200,
    "text": "we already have good answers for them."
  },
  {
    "startMs": 940920,
    "text": "As Uday said, the runtime, I think it's"
  },
  {
    "startMs": 943000,
    "text": "pretty much solved. We are so advanced"
  },
  {
    "startMs": 944920,
    "text": "in orchestration and we are running LLMs"
  },
  {
    "startMs": 947360,
    "text": "in kind of uh a very"
  },
  {
    "startMs": 949720,
    "text": "uh brute-force way. So, scaling is not a"
  },
  {
    "startMs": 952240,
    "text": "not a problem. Also, memory, I think uh"
  },
  {
    "startMs": 955520,
    "text": "as"
  },
  {
    "startMs": 956880,
    "text": "uh the frontier LLMs get better and as"
  },
  {
    "startMs": 960160,
    "text": "our practices get better,"
  },
  {
    "startMs": 962080,
    "text": "we will uh find a way to cover the"
  },
  {
    "startMs": 965080,
    "text": "majority of the use cases and there is"
  },
  {
    "startMs": 966840,
    "text": "good maturity around the the cloud"
  },
  {
    "startMs": 969000,
    "text": "providers."
  },
  {
    "startMs": 970520,
    "text": "Uh MCP has emerged as the de facto"
  },
  {
    "startMs": 973840,
    "text": "protocol and tool calling is now a"
  },
  {
    "startMs": 976120,
    "text": "feature that everybody supports. So, we"
  },
  {
    "startMs": 978920,
    "text": "are seeing some industry convergence"
  },
  {
    "startMs": 981880,
    "text": "around that as well and MCP as a"
  },
  {
    "startMs": 984080,
    "text": "standard is also evolving. Now, it's"
  },
  {
    "startMs": 985839,
    "text": "becoming stateless. It's uh we are"
  },
  {
    "startMs": 988760,
    "text": "reaching a point where kind of we know"
  },
  {
    "startMs": 990520,
    "text": "how to invoke uh services and and and"
  },
  {
    "startMs": 993440,
    "text": "tools with agents."
  },
  {
    "startMs": 995520,
    "text": "Uh in some areas, things are happening,"
  },
  {
    "startMs": 999400,
    "text": "but you know, there's still a lot of"
  },
  {
    "startMs": 1000920,
    "text": "unknown. Around observability, there is"
  },
  {
    "startMs": 1003839,
    "text": "a push towards OTEL, but does OTEL"
  },
  {
    "startMs": 1006360,
    "text": "really work for agentic calls?"
  },
  {
    "startMs": 1009240,
    "text": "Uh yeah, you can make it work as Uday"
  },
  {
    "startMs": 1011320,
    "text": "was saying."
  },
  {
    "startMs": 1012680,
    "text": "Um also, we are getting more comfortable"
  },
  {
    "startMs": 1015440,
    "text": "around um around that the the the the"
  },
  {
    "startMs": 1018160,
    "text": "testing patterns. It's very hard to"
  },
  {
    "startMs": 1019880,
    "text": "test, but we have found a way to give"
  },
  {
    "startMs": 1022440,
    "text": "customers um quality experiences even"
  },
  {
    "startMs": 1025280,
    "text": "with the unreliability of agentic system"
  },
  {
    "startMs": 1027600,
    "text": "and I think that's kind of"
  },
  {
    "startMs": 1029360,
    "text": "uh getting in a in a state that is uh"
  },
  {
    "startMs": 1032520,
    "text": "that is more better defined."
  },
  {
    "startMs": 1035079,
    "text": "Orchestration is another one"
  },
  {
    "startMs": 1037400,
    "text": "um,"
  },
  {
    "startMs": 1038079,
    "text": "where, you know, we have a"
  },
  {
    "startMs": 1041480,
    "text": "uh, we have patterns, uh, we can build,"
  },
  {
    "startMs": 1044640,
    "text": "you know, bigger agents, smaller agents."
  },
  {
    "startMs": 1047520,
    "text": "Uh, as we said previously, probably the"
  },
  {
    "startMs": 1050240,
    "text": "right answer is to not over-engineer."
  },
  {
    "startMs": 1054040,
    "text": "Uh, so we're learning there and and and"
  },
  {
    "startMs": 1056680,
    "text": "uh,"
  },
  {
    "startMs": 1057600,
    "text": "a pattern of school thought is also"
  },
  {
    "startMs": 1060360,
    "text": "emerging. Uh, where we're all struggling"
  },
  {
    "startMs": 1063160,
    "text": "with and the previous talk was about"
  },
  {
    "startMs": 1065440,
    "text": "this for the developer, um,"
  },
  {
    "startMs": 1067800,
    "text": "AI assistant development perspective,"
  },
  {
    "startMs": 1069400,
    "text": "but also we're seeing these issues from"
  },
  {
    "startMs": 1072760,
    "text": "our production agents. It's very hard to"
  },
  {
    "startMs": 1075320,
    "text": "predict cost and it's very hard to"
  },
  {
    "startMs": 1077080,
    "text": "manage cost, uh, and put guardrails and"
  },
  {
    "startMs": 1081080,
    "text": "solve this in a way where there is"
  },
  {
    "startMs": 1082960,
    "text": "reliable, maybe fallback or have agents"
  },
  {
    "startMs": 1086280,
    "text": "be, uh, using cheaper models for certain"
  },
  {
    "startMs": 1090360,
    "text": "tasks. Uh,"
  },
  {
    "startMs": 1091800,
    "text": "uh, this is all driven by kind of the"
  },
  {
    "startMs": 1094160,
    "text": "big AI vendors who, I think, their"
  },
  {
    "startMs": 1097760,
    "text": "interest is for us all to spend more"
  },
  {
    "startMs": 1100320,
    "text": "tokens. Um, replay and debugging, Woody"
  },
  {
    "startMs": 1103760,
    "text": "talked about that, that's also a big big"
  },
  {
    "startMs": 1105880,
    "text": "issue. It's very hard"
  },
  {
    "startMs": 1108080,
    "text": "to understand, but I think"
  },
  {
    "startMs": 1110320,
    "text": "this is also something that that is"
  },
  {
    "startMs": 1112520,
    "text": "going to be solved because we can now"
  },
  {
    "startMs": 1115400,
    "text": "use, uh, agents to"
  },
  {
    "startMs": 1117480,
    "text": "uh, get over the cognitive overload of"
  },
  {
    "startMs": 1120200,
    "text": "trying to debug what they do."
  },
  {
    "startMs": 1123400,
    "text": "And then standards, um, standards are"
  },
  {
    "startMs": 1126360,
    "text": "emerging uh, by,"
  },
  {
    "startMs": 1129400,
    "text": "you know, the community. Uh, Hotel, as I"
  },
  {
    "startMs": 1131680,
    "text": "mentioned, agent to agent is young, it's"
  },
  {
    "startMs": 1133800,
    "text": "kind of pushed by certain vendors, but I"
  },
  {
    "startMs": 1136360,
    "text": "think over time we will we will get, uh,"
  },
  {
    "startMs": 1138920,
    "text": "there."
  },
  {
    "startMs": 1140520,
    "text": "Uh, with all of this said, you know, we"
  },
  {
    "startMs": 1143480,
    "text": "know what we need and it's up to us to"
  },
  {
    "startMs": 1146400,
    "text": "write and build it."
  },
  {
    "startMs": 1147760,
    "text": "Thank you, everybody."
  },
  {
    "startMs": 1149538,
    "text": ">> [applause]"
  },
  {
    "startMs": 1167000,
    "text": ">> I"
  }
]