# Agents Are Where Microservices Were in 2015 — Roberto Milev & Uday Kanagala, Navan Roberto Milev, Navan · Uday Kanagala, Navan AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/32nrHU6zHU8-agents-are-where-microservices-were-in-2015) · [Video](https://www.youtube.com/watch?v=32nrHU6zHU8) ## Agents Are Where Microservices Were in 2015 Roberto Milev and Uday Kanagala explain Navan’s emerging production-agent stack: persistent sessions, layered memory, progressively loaded skills, tool-call controls, trajectory evaluation, and fine-grained authorization—plus the costs and debugging problems that remain open. ### Start with one reliable agentic loop [0:13](https://www.youtube.com/watch?v=32nrHU6zHU8&t=13s) The talk opens with an architectural warning borrowed from the rise of microservices. That shift eventually produced useful machinery—Kubernetes, service meshes, circuit breakers, and container orchestration—but teams needed time to learn how to use it well. Splitting a poorly structured application into services did not fix its underlying design. Milev applies the same test to agents: establish a working single agentic loop before taking on the added complexity of multi-agent orchestration. ![Useful overview of the emerging production stack: runtime, memory, context management, operational concerns, and orchestration.](https://ai.engineer/api/talk-essay-media/0044ce7b835f35ae7c0ab17675b59973d2a1490cfda6043e5d5cf8369c2ece82) Useful overview of the emerging production stack: runtime, memory, context management, operational concerns, and orchestration. Navan’s production experience suggests that a reference architecture is nevertheless taking shape. The speakers organize it around runtime, memory, context management, cross-cutting operational controls, and orchestration. The rest of the talk walks up this stack, distinguishing components that have become usable from those that still need engineering work. ### Agents turn stateless infrastructure into a state-management problem [2:12](https://www.youtube.com/watch?v=32nrHU6zHU8&t=132s) Traditional API services often scale by keeping instances stateless. Agents invert that assumption: they carry an ongoing session, need isolation, and may live much longer than a request-response handler. A production runtime therefore has to preserve session state and restore it when execution moves or restarts. AWS, GCP, and Azure all offer agent-oriented runtimes, but adopting one does not necessarily finish the job. ![Illustrates the layered-memory progression from ingestion and extraction through consolidation and retrieval, including short-, long-, and episodic memory.](https://ai.engineer/api/talk-essay-media/f63462c7f4f9d5e5c289b008f26ef657f153a9f589195104691543128dd8d065) Illustrates the layered-memory progression from ingestion and extraction through consolidation and retrieval, including short-, long-, and episodic memory. Navan runs on AWS and uses its agent runtime while supplying its own session persistence and rehydration. Rehydration is the operationally important step: the runtime must reconstruct enough of an earlier session for work to continue rather than treating a recovered process as a fresh agent. The talk does not describe the stored representation, consistency model, or recovery procedure, so this is an architectural requirement rather than a complete implementation recipe. Navan also uses multiple agent SDKs; the speakers characterize managed runtimes as broadly framework-agnostic even when each provider favors its own framework. Memory starts from the same constraint as retrieval-augmented generation: an agent cannot carry unlimited information in its active context. The memory pipeline described here ingests material, extracts useful information, consolidates it, and later retrieves selected pieces. Retrieval is only the last stage; the system must first decide what deserves to become memory and how separate observations should be combined. The speakers distinguish three useful timescales: short-term conversational memory preserves the current interaction, managed long-term memory carries information across interactions, and episodic memory records examples of attempts that worked or failed. Navan uses its cloud provider’s memory service but adapts it to its own use case. No retention policy, consolidation algorithm, or measured quality gain is given, leaving memory selection and lifecycle management as implementation-specific work. ### Package context and execution together as skills [5:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=314s) Larger context windows do not remove context management. Too little context leaves an agent uninformed; too much can dilute the instructions and evidence that matter for the current task. Navan’s answer is to make a **skill** the unit of context rather than loading every domain’s material into every run. ![Introduces skills as the unit of context and explains why larger context windows do not eliminate context-management problems.](https://ai.engineer/api/talk-essay-media/ad7a1fa643dceff28d6c31cf85d61719ad30f0d02b4d53c63ac0ec3d1f468188) Introduces skills as the unit of context and explains why larger context windows do not eliminate context-management problems. A skill has two sides. It contains instructions and setup for a domain or task, and it contains the tool-execution capability used to perform that work. Navan composes an agent’s context dynamically from these packages, treating them as pluggable units that can be reused and tested independently. This moves context selection closer to software composition: a task receives the capability and instructions it needs without inheriting the whole system’s knowledge. Progressive disclosure keeps the initial context small. The agent begins with limited information about available skills, then follows included metadata to load more detail as the task develops. The tradeoff is explicit: selective loading protects focus, but the system must reliably recognize which skill to load and when. The talk says skills are independently testable, though it does not show the triggering tests or an evaluation of skill-selection accuracy. Progressive skill loading Navan composes task context from modular skills and expands it only as the agent needs more detail. - Current task — The agent begins with a domain or use case to navigate. - Limited skill context — Initial metadata identifies relevant capabilities without loading everything. - Skill instructions and setup — More domain or task context is disclosed when needed. - Tool execution — The skill also supplies the agentic execution component. - Dynamically composed context — The active context contains the selected reusable units of work. - Current task → Limited skill context (select relevant domain) - Limited skill context → Skill instructions and setup (disclose progressively) - Skill instructions and setup → Dynamically composed context (add instructions) - Tool execution → Dynamically composed context (add executable capability) ### Ordinary logs give way to structured tool-call traces [6:45](https://www.youtube.com/watch?v=32nrHU6zHU8&t=405s) Kanagala makes the operational problem concrete with an agent that fails halfway through a 20-step or 30-step process. Conventional logs may contain the evidence, but an agent can emit too much reasoning and intermediate activity for an engineer to consume efficiently. The useful question changes from “What did the process print?” to “At which action did its progress diverge?” ![Documents the structured trace signals Navan records, including the current goal, reasons, belief status, and tool calls.](https://ai.engineer/api/talk-essay-media/e2692020904322cb3619b1e06e5a6d42a00c8fe51089507e4d2aba2645d353f9) Documents the structured trace signals Navan records, including the current goal, reasons, belief status, and tool calls. Hooks at tool-call boundaries create structured interception points. Before or after a tool call, the surrounding system can block an operation, record an audit event, or emit a metric. Navan sends these events to Braintrust and examines traces and spans to locate where an agent became stuck. This is described as a day-two concern: frameworks make an initial agent relatively easy to build, while operating and diagnosing it later remains the harder job. Navan’s traces include the current goal, reasons associated with operations, belief status, and tool calls. Decisions also carry a confidence score and signals indicating whether an answer was inferred or supported through multiple paths. An inferred result can be routed to a human for guidance. The speakers do not explain how confidence is calculated, calibrated, or converted into an escalation threshold, so these fields should be understood as review signals rather than proven probabilities. Tool-call boundaries become control and observation points The same hooks can enforce policy, emit structured traces, and route uncertain decisions to a person. - Agent goal and decision — The trace records the current goal, reasons, belief status, and confidence-related signals. - Pre-tool hook — The surrounding system can inspect, block, or record the proposed operation. - Tool call — The agent invokes an external capability. - Post-tool hook — The system records the result and emits metrics or audit traces. - Trace and span — Operators inspect structured events to locate where execution became stuck. - Human review — An inferred or uncertain answer can be sent to a person for guidance. - Agent goal and decision → Pre-tool hook (propose action) - Pre-tool hook → Tool call (allow if acceptable) - Tool call → Post-tool hook (return result) - Pre-tool hook → Trace and span (emit proposed-action signal) - Post-tool hook → Trace and span (emit outcome signal) - Trace and span → Human review (escalate inferred answer) ### Evaluate progress without requiring an identical path [9:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=572s) Testing exposes the same nondeterminism from another direction. A conventional program has an expected sequence of operations that an engineer can explain. An agent may choose different intermediate steps on separate runs, and a change that fixes one failure can introduce another. An assertion over one final string therefore says little about whether the agent reached a goal efficiently or by an acceptable route. ![Introduces trajectory evaluations as Navan’s response to regressions and nondeterministic multi-step agent behavior.](https://ai.engineer/api/talk-essay-media/e40c78cceb4a0e8f57d8ab760f7c13bc2257a2705a5561f866a1c51264302249) Introduces trajectory evaluations as Navan’s response to regressions and nondeterministic multi-step agent behavior. Navan relies heavily on **trajectory evaluations** for multistep work. Instead of demanding one deterministic 30-step graph, an evaluation observes movement from the initial intent toward the destination. It can then assess how far the run progressed and use that progress to reason about completeness and efficiency while allowing the chosen path to vary. Trace signals connect diagnosis to evaluation. If a result depends on an inferred answer, that fact can help classify a behavioral change as a regression and guide the next fix. The talk does not provide a trajectory-distance formula, reference path, weighting scheme, or passing threshold. The mechanism is therefore a testing direction—score the path and its progress—not a fully specified evaluation standard. ### A delayed purchase blurs who is acting [12:03](https://www.youtube.com/watch?v=32nrHU6zHU8&t=723s) Guardrails begin with information governance: enterprise agents can send sensitive material to models, sometimes without sufficient awareness. They also expose a deeper identity problem. Traditional authorization usually recognizes a person or a service account. An agent may act for the former while using the latter, which makes the familiar identity model incomplete. ![Concise evidence that Navan places guardrails before and after every tool call to inspect, block, and inform decisions.](https://ai.engineer/api/talk-essay-media/13cff86fa5e0d60ff452520843c30b804f9f30f2098119bbba56582e1b7f58e3) Concise evidence that Navan places guardrails before and after every tool call to inspect, block, and inform decisions. The travel example makes the ambiguity visible: a user asks an agent to book a flight whenever the fare falls below $200. The user sets the condition now; the agent may decide to execute much later. Who made the eventual purchase—the user, the agent acting for the user, or a service identity used by the agent? The price threshold does not answer which credentials, permissions, or delegation policy should govern the transaction. Navan responds with fine-grained policy checks before and after every tool call. These guardrails can inspect and block actions where they cross into external systems, rather than relying on a single check at the beginning of a session. The talk identifies the enforcement points but does not specify the delegation token, purchase policy, revocation behavior, or treatment of a session whose authority changes before the delayed condition becomes true. ### Use one master agent until a real boundary demands another [13:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=832s) The opening microservices analogy returns when the speakers compare single-agent and multi-agent orchestration. Navan chose a single master agent that progressively loads sub-skills and decides what belongs in context for the current use case. The speakers also mention sub-agents within this arrangement, but do not define precisely where a skill ends and a sub-agent begins. The supported design principle is centralized navigation with modular capabilities, not an absolute ban on internal delegation. ![Shows Navan’s orchestration pattern: one master agent progressively loads sub-skills, with sub-agents inside the arrangement.](https://ai.engineer/api/talk-essay-media/e7d3d992c835d067a5f6ba84e8d43785cb0c1d2abf5d045de3a08fb11be6f48a) Shows Navan’s orchestration pattern: one master agent progressively loads sub-skills, with sub-agents inside the arrangement. Agent-to-agent communication becomes more compelling when it reflects an organizational boundary. Two teams may own separate agents and need an explicit way to communicate without collapsing their systems into one implementation. The speakers present A2A as an emerging protocol for expressing contracts in terms of skills across that boundary. This is a narrower justification for multiple agents than orchestration for its own sake: separate ownership creates the interface. ### The stack is crystallizing, but cost and replay remain open [15:29](https://www.youtube.com/watch?v=32nrHU6zHU8&t=929s) Milev closes by grading the stack unevenly. In his assessment, runtime and scaling are largely solved, while memory is maturing through better models, operating practices, and cloud services. He also sees tool calling and MCP as points of industry convergence. These are the speaker’s production judgments, not guarantees that managed runtimes or memory services cover every reliability requirement. ![Highlights the major unresolved operating issue: agent cost remains difficult to predict, manage, and constrain.](https://ai.engineer/api/talk-essay-media/f6143773898700d6784e02be861e241018808295316b56fe02fbc25c74bd08c2) Highlights the major unresolved operating issue: agent cost remains difficult to predict, manage, and constrain. Observability remains less settled. OpenTelemetry can be adapted to agentic calls, but the speakers question how naturally it represents them. Testing patterns are becoming more usable despite agent unreliability, and orchestration now offers recognizable large-agent and small-agent patterns. Their recommendation stays conservative: use the simplest arrangement that works and avoid overengineering while these patterns continue to change. Cost is the clearest unresolved production problem. Navan finds agent spending difficult to predict and control. Reliable fallbacks and routing selected tasks to cheaper models are possible directions, but the talk does not present either as a finished strategy. Milev also points out an incentive mismatch: major AI vendors benefit when customers consume more tokens, while application teams must make that consumption economically predictable. Replay and debugging remain difficult because engineers must reconstruct long, variable runs from large volumes of agent activity. The speakers suggest using agents to help analyze other agents’ traces and reduce that cognitive load, but no replay mechanism or measured debugging improvement is demonstrated. Standards such as OpenTelemetry and Agent2Agent are still evolving, with the latter described as young and influenced by particular vendors. The ending is pragmatic rather than triumphant: the industry increasingly knows which capabilities it needs, but engineers still have to build the dependable operating layer around them. ## Key takeaways - Establish a reliable single agentic loop before adding multi-agent orchestration; the speakers recommend avoiding that extra complexity until it solves a real need. - Production agents require persistent, isolated sessions and recovery through rehydration, even when a managed runtime supplies the basic execution environment. - Treat skills as reusable units that combine domain instructions with execution capabilities, then load them progressively to protect context focus. - Instrument pre- and post-tool boundaries so the system can block actions, emit structured traces, and route inferred decisions to human review. - For nondeterministic multistep agents, evaluate trajectory progress, efficiency, and completeness rather than requiring every successful run to follow an identical sequence. - Authorization must represent delegated action explicitly. A user’s earlier instruction does not by itself settle which identity and permissions govern a later purchase. - Runtime and tool invocation may be maturing, but predictable cost, replay, debugging, observability semantics, and agent-to-agent standards remain open work. ## Resources - [Agents Are Where Microservices Were in 2015 — official AI Engineer talk page](https://ai.engineer/talks/32nrHU6zHU8-youtube-0b98a6ac067ca831c5fe) — The recording’s official page with video playback, chapters, transcript, and a concise reading version. ## Related talks - [12-Factor Agents: Patterns of reliable LLM applications](https://ai.engineer/talks/8kMaTybvDUw-12-factor-agents) — A complementary production-oriented argument for keeping agent systems modular, controlling orchestration explicitly, and surrounding probabilistic model behavior with deterministic software. - [Don't Build Agents, Build Skills Instead](https://ai.engineer/talks/CEvIs9y1uog-agent-skills) — Develops the skill abstraction in greater depth, including progressive disclosure, reusable procedural knowledge, and the separation between runtime, connectivity, and expertise. - [How to Secure Agents using OAuth](https://ai.engineer/talks/blmAkayzE8M-secure-agents-using-oauth) — Extends the flight-purchase authorization problem with concrete delegated-access concepts including OAuth clients, scopes, access tokens, consent, and agent-specific security questions. ## Slides ### Agents Are Where Microservices Were in 2015 Roberto Milev and Uday Kanagala share Navan’s AI learnings, drawing a parallel between today’s agents and the mistakes made with microservices in 2015. [0:19](https://www.youtube.com/watch?v=32nrHU6zHU8&t=19s) [Title slide introducing a talk by Roberto Milev and Uday Kanagala.](https://assets.ai.engineer/sites/aiecode2025/sha256/8fdcc7670b81c4d7bd2e01a45b1f7e8921b92aeee1522300a0df4dd821c18a55.webp) Agents Are Where Microservices Were in 2015 We're making all the same mistakes. Navan ENGINEERING AI Engineer World's Fair 2026 | July 1 Roberto Milev Chief Architect @ Navan Uday Kanagala Principal Architect @ Navan 20 MIN · 12 SLIDES ### A Question About Microservices Simon Brown’s 2015 question challenges the leap from a poorly structured monolith to microservices. The speaker applies the same test to a single agentic loop and a multi-agent system. [0:36](https://www.youtube.com/watch?v=32nrHU6zHU8&t=36s) [A Simon Brown quotation questions whether microservices help when a monolith is poorly structured.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a5c25aebecf71d384243778fc83e9ead00178ec8b2b21ce345082e20b0dcbe9.webp) If you can't build a well-structured monolith, what makes you think microservices is the answer? Simon Brown 2015 NAVAN ### From Stateless by Design to Stateful by Necessity Agent runtimes isolate sessions and preserve state through restarts. Navan stores conversations in S3 and rehydrates them on resume, so session state follows the conversation rather than the instance. [2:27](https://www.youtube.com/watch?v=32nrHU6zHU8&t=147s) [Runtime capabilities include isolation, session persistence, lifecycle management, and framework independence.](https://assets.ai.engineer/sites/aiecode2025/sha256/c91f7319fc3eb9205adbbb8ddafa8ba0a7f0a76666423fc21347700c61e832e0.webp) 01 RUNTIME · LAYER 01 / 07 NAVAN From Stateless by Design to Stateful by Necessity The problem: agents need isolated, stateful, resumable execution environments. KEY CAPABILITIES Sandboxed isolation each session gets its own VM — no shared kernel, no cross-session leakage Session persistence filesystem state survives idle / restart cycles Lifecycle management auto-destruction after idle timeout, max lifetime caps Framework-agnostic bring any agent framework, any model CLOUD COMPARISON AWS · AZURE · GCP Scan for the full table Product · isolation · state · status for all three clouds. ↳ /agent-stack#runtime All shipped 2025–2026 — AWS & GCP GA, Azure preview. At NAVAN IN PRODUCTION • Conversations persisted to S3, hydrated back into runtime on resume • Session state follows the conversation, not the instance • Frameworks: Claude Agent SDK, OpenAI Agents SDK, LangGraph, Google ADK ### Memory Is Not Retrieval Memory moves from ingested events through extraction and consolidation to retrieval. Navan pairs short-term events with long-term summaries and lets agents decide what to retrieve. [3:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=236s) [A memory pipeline and four memory types support agents across sessions.](https://assets.ai.engineer/sites/aiecode2025/sha256/69fee6dbb694f3d8b85fa0b6bea01f0e9a86e0615e3faa4c84e82feddd6fb12e.webp) 02 MEMORY · LAYER 02 / 07 NAVAN Memory Is Not Retrieval The problem: agents need to remember across sessions — not just retrieve from static knowledge bases. THE PIPELINE — SAME ACROSS PROVIDERS 01 Ingest — raw events flow in 02 Extract — LLM picks what matters 03 Consolidate — merge, dedupe, resolve 04 Retrieve — semantic at query TIERED MEMORY Short-term · single session turns, tool results, working state Long-term · indefinite preferences, knowledge, decisions Episodic · cross-session what worked, what didn't — with reflections Procedural · cross-session how-to routines learned over time CLOUD COMPARISON AWS · AZURE · GCP Scan for the full table Products · memory types · scoping · status. ↳ /agent-stack#memory AWS & GCP GA · Azure preview. At NAVAN · IN PRODUCTION • RAG broke: top-K retrieval doesn't reason — it fetches, not investigates • Fix: tiered memory (short-term raw events + long-term summaries) + agent-driven retrieval decisions ### Skills Are the Units of Context Navan composes context from reusable skills: booking includes tools and knowledge, policy supplies knowledge, and payment stays unloaded. Progressive disclosure adds context as needed. [5:12](https://www.youtube.com/watch?v=32nrHU6zHU8&t=312s) [Context approaches and a diagram show dynamically selected skills for an agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/c81a612e447ae425f8d817da7c8cda45dd10cfceca2f8dadbfd618a02cd95370.webp) 03 SKILLS + CONTEXT · LAYER 03 / 07 NAVAN Skills Are the Units of Context The problem: what the agent knows at decision time determines everything. Too little context → wrong answers. Too much → confusion, cost, latency. COMMON APPROACHES 01 Retrieval pull relevant docs at query time (RAG) 02 Summarization compress prior context to fit the window 03 Compression / pruning drop low-value tokens 04 Structured prompting organize context for better attention At NAVAN SKILLS = KNOWLEDGE + TOOLS AGENT EXECUTION Booking [tools] [knowledge] Policy [knowledge] only Payment not loaded (pay-later) [illegible] skills reference each other • Dynamic selection — load skills at session start • Progressive disclosure — context revealed as needed • Testable, versionable, reusable, composable — not one giant prompt ### You Can't tail -f an Agent Navan traces tool calls alongside the agent’s goal, reasoning and belief state, and records confidence signals. Replay hydrates a conversation from S3 so it can resume. [6:38](https://www.youtube.com/watch?v=32nrHU6zHU8&t=398s) [Agent observability needs execution traces, reasoning, confidence signals and replay; Navan describes its production approach.](https://assets.ai.engineer/sites/aiecode2025/sha256/a1e760bc44ab3e60d5c5ede4ee0bf2d9aa3a81f5faa3b0e57f1a3d3c81c10d40.webp) 04 OBSERVABILITY · CROSS-CUTTING 04 / 07 NAVAN You Can't tail -f an Agent The problem: agent makes a bad decision on step 7 of 12 — and you find out from the user, not your monitoring. WHAT'S NEEDED Execution traces what did it do? Reasoning chains why did it do it? Confidence signals how sure was it? Replay capability re-run with different context? CLOUD COMPARISON AWS · AZURE · GCP Scan for the full table Tracing, reasoning visibility, OTel & status, all three clouds. ↳ /agent-stack#observability X-Ray · App Insights · Cloud Trace — none agent-native. At NAVAN IN PRODUCTION • OTel hooks on every tool call • Confidence per response: confident_answer | multiple_possible_paths | inferred_answer | answer_not_found • Traces capture current_goal · reasoning · belief_state · tool_call • Replay: conversation_id → S3 (UUIDv7) → hydrate & resume ### Testing: The Biggest Gap Agent paths vary between runs. Navan evaluates trajectories toward a goal alongside automated end-to-end tests, and uses production confidence signals to spot regressions. [9:29](https://www.youtube.com/watch?v=32nrHU6zHU8&t=569s) [Challenges in evaluating agents and Navan's automated and production evaluation practices.](https://assets.ai.engineer/sites/aiecode2025/sha256/f747cef48b737b440a94c4fb30d228102a99de96b455f3a7913fd61fc3741816.webp) Testing: The Biggest Gap 05 TESTING / EVALS · CROSS-CUTTING 05 / 07 NAVAN The problem: how do you know your agent got worse before users do? THE CHALLENGES Non-deterministic behavior — same input, different output Prompt changes have invisible blast radius No ground truth for open-ended agent behavior LLM-as-a-judge works but is expensive CLOUD COMPARISON AWS · AZURE · GCP Scan for the full table Evaluators, task completion, tool accuracy, CI/CD, status. ↳ /agent-stack#testing Azure & GCP ship agent evaluators; AWS agent-eval maturity unclear. At NAVAN IN PRODUCTION • E2E evals + trajectory evals, all automated • Skills as testable units: change one skill → do these 50 scenarios still pass? • Online scoring — production, not just pre-deploy • Confidence signals feed eval: spike in inferred_answer = regression ### What Is the Agent Allowed to Do? Agents may act on a user’s behalf or use a service account. Navan checks policy before tool calls and filters input and output; scoped credentials can allow flight booking without a $50k wire. [11:59](https://www.youtube.com/watch?v=32nrHU6zHU8&t=719s) [Agent authorization and safety capabilities include scoped credentials, budgets, least privilege, content safety and kill switches.](https://assets.ai.engineer/sites/aiecode2025/sha256/24ae8a063639a93782d6295babbfd59df535ff1be8b05e3720e47ff309f9c269.webp) What Is the Agent Allowed to Do? 06 GUARDRAILS / AUTH · CROSS-CUTTING 06 / 07 NAVAN The problem: the agent is a new kind of principal — not a user, not a service. It needs identity, scoped authorization, and enforcement. KEY CAPABILITIES NEEDED Scoped credentials — book a flight but not wire $50k Budget limits — token spend, action count, time limits Least-privilege — the IAM of agents Content safety — prevent harmful outputs, PII leakage Kill switches — halt a runaway agent CLOUD COMPARISON AWS · AZURE · GCP Scan for the full table Identity, policy enforcement & kill-switch, each cloud's approach. ↳ /agent-stack#guardrails The gap: no cloud has native agent-level budget / token controls — you build that yourself. At NAVAN IN PRODUCTION • Auth model: on-behalf-of-user OR service account (depends on use case) • Pre-tool-call policy checks — guardrail fires before the tool executes • Content filters on both input and output ### Single vs. Multi-Agent Agent handoffs can compound hallucinations and lose the original goal. Navan starts with one agent and domain subagents, splitting across domains when expertise separates. [13:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=829s) [Multi-agent failure modes and Navan's approach to agent orchestration.](https://assets.ai.engineer/sites/aiecode2025/sha256/2d06aa1bd4718035138441e2c47c196d3c7e7cb1d1da3204d7441189242ab25d.webp) Single vs. Multi-Agent 07 ORCHESTRATION · LAYER 07 / 07 NAVAN The problem: one agent or many? Every agent boundary = context lost + another inference call + another point of non-determinism. FAILURE MODES Cascading hallucinations Agent 1 hallucinates → Agent 2 builds on it → compounds Goal drift original intent lost across agent handoffs Degraded critique agents agreeing with each other instead of challenging CLOUD COMPARISON AWS · AZURE · GCP Scan for the full table Patterns & inter-agent protocol (A2A) across all three clouds. ↳ /agent-stack#orchestration GCP is the A2A originator (open standard, 50+ partners). At NAVAN IN PRODUCTION • Default: single agent with subagents per domain • MCP Gateway: User → OAuth → AgentCore Gateway → REST + internal MCP servers • Cross-domain: agent-as-MCP-tool (A2A protocol) • Lesson: start single + good tools; split only when domain expertise separates ### What's Still Painfully Missing Navan sees runtime and tools as mature, but identifies gaps in agent-level spend caps and the ability to replay decisions with different context. [15:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=928s) [Three columns categorize agent infrastructure capabilities as solved, emerging or missing.](https://assets.ai.engineer/sites/aiecode2025/sha256/afb29c1f7993c7b5cc78a47a8c8a3fcffdbabfef6a6714ee008aaf53e383e492.webp) What's Still Painfully Missing WHERE THE STACK STANDS SOLVED Runtime sandboxed, session-persistent, all clouds ship it Memory converging on the same pipeline Tools / MCP standard interface layer, gateway support across clouds EMERGING Observability traces exist, reasoning visibility improving, no standard yet Testing Azure / GCP shipping evaluators, but no consensus on approach Orchestration patterns clear, tooling experimental MISSING Cost & budget controls no circuit breakers for reasoning spirals, no agent-level spend caps Replay & debugging can't re-run a decision with different context Standards OTel GenAI experimental, A2A young, no universal trace format NAVAN ### Go Build the Rest The speakers close by urging the audience to build what is still needed. [19:00](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1140s) [Closing slide: most of what is needed exists; build the rest.](https://assets.ai.engineer/sites/aiecode2025/sha256/858889d280de9e74ccf91804eb0e52133d7612c2f5950b45cbe031de417a9794.webp) THE LANDING We know what's needed. Most of it exists. Go build the rest. Roberto Milev · Uday Kanagala — Architects, Navan AI Engineer World's Fair 2026 navan ### AI Engineer World's Fair [19:11](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1151s) [AI Engineer World's Fair title card on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/c6fa7775df656dc5c9d5c34bf222030664ed63e3195c567e2f71a99729ec7e9b.webp) AI Engineer World’s Fair HTTPS://AI.ENGINEER ## Full transcript [0:01](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1s) [music] [0:13](https://www.youtube.com/watch?v=32nrHU6zHU8&t=13s) >> Right. Hello, everybody. Um welcome to [0:16](https://www.youtube.com/watch?v=32nrHU6zHU8&t=16s) our talk. My name is Roberto Milev. I am [0:19](https://www.youtube.com/watch?v=32nrHU6zHU8&t=19s) the chief architect at Navan. [0:21](https://www.youtube.com/watch?v=32nrHU6zHU8&t=21s) And I have Uday here, who's also part of [0:23](https://www.youtube.com/watch?v=32nrHU6zHU8&t=23s) the architecture team. [0:25](https://www.youtube.com/watch?v=32nrHU6zHU8&t=25s) Uh Navan is a travel and expense [0:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=28s) management company. [0:30](https://www.youtube.com/watch?v=32nrHU6zHU8&t=30s) And we'll share with you some of our [0:31](https://www.youtube.com/watch?v=32nrHU6zHU8&t=31s) learnings around how you run an AI and [0:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=34s) what have we [0:35](https://www.youtube.com/watch?v=32nrHU6zHU8&t=35s) uh discovered. [0:37](https://www.youtube.com/watch?v=32nrHU6zHU8&t=37s) So, [0:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=39s) uh if you've been long enough um in this [0:42](https://www.youtube.com/watch?v=32nrHU6zHU8&t=42s) industry, you remember that [0:44](https://www.youtube.com/watch?v=32nrHU6zHU8&t=44s) over time there are a few paradigm [0:46](https://www.youtube.com/watch?v=32nrHU6zHU8&t=46s) shifts. [0:47](https://www.youtube.com/watch?v=32nrHU6zHU8&t=47s) And we all tend to jump on a bandwagon [0:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=49s) and try to uh kind of do things, all [0:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=52s) right? [0:53](https://www.youtube.com/watch?v=32nrHU6zHU8&t=53s) Last time was when we all jumped on the [0:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=56s) microservices bandwagon. And out of [0:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=58s) that, a lot of good things came out, [1:00](https://www.youtube.com/watch?v=32nrHU6zHU8&t=60s) like container orchestration, [1:02](https://www.youtube.com/watch?v=32nrHU6zHU8&t=62s) Kubernetes. [1:04](https://www.youtube.com/watch?v=32nrHU6zHU8&t=64s) Then we had service mesh, [1:07](https://www.youtube.com/watch?v=32nrHU6zHU8&t=67s) uh circuit breakers, all of those good [1:10](https://www.youtube.com/watch?v=32nrHU6zHU8&t=70s) things. [1:10](https://www.youtube.com/watch?v=32nrHU6zHU8&t=70s) But it didn't happen overnight. Like it [1:12](https://www.youtube.com/watch?v=32nrHU6zHU8&t=72s) took a long time. It took some time for [1:15](https://www.youtube.com/watch?v=32nrHU6zHU8&t=75s) us to learn how to do these things. [1:18](https://www.youtube.com/watch?v=32nrHU6zHU8&t=78s) So, one of the quotes from there is, "If [1:20](https://www.youtube.com/watch?v=32nrHU6zHU8&t=80s) you can't build a well-structured [1:21](https://www.youtube.com/watch?v=32nrHU6zHU8&t=81s) monolith, why even try to build [1:23](https://www.youtube.com/watch?v=32nrHU6zHU8&t=83s) microservices?" [1:25](https://www.youtube.com/watch?v=32nrHU6zHU8&t=85s) Uh it kind of translates today because [1:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=88s) if you can't build a single agentic [1:30](https://www.youtube.com/watch?v=32nrHU6zHU8&t=90s) loop, why go in and try to build a [1:33](https://www.youtube.com/watch?v=32nrHU6zHU8&t=93s) multi-agent orchestrated system? [1:37](https://www.youtube.com/watch?v=32nrHU6zHU8&t=97s) So, over time, just like previously, [1:41](https://www.youtube.com/watch?v=32nrHU6zHU8&t=101s) uh a reference architecture is emerging. [1:44](https://www.youtube.com/watch?v=32nrHU6zHU8&t=104s) So, we [1:46](https://www.youtube.com/watch?v=32nrHU6zHU8&t=106s) have learned a few things by by doing in [1:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=109s) production. We have a [1:51](https://www.youtube.com/watch?v=32nrHU6zHU8&t=111s) lot of agents, a lot of tokens per day [1:53](https://www.youtube.com/watch?v=32nrHU6zHU8&t=113s) being used. [1:55](https://www.youtube.com/watch?v=32nrHU6zHU8&t=115s) And as I said, there are few layers that [1:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=118s) have standardized, that have [1:59](https://www.youtube.com/watch?v=32nrHU6zHU8&t=119s) crystallized around what do we need to [2:02](https://www.youtube.com/watch?v=32nrHU6zHU8&t=122s) run agentic flows [2:05](https://www.youtube.com/watch?v=32nrHU6zHU8&t=125s) reliably in production. [2:07](https://www.youtube.com/watch?v=32nrHU6zHU8&t=127s) Runtime memory, context management, all [2:11](https://www.youtube.com/watch?v=32nrHU6zHU8&t=131s) around operational [2:12](https://www.youtube.com/watch?v=32nrHU6zHU8&t=132s) cross-cutting concerns, and around [2:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=134s) orchestration as well. So, today we'll [2:16](https://www.youtube.com/watch?v=32nrHU6zHU8&t=136s) go over some of these layers, all of [2:18](https://www.youtube.com/watch?v=32nrHU6zHU8&t=138s) these layers actually, and we will show [2:22](https://www.youtube.com/watch?v=32nrHU6zHU8&t=142s) kind of where the industry is, what we [2:24](https://www.youtube.com/watch?v=32nrHU6zHU8&t=144s) have done, what we have learned, and and [2:26](https://www.youtube.com/watch?v=32nrHU6zHU8&t=146s) so on. [2:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=148s) So, starting at the runtime layer, [2:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=152s) we've talked a lot and we've built a lot [2:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=154s) of services [2:35](https://www.youtube.com/watch?v=32nrHU6zHU8&t=155s) in order to scale them [2:38](https://www.youtube.com/watch?v=32nrHU6zHU8&t=158s) statelessly [2:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=159s) before. [2:40](https://www.youtube.com/watch?v=32nrHU6zHU8&t=160s) And now we're in a new world where, you [2:42](https://www.youtube.com/watch?v=32nrHU6zHU8&t=162s) know, agents are stateful by nature. [2:44](https://www.youtube.com/watch?v=32nrHU6zHU8&t=164s) They need to have persistent sessions. [2:48](https://www.youtube.com/watch?v=32nrHU6zHU8&t=168s) They need to have isolation. Their life [2:51](https://www.youtube.com/watch?v=32nrHU6zHU8&t=171s) cycle is different than the life cycle [2:53](https://www.youtube.com/watch?v=32nrHU6zHU8&t=173s) of a traditional API service, and so on. [2:57](https://www.youtube.com/watch?v=32nrHU6zHU8&t=177s) So, [2:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=178s) the cloud providers have jumped in and [3:01](https://www.youtube.com/watch?v=32nrHU6zHU8&t=181s) try to fill this gap. [3:03](https://www.youtube.com/watch?v=32nrHU6zHU8&t=183s) Um, you know, AWS, GCP, Azure, they all [3:08](https://www.youtube.com/watch?v=32nrHU6zHU8&t=188s) have a [3:10](https://www.youtube.com/watch?v=32nrHU6zHU8&t=190s) some incarnation of a agentic runtime. [3:13](https://www.youtube.com/watch?v=32nrHU6zHU8&t=193s) If you scan the QR code for this slide [3:15](https://www.youtube.com/watch?v=32nrHU6zHU8&t=195s) and for the following slides, [3:17](https://www.youtube.com/watch?v=32nrHU6zHU8&t=197s) you will see a comparison of some of the [3:19](https://www.youtube.com/watch?v=32nrHU6zHU8&t=199s) features and how different cloud [3:20](https://www.youtube.com/watch?v=32nrHU6zHU8&t=200s) providers try to try to [3:23](https://www.youtube.com/watch?v=32nrHU6zHU8&t=203s) approach this. [3:25](https://www.youtube.com/watch?v=32nrHU6zHU8&t=205s) At Nvono, we run everything on AWS. AWS [3:29](https://www.youtube.com/watch?v=32nrHU6zHU8&t=209s) has an agent core runtime. We heavily [3:31](https://www.youtube.com/watch?v=32nrHU6zHU8&t=211s) use that, but we have filled some gaps [3:33](https://www.youtube.com/watch?v=32nrHU6zHU8&t=213s) around that, like the [3:36](https://www.youtube.com/watch?v=32nrHU6zHU8&t=216s) session persistence and rehydration [3:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=219s) is something that we have built. [3:41](https://www.youtube.com/watch?v=32nrHU6zHU8&t=221s) And we also run a bunch of other [3:44](https://www.youtube.com/watch?v=32nrHU6zHU8&t=224s) bunch of SDKs for writing agents. And [3:47](https://www.youtube.com/watch?v=32nrHU6zHU8&t=227s) part of [3:48](https://www.youtube.com/watch?v=32nrHU6zHU8&t=228s) these runtimes is typically they are [3:50](https://www.youtube.com/watch?v=32nrHU6zHU8&t=230s) framework agnostic, although they all [3:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=232s) prefer their [3:53](https://www.youtube.com/watch?v=32nrHU6zHU8&t=233s) native framework in a way. [3:57](https://www.youtube.com/watch?v=32nrHU6zHU8&t=237s) Um the next [3:59](https://www.youtube.com/watch?v=32nrHU6zHU8&t=239s) layer in the stack is around memory. [4:03](https://www.youtube.com/watch?v=32nrHU6zHU8&t=243s) Um [4:04](https://www.youtube.com/watch?v=32nrHU6zHU8&t=244s) we started with rag. Rag was kind of a [4:06](https://www.youtube.com/watch?v=32nrHU6zHU8&t=246s) big thing for a while. We were [4:10](https://www.youtube.com/watch?v=32nrHU6zHU8&t=250s) kind of driven to that out of necessity [4:12](https://www.youtube.com/watch?v=32nrHU6zHU8&t=252s) because you cannot fit an unlimited [4:15](https://www.youtube.com/watch?v=32nrHU6zHU8&t=255s) amount of context into an agent. [4:18](https://www.youtube.com/watch?v=32nrHU6zHU8&t=258s) And over time um all of these cloud [4:21](https://www.youtube.com/watch?v=32nrHU6zHU8&t=261s) providers and the industry has [4:23](https://www.youtube.com/watch?v=32nrHU6zHU8&t=263s) implemented a pipeline where memory is [4:26](https://www.youtube.com/watch?v=32nrHU6zHU8&t=266s) kind of automatically generated by [4:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=268s) following a workflow of ingestion, [4:31](https://www.youtube.com/watch?v=32nrHU6zHU8&t=271s) extraction, and then consolidation and [4:33](https://www.youtube.com/watch?v=32nrHU6zHU8&t=273s) retrieval. [4:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=274s) And there are parts of rag that are [4:37](https://www.youtube.com/watch?v=32nrHU6zHU8&t=277s) built in things like a long-term memory [4:40](https://www.youtube.com/watch?v=32nrHU6zHU8&t=280s) that inherently has some semantic [4:42](https://www.youtube.com/watch?v=32nrHU6zHU8&t=282s) characteristics. But memory is built up [4:44](https://www.youtube.com/watch?v=32nrHU6zHU8&t=284s) over time from short-term conversational [4:47](https://www.youtube.com/watch?v=32nrHU6zHU8&t=287s) memory [4:48](https://www.youtube.com/watch?v=32nrHU6zHU8&t=288s) to long-term memory that you kind of [4:50](https://www.youtube.com/watch?v=32nrHU6zHU8&t=290s) manage yourself. Uh then episodic [4:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=292s) memories about kind of instances that [4:55](https://www.youtube.com/watch?v=32nrHU6zHU8&t=295s) worked well and didn't work well. [4:57](https://www.youtube.com/watch?v=32nrHU6zHU8&t=297s) Uh and so on. We at Navan again being a [5:01](https://www.youtube.com/watch?v=32nrHU6zHU8&t=301s) AWS shop, um [5:03](https://www.youtube.com/watch?v=32nrHU6zHU8&t=303s) utilize their agent core memory. [5:06](https://www.youtube.com/watch?v=32nrHU6zHU8&t=306s) But we are also kind of doing it in a [5:07](https://www.youtube.com/watch?v=32nrHU6zHU8&t=307s) way that [5:09](https://www.youtube.com/watch?v=32nrHU6zHU8&t=309s) uh matches matches our our use case. [5:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=314s) And then the next thing is context [5:16](https://www.youtube.com/watch?v=32nrHU6zHU8&t=316s) management. You know, it's a hot topic. [5:18](https://www.youtube.com/watch?v=32nrHU6zHU8&t=318s) It was a hot topic and it's still a hot [5:19](https://www.youtube.com/watch?v=32nrHU6zHU8&t=319s) topic. Context windows are growing [5:21](https://www.youtube.com/watch?v=32nrHU6zHU8&t=321s) bigger, but there's never enough context [5:24](https://www.youtube.com/watch?v=32nrHU6zHU8&t=324s) or if there is too much context again, [5:27](https://www.youtube.com/watch?v=32nrHU6zHU8&t=327s) agents struggle with that cuz you lose [5:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=328s) focus and so on. Um [5:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=332s) what we found working is that [5:35](https://www.youtube.com/watch?v=32nrHU6zHU8&t=335s) uh focusing on skills as a unit of [5:38](https://www.youtube.com/watch?v=32nrHU6zHU8&t=338s) context. And I'll explain what I mean by [5:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=339s) that. [5:40](https://www.youtube.com/watch?v=32nrHU6zHU8&t=340s) Uh we look at skills as both having [5:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=343s) context, meaning instructions and uh [5:46](https://www.youtube.com/watch?v=32nrHU6zHU8&t=346s) setup about a certain domain or a task. [5:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=349s) And there's also the [5:51](https://www.youtube.com/watch?v=32nrHU6zHU8&t=351s) the second part of the skill, which is [5:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=352s) the tool execution and you know, the [5:55](https://www.youtube.com/watch?v=32nrHU6zHU8&t=355s) agentic part. [5:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=356s) And we compose context dynamically out [6:01](https://www.youtube.com/watch?v=32nrHU6zHU8&t=361s) of skills that we [6:03](https://www.youtube.com/watch?v=32nrHU6zHU8&t=363s) use as units of work that are [6:06](https://www.youtube.com/watch?v=32nrHU6zHU8&t=366s) pluggable, that we can test [6:07](https://www.youtube.com/watch?v=32nrHU6zHU8&t=367s) independently, and that we can reuse. [6:11](https://www.youtube.com/watch?v=32nrHU6zHU8&t=371s) So, for example, when we are [6:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=374s) we have an agent, we have skills that [6:16](https://www.youtube.com/watch?v=32nrHU6zHU8&t=376s) are [6:17](https://www.youtube.com/watch?v=32nrHU6zHU8&t=377s) that are specific to a domain. [6:19](https://www.youtube.com/watch?v=32nrHU6zHU8&t=379s) And based on that, we compose them. And [6:21](https://www.youtube.com/watch?v=32nrHU6zHU8&t=381s) we rely on the [6:25](https://www.youtube.com/watch?v=32nrHU6zHU8&t=385s) you know, the progressive disclosure, [6:26](https://www.youtube.com/watch?v=32nrHU6zHU8&t=386s) which is a feature of the skills itself [6:29](https://www.youtube.com/watch?v=32nrHU6zHU8&t=389s) to start with a limited scope of context [6:31](https://www.youtube.com/watch?v=32nrHU6zHU8&t=391s) and then [6:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=392s) expand by included metadata [6:36](https://www.youtube.com/watch?v=32nrHU6zHU8&t=396s) further down the the line. [6:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=399s) I'll hand it over to Uday Uday now to [6:42](https://www.youtube.com/watch?v=32nrHU6zHU8&t=402s) kind of walk us through the rest of [6:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=403s) this. [6:44](https://www.youtube.com/watch?v=32nrHU6zHU8&t=404s) >> Thanks, Rudra. [6:46](https://www.youtube.com/watch?v=32nrHU6zHU8&t=406s) All right. Can I have a quick show of [6:48](https://www.youtube.com/watch?v=32nrHU6zHU8&t=408s) hands [6:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=409s) here who have who had built an agent uh [6:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=412s) which failed halfway through multi 20 20 [6:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=416s) step or 30 step process and be able to [6:59](https://www.youtube.com/watch?v=32nrHU6zHU8&t=419s) figure out quickly or reason about why [7:01](https://www.youtube.com/watch?v=32nrHU6zHU8&t=421s) the agent failed. [7:05](https://www.youtube.com/watch?v=32nrHU6zHU8&t=425s) So, again, logs we've generally been [7:08](https://www.youtube.com/watch?v=32nrHU6zHU8&t=428s) traditionally with microservices, we all [7:10](https://www.youtube.com/watch?v=32nrHU6zHU8&t=430s) are familiar with logs. There's logs out [7:12](https://www.youtube.com/watch?v=32nrHU6zHU8&t=432s) there and then we go check out the logs. [7:13](https://www.youtube.com/watch?v=32nrHU6zHU8&t=433s) But this changes everything the moment [7:16](https://www.youtube.com/watch?v=32nrHU6zHU8&t=436s) we switch to agents. [7:17](https://www.youtube.com/watch?v=32nrHU6zHU8&t=437s) Agents [7:19](https://www.youtube.com/watch?v=32nrHU6zHU8&t=439s) output a lot of thinking. There's too [7:20](https://www.youtube.com/watch?v=32nrHU6zHU8&t=440s) much to consume. So, that's not the [7:22](https://www.youtube.com/watch?v=32nrHU6zHU8&t=442s) right way to do it, right? So, [7:24](https://www.youtube.com/watch?v=32nrHU6zHU8&t=444s) traditionally, that was the way, but our [7:26](https://www.youtube.com/watch?v=32nrHU6zHU8&t=446s) thought has to be changed right now. [7:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=448s) In the in the way they [7:30](https://www.youtube.com/watch?v=32nrHU6zHU8&t=450s) Claude as an example, when we take [7:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=452s) Claude as an example for an agent, [7:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=454s) there is hooks and we can intercept [7:37](https://www.youtube.com/watch?v=32nrHU6zHU8&t=457s) everything that Claude as an agent that [7:38](https://www.youtube.com/watch?v=32nrHU6zHU8&t=458s) does at that level. So, what kind of [7:41](https://www.youtube.com/watch?v=32nrHU6zHU8&t=461s) tool it calls, right? What kind of [7:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=463s) decision it's making? So, before [7:45](https://www.youtube.com/watch?v=32nrHU6zHU8&t=465s) pre-tool and post-tool call or a [7:47](https://www.youtube.com/watch?v=32nrHU6zHU8&t=467s) pre-decision or a post-decision, so all [7:48](https://www.youtube.com/watch?v=32nrHU6zHU8&t=468s) of that are a [7:50](https://www.youtube.com/watch?v=32nrHU6zHU8&t=470s) point point in time for us to intercept [7:53](https://www.youtube.com/watch?v=32nrHU6zHU8&t=473s) and make a decision [7:54](https://www.youtube.com/watch?v=32nrHU6zHU8&t=474s) and either block [7:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=476s) to do a [7:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=478s) blocking operation or to log in metric [8:00](https://www.youtube.com/watch?v=32nrHU6zHU8&t=480s) or emit a metric, right? So, this is a [8:02](https://www.youtube.com/watch?v=32nrHU6zHU8&t=482s) critical [8:04](https://www.youtube.com/watch?v=32nrHU6zHU8&t=484s) place where we can emit auto traces. [8:06](https://www.youtube.com/watch?v=32nrHU6zHU8&t=486s) At Nvone, we use one of our provider to [8:10](https://www.youtube.com/watch?v=32nrHU6zHU8&t=490s) interest to emit these auto traces and [8:13](https://www.youtube.com/watch?v=32nrHU6zHU8&t=493s) through these traces we should be able [8:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=494s) to figure out the spans, the traces and [8:17](https://www.youtube.com/watch?v=32nrHU6zHU8&t=497s) at what point in time where the agent is [8:19](https://www.youtube.com/watch?v=32nrHU6zHU8&t=499s) stuck, which gives much more confidence [8:22](https://www.youtube.com/watch?v=32nrHU6zHU8&t=502s) into [8:23](https://www.youtube.com/watch?v=32nrHU6zHU8&t=503s) how we operate and build the agent. This [8:26](https://www.youtube.com/watch?v=32nrHU6zHU8&t=506s) is [8:27](https://www.youtube.com/watch?v=32nrHU6zHU8&t=507s) day-to-day operational challenge. [8:29](https://www.youtube.com/watch?v=32nrHU6zHU8&t=509s) Building agent these days there's so [8:31](https://www.youtube.com/watch?v=32nrHU6zHU8&t=511s) many frameworks, but how do you navigate [8:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=514s) building and operating an agent later is [8:36](https://www.youtube.com/watch?v=32nrHU6zHU8&t=516s) primary concern. Um [8:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=519s) And moreover, the reasoning chain, the [8:41](https://www.youtube.com/watch?v=32nrHU6zHU8&t=521s) thought process and critical signals [8:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=523s) that we emit [8:45](https://www.youtube.com/watch?v=32nrHU6zHU8&t=525s) here as part of the trace captures, we [8:47](https://www.youtube.com/watch?v=32nrHU6zHU8&t=527s) emit a few primary signals here. What is [8:50](https://www.youtube.com/watch?v=32nrHU6zHU8&t=530s) the current goal the agent is going [8:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=532s) through, the reasons behind its [8:54](https://www.youtube.com/watch?v=32nrHU6zHU8&t=534s) operations and the belief status and [8:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=536s) the tool calls that it's making. So, [8:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=538s) this kind of gives us a judgment [8:59](https://www.youtube.com/watch?v=32nrHU6zHU8&t=539s) pointers. Um [9:01](https://www.youtube.com/watch?v=32nrHU6zHU8&t=541s) Um in the traces. And when we make when [9:04](https://www.youtube.com/watch?v=32nrHU6zHU8&t=544s) the agent makes a decision, there is a [9:07](https://www.youtube.com/watch?v=32nrHU6zHU8&t=547s) confidence score, how confident it is [9:10](https://www.youtube.com/watch?v=32nrHU6zHU8&t=550s) when it makes the judgment, right? So, [9:12](https://www.youtube.com/watch?v=32nrHU6zHU8&t=552s) whether there are multiple paths that it [9:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=554s) leads to this choice [9:16](https://www.youtube.com/watch?v=32nrHU6zHU8&t=556s) or whether this is an inferred answer. [9:18](https://www.youtube.com/watch?v=32nrHU6zHU8&t=558s) So, basically these are signals that [9:20](https://www.youtube.com/watch?v=32nrHU6zHU8&t=560s) gives us confidence later to review. If [9:23](https://www.youtube.com/watch?v=32nrHU6zHU8&t=563s) this is an inferred answer, there could [9:24](https://www.youtube.com/watch?v=32nrHU6zHU8&t=564s) be a human in the loop to guide through [9:26](https://www.youtube.com/watch?v=32nrHU6zHU8&t=566s) and tweak the agent to perform a little [9:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=568s) better. [9:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=572s) Again, um [9:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=574s) Can I have a raise of hands again to see [9:36](https://www.youtube.com/watch?v=32nrHU6zHU8&t=576s) how confident are you like 100% [9:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=579s) confident in testing pipelines with your [9:41](https://www.youtube.com/watch?v=32nrHU6zHU8&t=581s) agents? [9:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=583s) Right. So, this is one of the other um [9:47](https://www.youtube.com/watch?v=32nrHU6zHU8&t=587s) critical aspect today. Um [9:51](https://www.youtube.com/watch?v=32nrHU6zHU8&t=591s) Because agents are non-deterministic. [9:53](https://www.youtube.com/watch?v=32nrHU6zHU8&t=593s) We've all been used to program and write [9:55](https://www.youtube.com/watch?v=32nrHU6zHU8&t=595s) much more deterministic flows. [9:57](https://www.youtube.com/watch?v=32nrHU6zHU8&t=597s) And [9:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=598s) we know how it works. The Can I ask an [10:01](https://www.youtube.com/watch?v=32nrHU6zHU8&t=601s) engineer? Engineer can come and tell me [10:02](https://www.youtube.com/watch?v=32nrHU6zHU8&t=602s) how this the algorithm, the sequence of [10:04](https://www.youtube.com/watch?v=32nrHU6zHU8&t=604s) operations. Everything is programmed in [10:06](https://www.youtube.com/watch?v=32nrHU6zHU8&t=606s) our mind. Everything is expectations. [10:08](https://www.youtube.com/watch?v=32nrHU6zHU8&t=608s) But now the agents come into a [10:09](https://www.youtube.com/watch?v=32nrHU6zHU8&t=609s) non-deterministic way. And how do we [10:11](https://www.youtube.com/watch?v=32nrHU6zHU8&t=611s) test them, right? So, that is very [10:12](https://www.youtube.com/watch?v=32nrHU6zHU8&t=612s) criticality here. [10:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=614s) And yeah. We are also struggling. Um [10:17](https://www.youtube.com/watch?v=32nrHU6zHU8&t=617s) we've uh started [10:19](https://www.youtube.com/watch?v=32nrHU6zHU8&t=619s) doing building agents. We the day to [10:21](https://www.youtube.com/watch?v=32nrHU6zHU8&t=621s) operations was challenging and then we [10:22](https://www.youtube.com/watch?v=32nrHU6zHU8&t=622s) failed in a lot of steps. How do we [10:24](https://www.youtube.com/watch?v=32nrHU6zHU8&t=624s) course correct? The moment we change [10:26](https://www.youtube.com/watch?v=32nrHU6zHU8&t=626s) something, something else broke breaks, [10:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=628s) right? So, how do we do that? Um one one [10:31](https://www.youtube.com/watch?v=32nrHU6zHU8&t=631s) approach that we took uh this is from um [10:33](https://www.youtube.com/watch?v=32nrHU6zHU8&t=633s) research papers uh around the [10:38](https://www.youtube.com/watch?v=32nrHU6zHU8&t=638s) in a multi-step [10:40](https://www.youtube.com/watch?v=32nrHU6zHU8&t=640s) uh orchestration, when an agent makes uh [10:42](https://www.youtube.com/watch?v=32nrHU6zHU8&t=642s) 30 steps or decisions to make to reach [10:45](https://www.youtube.com/watch?v=32nrHU6zHU8&t=645s) to a goal, [10:46](https://www.youtube.com/watch?v=32nrHU6zHU8&t=646s) if that is a program or that's a [10:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=649s) different story. But this is not a [10:50](https://www.youtube.com/watch?v=32nrHU6zHU8&t=650s) program. This is non-deterministic way [10:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=652s) of It makes up its own steps every time [10:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=656s) uh differently. [10:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=658s) So, how can we [11:00](https://www.youtube.com/watch?v=32nrHU6zHU8&t=660s) chart a deterministic graph here? [11:03](https://www.youtube.com/watch?v=32nrHU6zHU8&t=663s) Is it possible? No. [11:05](https://www.youtube.com/watch?v=32nrHU6zHU8&t=665s) Can we have a trajectory of its starting [11:08](https://www.youtube.com/watch?v=32nrHU6zHU8&t=668s) from an end to a goal and then see how [11:10](https://www.youtube.com/watch?v=32nrHU6zHU8&t=670s) much how far it went in the trajectory [11:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=674s) and how far it went from the source to [11:15](https://www.youtube.com/watch?v=32nrHU6zHU8&t=675s) the destination is what we can compute [11:18](https://www.youtube.com/watch?v=32nrHU6zHU8&t=678s) to evaluate the efficiency or the [11:22](https://www.youtube.com/watch?v=32nrHU6zHU8&t=682s) completeness of the [11:23](https://www.youtube.com/watch?v=32nrHU6zHU8&t=683s) agent [11:24](https://www.youtube.com/watch?v=32nrHU6zHU8&t=684s) agent evaluation. [11:26](https://www.youtube.com/watch?v=32nrHU6zHU8&t=686s) So, we we heavily rely on um trajectory [11:29](https://www.youtube.com/watch?v=32nrHU6zHU8&t=689s) vals um [11:30](https://www.youtube.com/watch?v=32nrHU6zHU8&t=690s) and uh [11:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=692s) this [11:33](https://www.youtube.com/watch?v=32nrHU6zHU8&t=693s) There are few other signals uh as I [11:35](https://www.youtube.com/watch?v=32nrHU6zHU8&t=695s) briefly spoke around uh in the previous [11:36](https://www.youtube.com/watch?v=32nrHU6zHU8&t=696s) slide around the inferred signal. Um [11:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=699s) If the answer is from an in in inferred [11:41](https://www.youtube.com/watch?v=32nrHU6zHU8&t=701s) answer, uh how can we uh [11:44](https://www.youtube.com/watch?v=32nrHU6zHU8&t=704s) loop that into uh and make a [11:48](https://www.youtube.com/watch?v=32nrHU6zHU8&t=708s) signals around uh how can we classify [11:50](https://www.youtube.com/watch?v=32nrHU6zHU8&t=710s) that this is a regression and make fixes [11:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=712s) towards the agent? [11:59](https://www.youtube.com/watch?v=32nrHU6zHU8&t=719s) Uh [12:03](https://www.youtube.com/watch?v=32nrHU6zHU8&t=723s) So, the next is the uh guardrails. Um [12:07](https://www.youtube.com/watch?v=32nrHU6zHU8&t=727s) Where [12:11](https://www.youtube.com/watch?v=32nrHU6zHU8&t=731s) Is this the one? Yeah. [12:15](https://www.youtube.com/watch?v=32nrHU6zHU8&t=735s) So, guardrails and authorization, um [12:18](https://www.youtube.com/watch?v=32nrHU6zHU8&t=738s) this is uh critical [12:21](https://www.youtube.com/watch?v=32nrHU6zHU8&t=741s) displays a critical role in enterprise [12:22](https://www.youtube.com/watch?v=32nrHU6zHU8&t=742s) AI. [12:24](https://www.youtube.com/watch?v=32nrHU6zHU8&t=744s) A lot of information is being piped to [12:27](https://www.youtube.com/watch?v=32nrHU6zHU8&t=747s) models. Um there could be sensitive [12:29](https://www.youtube.com/watch?v=32nrHU6zHU8&t=749s) information that goes into it uh without [12:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=752s) our knowledge. And [12:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=754s) we as uh uh leaders, how can we put in [12:36](https://www.youtube.com/watch?v=32nrHU6zHU8&t=756s) this governance layer um [12:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=759s) to stop this um is very uh critical [12:42](https://www.youtube.com/watch?v=32nrHU6zHU8&t=762s) here. [12:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=763s) And and the concept of uh authentication [12:45](https://www.youtube.com/watch?v=32nrHU6zHU8&t=765s) and authorization um is [12:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=769s) taking up a different approach here. Um [12:51](https://www.youtube.com/watch?v=32nrHU6zHU8&t=771s) traditionally, we've seen um a user or a [12:54](https://www.youtube.com/watch?v=32nrHU6zHU8&t=774s) service account, but now what is an [12:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=776s) agent? Agent can be [12:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=778s) acting as on behalf of users. There is [13:00](https://www.youtube.com/watch?v=32nrHU6zHU8&t=780s) so much of things uh so many of use [13:02](https://www.youtube.com/watch?v=32nrHU6zHU8&t=782s) cases there. Hey, book me a flight [13:05](https://www.youtube.com/watch?v=32nrHU6zHU8&t=785s) whenever it's cheaper than $200, right? [13:07](https://www.youtube.com/watch?v=32nrHU6zHU8&t=787s) So, we just tell this assertion and then [13:09](https://www.youtube.com/watch?v=32nrHU6zHU8&t=789s) agent go figures out and does this [13:11](https://www.youtube.com/watch?v=32nrHU6zHU8&t=791s) action on behalf of me. So, is it me [13:13](https://www.youtube.com/watch?v=32nrHU6zHU8&t=793s) making this [13:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=794s) purchase or is it agent me making on [13:17](https://www.youtube.com/watch?v=32nrHU6zHU8&t=797s) behalf of me? So, there is [13:20](https://www.youtube.com/watch?v=32nrHU6zHU8&t=800s) Agent acts as a on behalf of user or [13:22](https://www.youtube.com/watch?v=32nrHU6zHU8&t=802s) agent uses a service account as well. [13:24](https://www.youtube.com/watch?v=32nrHU6zHU8&t=804s) So, [13:25](https://www.youtube.com/watch?v=32nrHU6zHU8&t=805s) the line is being blurred here and we [13:27](https://www.youtube.com/watch?v=32nrHU6zHU8&t=807s) need to make fine-grained authorization [13:29](https://www.youtube.com/watch?v=32nrHU6zHU8&t=809s) decisions here, and the policy layer [13:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=812s) that's where the guardrails and [13:33](https://www.youtube.com/watch?v=32nrHU6zHU8&t=813s) authentication authorization plays a [13:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=814s) critical role. [13:36](https://www.youtube.com/watch?v=32nrHU6zHU8&t=816s) And in the one what we employ here is [13:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=819s) before every tool call [13:41](https://www.youtube.com/watch?v=32nrHU6zHU8&t=821s) pre-tool and post-tool, we have this [13:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=823s) guardrails to check and block [13:46](https://www.youtube.com/watch?v=32nrHU6zHU8&t=826s) and make a informed decisions. [13:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=832s) And [13:53](https://www.youtube.com/watch?v=32nrHU6zHU8&t=833s) this single agent versus multi-agent, [13:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=836s) again, this is kind of a [13:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=838s) orchestration wars you can think of [14:00](https://www.youtube.com/watch?v=32nrHU6zHU8&t=840s) with it to build a single agent or a [14:01](https://www.youtube.com/watch?v=32nrHU6zHU8&t=841s) multi-agent. [14:03](https://www.youtube.com/watch?v=32nrHU6zHU8&t=843s) Again, as Roberto briefly hinted [14:05](https://www.youtube.com/watch?v=32nrHU6zHU8&t=845s) if you can't perfect and build a [14:08](https://www.youtube.com/watch?v=32nrHU6zHU8&t=848s) single agent, why go towards [14:11](https://www.youtube.com/watch?v=32nrHU6zHU8&t=851s) multi-agent, right? So, learn from our [14:13](https://www.youtube.com/watch?v=32nrHU6zHU8&t=853s) uh [14:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=854s) failures, experiences, and build towards [14:17](https://www.youtube.com/watch?v=32nrHU6zHU8&t=857s) that. [14:18](https://www.youtube.com/watch?v=32nrHU6zHU8&t=858s) At Navan, yeah. [14:20](https://www.youtube.com/watch?v=32nrHU6zHU8&t=860s) What the approach that we have taken is [14:22](https://www.youtube.com/watch?v=32nrHU6zHU8&t=862s) single master, and then we adopted [14:25](https://www.youtube.com/watch?v=32nrHU6zHU8&t=865s) sub-skills. Um [14:27](https://www.youtube.com/watch?v=32nrHU6zHU8&t=867s) There are sub-agents within it. [14:29](https://www.youtube.com/watch?v=32nrHU6zHU8&t=869s) So, it's a single agent that can [14:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=872s) progressively load the skills and [14:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=874s) understand decisively what needs to be [14:37](https://www.youtube.com/watch?v=32nrHU6zHU8&t=877s) loaded into the context, and then [14:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=879s) make this [14:40](https://www.youtube.com/watch?v=32nrHU6zHU8&t=880s) navigation [14:41](https://www.youtube.com/watch?v=32nrHU6zHU8&t=881s) through the use case. [14:44](https://www.youtube.com/watch?v=32nrHU6zHU8&t=884s) But there are [14:45](https://www.youtube.com/watch?v=32nrHU6zHU8&t=885s) other patterns that are also emerging. [14:48](https://www.youtube.com/watch?v=32nrHU6zHU8&t=888s) There are different class of use cases [14:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=889s) here. One is um [14:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=892s) agent-to-agent communication. So, there [14:53](https://www.youtube.com/watch?v=32nrHU6zHU8&t=893s) are If you take a large scale [14:55](https://www.youtube.com/watch?v=32nrHU6zHU8&t=895s) organization, and there are so many of [14:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=896s) these teams that are [14:59](https://www.youtube.com/watch?v=32nrHU6zHU8&t=899s) that are acting as the boundaries, and [15:00](https://www.youtube.com/watch?v=32nrHU6zHU8&t=900s) they don't talk to each other, let's [15:01](https://www.youtube.com/watch?v=32nrHU6zHU8&t=901s) say. [15:02](https://www.youtube.com/watch?v=32nrHU6zHU8&t=902s) How do we communicate? There are two [15:04](https://www.youtube.com/watch?v=32nrHU6zHU8&t=904s) agents on either of the side, right? How [15:06](https://www.youtube.com/watch?v=32nrHU6zHU8&t=906s) do we do it? So, there is A2A protocol [15:09](https://www.youtube.com/watch?v=32nrHU6zHU8&t=909s) which can help us establish the [15:12](https://www.youtube.com/watch?v=32nrHU6zHU8&t=912s) contracts in terms of skills. [15:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=914s) And we can use A2A as a protocol there, [15:17](https://www.youtube.com/watch?v=32nrHU6zHU8&t=917s) which kind of [15:19](https://www.youtube.com/watch?v=32nrHU6zHU8&t=919s) is a boundary between the teams. [15:22](https://www.youtube.com/watch?v=32nrHU6zHU8&t=922s) Yeah, over to you, Uday. [15:29](https://www.youtube.com/watch?v=32nrHU6zHU8&t=929s) >> All right. So, as we went through the [15:30](https://www.youtube.com/watch?v=32nrHU6zHU8&t=930s) stack, [15:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=932s) it's obvious that um some components of [15:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=934s) the stack are in a more mature state and [15:38](https://www.youtube.com/watch?v=32nrHU6zHU8&t=938s) we already have good answers for them. [15:40](https://www.youtube.com/watch?v=32nrHU6zHU8&t=940s) As Uday said, the runtime, I think it's [15:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=943s) pretty much solved. We are so advanced [15:44](https://www.youtube.com/watch?v=32nrHU6zHU8&t=944s) in orchestration and we are running LLMs [15:47](https://www.youtube.com/watch?v=32nrHU6zHU8&t=947s) in kind of uh a very [15:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=949s) uh brute-force way. So, scaling is not a [15:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=952s) not a problem. Also, memory, I think uh [15:55](https://www.youtube.com/watch?v=32nrHU6zHU8&t=955s) as [15:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=956s) uh the frontier LLMs get better and as [16:00](https://www.youtube.com/watch?v=32nrHU6zHU8&t=960s) our practices get better, [16:02](https://www.youtube.com/watch?v=32nrHU6zHU8&t=962s) we will uh find a way to cover the [16:05](https://www.youtube.com/watch?v=32nrHU6zHU8&t=965s) majority of the use cases and there is [16:06](https://www.youtube.com/watch?v=32nrHU6zHU8&t=966s) good maturity around the the cloud [16:09](https://www.youtube.com/watch?v=32nrHU6zHU8&t=969s) providers. [16:10](https://www.youtube.com/watch?v=32nrHU6zHU8&t=970s) Uh MCP has emerged as the de facto [16:13](https://www.youtube.com/watch?v=32nrHU6zHU8&t=973s) protocol and tool calling is now a [16:16](https://www.youtube.com/watch?v=32nrHU6zHU8&t=976s) feature that everybody supports. So, we [16:18](https://www.youtube.com/watch?v=32nrHU6zHU8&t=978s) are seeing some industry convergence [16:21](https://www.youtube.com/watch?v=32nrHU6zHU8&t=981s) around that as well and MCP as a [16:24](https://www.youtube.com/watch?v=32nrHU6zHU8&t=984s) standard is also evolving. Now, it's [16:25](https://www.youtube.com/watch?v=32nrHU6zHU8&t=985s) becoming stateless. It's uh we are [16:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=988s) reaching a point where kind of we know [16:30](https://www.youtube.com/watch?v=32nrHU6zHU8&t=990s) how to invoke uh services and and and [16:33](https://www.youtube.com/watch?v=32nrHU6zHU8&t=993s) tools with agents. [16:35](https://www.youtube.com/watch?v=32nrHU6zHU8&t=995s) Uh in some areas, things are happening, [16:39](https://www.youtube.com/watch?v=32nrHU6zHU8&t=999s) but you know, there's still a lot of [16:40](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1000s) unknown. Around observability, there is [16:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1003s) a push towards OTEL, but does OTEL [16:46](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1006s) really work for agentic calls? [16:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1009s) Uh yeah, you can make it work as Uday [16:51](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1011s) was saying. [16:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1012s) Um also, we are getting more comfortable [16:55](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1015s) around um around that the the the the [16:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1018s) testing patterns. It's very hard to [16:59](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1019s) test, but we have found a way to give [17:02](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1022s) customers um quality experiences even [17:05](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1025s) with the unreliability of agentic system [17:07](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1027s) and I think that's kind of [17:09](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1029s) uh getting in a in a state that is uh [17:12](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1032s) that is more better defined. [17:15](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1035s) Orchestration is another one [17:17](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1037s) um, [17:18](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1038s) where, you know, we have a [17:21](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1041s) uh, we have patterns, uh, we can build, [17:24](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1044s) you know, bigger agents, smaller agents. [17:27](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1047s) Uh, as we said previously, probably the [17:30](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1050s) right answer is to not over-engineer. [17:34](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1054s) Uh, so we're learning there and and and [17:36](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1056s) uh, [17:37](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1057s) a pattern of school thought is also [17:40](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1060s) emerging. Uh, where we're all struggling [17:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1063s) with and the previous talk was about [17:45](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1065s) this for the developer, um, [17:47](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1067s) AI assistant development perspective, [17:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1069s) but also we're seeing these issues from [17:52](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1072s) our production agents. It's very hard to [17:55](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1075s) predict cost and it's very hard to [17:57](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1077s) manage cost, uh, and put guardrails and [18:01](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1081s) solve this in a way where there is [18:02](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1082s) reliable, maybe fallback or have agents [18:06](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1086s) be, uh, using cheaper models for certain [18:10](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1090s) tasks. Uh, [18:11](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1091s) uh, this is all driven by kind of the [18:14](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1094s) big AI vendors who, I think, their [18:17](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1097s) interest is for us all to spend more [18:20](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1100s) tokens. Um, replay and debugging, Woody [18:23](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1103s) talked about that, that's also a big big [18:25](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1105s) issue. It's very hard [18:28](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1108s) to understand, but I think [18:30](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1110s) this is also something that that is [18:32](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1112s) going to be solved because we can now [18:35](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1115s) use, uh, agents to [18:37](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1117s) uh, get over the cognitive overload of [18:40](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1120s) trying to debug what they do. [18:43](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1123s) And then standards, um, standards are [18:46](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1126s) emerging uh, by, [18:49](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1129s) you know, the community. Uh, Hotel, as I [18:51](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1131s) mentioned, agent to agent is young, it's [18:53](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1133s) kind of pushed by certain vendors, but I [18:56](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1136s) think over time we will we will get, uh, [18:58](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1138s) there. [19:00](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1140s) Uh, with all of this said, you know, we [19:03](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1143s) know what we need and it's up to us to [19:06](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1146s) write and build it. [19:07](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1147s) Thank you, everybody. [19:09](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1149s) >> [applause] [19:27](https://www.youtube.com/watch?v=32nrHU6zHU8&t=1167s) >> I