[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12480,
    "text": "Hey everyone, how all of you are doing"
  },
  {
    "startMs": 15360,
    "text": "today?"
  },
  {
    "startMs": 17520,
    "text": "Yeah. Uh so nice to meet you everyone."
  },
  {
    "startMs": 20960,
    "text": "Uh today uh I and my friend Avery are"
  },
  {
    "startMs": 24240,
    "text": "going to talk about build time versus"
  },
  {
    "startMs": 25680,
    "text": "runtime. Why your developer tools fail"
  },
  {
    "startMs": 28160,
    "text": "in production."
  },
  {
    "startMs": 30480,
    "text": "So firstly, know about us."
  },
  {
    "startMs": 33440,
    "text": ">> Hi everybody. I'm Avery Kit and I'm a"
  },
  {
    "startMs": 36000,
    "text": "staff software engineer working on"
  },
  {
    "startMs": 37600,
    "text": "Google Cloud databases. I'm currently"
  },
  {
    "startMs": 40000,
    "text": "the technical lead for MCP toolbox for"
  },
  {
    "startMs": 42480,
    "text": "databases, our open-source uh database"
  },
  {
    "startMs": 45440,
    "text": "MCP server and our Google Cloud MCP"
  },
  {
    "startMs": 48160,
    "text": "server um maintainer."
  },
  {
    "startMs": 51360,
    "text": "Hi, I'm Pna and I am currently working"
  },
  {
    "startMs": 53920,
    "text": "as senior software engineer at Google"
  },
  {
    "startMs": 56559,
    "text": "and I am currently tech lead for Eval"
  },
  {
    "startMs": 59440,
    "text": "bench which is the evaluation framework"
  },
  {
    "startMs": 61359,
    "text": "for all your agent tech MCP and skills"
  },
  {
    "startMs": 63520,
    "text": "need and I'm also an active contributor"
  },
  {
    "startMs": 66000,
    "text": "to MCP toolbox."
  },
  {
    "startMs": 69360,
    "text": "So today we are going to cover three"
  },
  {
    "startMs": 71760,
    "text": "areas broadly. We will firstly start"
  },
  {
    "startMs": 73840,
    "text": "with the history of MCP at Google. Then"
  },
  {
    "startMs": 77040,
    "text": "we will cover on the common tool"
  },
  {
    "startMs": 78560,
    "text": "patterns that we have found from our own"
  },
  {
    "startMs": 81360,
    "text": "work and practices and how did we use"
  },
  {
    "startMs": 84320,
    "text": "all those practices to build some tools"
  },
  {
    "startMs": 86479,
    "text": "for database access and how you can use"
  },
  {
    "startMs": 88960,
    "text": "them and then lastly we will talk about"
  },
  {
    "startMs": 91840,
    "text": "security guard rails how you can stop"
  },
  {
    "startMs": 94400,
    "text": "data leaks using identity aware"
  },
  {
    "startMs": 96560,
    "text": "guardrails."
  },
  {
    "startMs": 98960,
    "text": "So let's get to know the background"
  },
  {
    "startMs": 101439,
    "text": "quickly. Um I'll talk about MCB toolbox"
  },
  {
    "startMs": 104799,
    "text": "for database. It's an open-source"
  },
  {
    "startMs": 107439,
    "text": "self-managed uh serving that we provide."
  },
  {
    "startMs": 110640,
    "text": "Uh it has currently about 15.7K GitHub"
  },
  {
    "startMs": 114159,
    "text": "stars. We have 132 plus active"
  },
  {
    "startMs": 116560,
    "text": "contributors across 40 plus different"
  },
  {
    "startMs": 118960,
    "text": "databases. It's highly customizable"
  },
  {
    "startMs": 121520,
    "text": "framework and basically we provide you"
  },
  {
    "startMs": 124399,
    "text": "with connection pooling integrated O and"
  },
  {
    "startMs": 127040,
    "text": "you don't even need to care about the"
  },
  {
    "startMs": 128560,
    "text": "observability. You will get all of them"
  },
  {
    "startMs": 130720,
    "text": "out of the box."
  },
  {
    "startMs": 132879,
    "text": "Then if you don't want to do a"
  },
  {
    "startMs": 134720,
    "text": "self-managed one but you want to have a"
  },
  {
    "startMs": 137200,
    "text": "hosted scaled version, we provide"
  },
  {
    "startMs": 140160,
    "text": "something as Google managed MCP. It's"
  },
  {
    "startMs": 143360,
    "text": "fully managed. Uh you can plug it across"
  },
  {
    "startMs": 146239,
    "text": "various agents and ids or harnesses like"
  },
  {
    "startMs": 149200,
    "text": "Gemini CLI, anti-gravity CLI, cloud"
  },
  {
    "startMs": 152800,
    "text": "code, you name any. uh it's co uh it's"
  },
  {
    "startMs": 156879,
    "text": "governed and the discovery is simple and"
  },
  {
    "startMs": 159840,
    "text": "we also provide model armor which"
  },
  {
    "startMs": 162319,
    "text": "provides secure access management and"
  },
  {
    "startMs": 164879,
    "text": "identity control. So combined with uh"
  },
  {
    "startMs": 168160,
    "text": "the managed version of MCP and the MCP"
  },
  {
    "startMs": 171280,
    "text": "toolbox last month we had 20 million"
  },
  {
    "startMs": 173920,
    "text": "tool calls."
  },
  {
    "startMs": 176800,
    "text": "Um some of the common tool patterns that"
  },
  {
    "startMs": 179040,
    "text": "we have observed specifically for"
  },
  {
    "startMs": 180720,
    "text": "databases. So I'm going to quickly talk"
  },
  {
    "startMs": 182800,
    "text": "about them."
  },
  {
    "startMs": 184800,
    "text": "Firstly uh is the control plane tools."
  },
  {
    "startMs": 187440,
    "text": "What we like to call them is admin tools"
  },
  {
    "startMs": 189840,
    "text": "or manage tools. It is basically in"
  },
  {
    "startMs": 192560,
    "text": "developer assistance space. So it will"
  },
  {
    "startMs": 194879,
    "text": "help you create like instance, manage"
  },
  {
    "startMs": 197680,
    "text": "your instance, create your databases,"
  },
  {
    "startMs": 200159,
    "text": "manage your databases. It will help you"
  },
  {
    "startMs": 202080,
    "text": "with all your DBA needs. But you need to"
  },
  {
    "startMs": 205280,
    "text": "be very careful. You need to be you need"
  },
  {
    "startMs": 208080,
    "text": "to have a human in the loop because we"
  },
  {
    "startMs": 210319,
    "text": "don't want to carry out any dangerous"
  },
  {
    "startMs": 212319,
    "text": "activities."
  },
  {
    "startMs": 214000,
    "text": "Um so these tools are built on already"
  },
  {
    "startMs": 217040,
    "text": "provisioned public API so you get"
  },
  {
    "startMs": 219360,
    "text": "monitoring and other things out of the"
  },
  {
    "startMs": 221840,
    "text": "box."
  },
  {
    "startMs": 224239,
    "text": "Next one is natural language to SQL or"
  },
  {
    "startMs": 226480,
    "text": "NL2SQL tools. So basically we are"
  },
  {
    "startMs": 229599,
    "text": "relying on a tool called as execute SQL"
  },
  {
    "startMs": 232480,
    "text": "and with the help of agent we generate"
  },
  {
    "startMs": 234799,
    "text": "raw SQL queries. So you can use this"
  },
  {
    "startMs": 237280,
    "text": "cases where you don't know uh what"
  },
  {
    "startMs": 239680,
    "text": "queries you would require beforehand. So"
  },
  {
    "startMs": 241840,
    "text": "you will get all these queries out of"
  },
  {
    "startMs": 244080,
    "text": "the out of the box. So it it focuses on"
  },
  {
    "startMs": 248239,
    "text": "the developer assistance and analytical"
  },
  {
    "startMs": 250319,
    "text": "agents and uh you can use it for"
  },
  {
    "startMs": 252640,
    "text": "flexible explorations. So for example,"
  },
  {
    "startMs": 255599,
    "text": "we have one of the examples like find"
  },
  {
    "startMs": 257840,
    "text": "all customers in California who bought a"
  },
  {
    "startMs": 260320,
    "text": "winter coat in July and returned it"
  },
  {
    "startMs": 262880,
    "text": "within 14 days and group them by the"
  },
  {
    "startMs": 265199,
    "text": "marketing campaign that originally"
  },
  {
    "startMs": 268000,
    "text": "acquired them. So this is one of the"
  },
  {
    "startMs": 270080,
    "text": "queries where uh you can use this tool"
  },
  {
    "startMs": 272880,
    "text": "uh to get your answers."
  },
  {
    "startMs": 277360,
    "text": "But then we have something called a"
  },
  {
    "startMs": 279120,
    "text": "structure SQL tools which is getting"
  },
  {
    "startMs": 281040,
    "text": "quite popular and this targets mainly"
  },
  {
    "startMs": 283600,
    "text": "the production use cases where you know"
  },
  {
    "startMs": 286080,
    "text": "like what SQL query you want to use and"
  },
  {
    "startMs": 288800,
    "text": "you want to have security built in and"
  },
  {
    "startMs": 291840,
    "text": "uh you the parameters are already"
  },
  {
    "startMs": 294160,
    "text": "configured so uh you prevent SQL"
  },
  {
    "startMs": 296880,
    "text": "injection"
  },
  {
    "startMs": 298400,
    "text": "and ensure highly controlled access by"
  },
  {
    "startMs": 300639,
    "text": "restricting agent to predefined logic."
  },
  {
    "startMs": 303919,
    "text": "It also helps you with your latency"
  },
  {
    "startMs": 305840,
    "text": "needs and reduce the hallucination on"
  },
  {
    "startMs": 307840,
    "text": "the agent side."
  },
  {
    "startMs": 310800,
    "text": "Now we come to the main topic I guess"
  },
  {
    "startMs": 313120,
    "text": "for which you guys are here for"
  },
  {
    "startMs": 315039,
    "text": "buildtime versus runtime. So buildtime"
  },
  {
    "startMs": 317840,
    "text": "are the developer assistant use cases."
  },
  {
    "startMs": 320080,
    "text": "Um you can think about the initial two"
  },
  {
    "startMs": 322240,
    "text": "cases that we presented to you like the"
  },
  {
    "startMs": 324080,
    "text": "NL2SQL tools and the control plane"
  },
  {
    "startMs": 326560,
    "text": "tools. They come into the category of"
  },
  {
    "startMs": 328320,
    "text": "buildtime tools. uh it's atomic and f"
  },
  {
    "startMs": 331199,
    "text": "flexible but again you don't want to"
  },
  {
    "startMs": 333840,
    "text": "delete your databases so it requires to"
  },
  {
    "startMs": 336639,
    "text": "be a human in the loop case and you"
  },
  {
    "startMs": 338639,
    "text": "can't run them on the on production use"
  },
  {
    "startMs": 340639,
    "text": "cases but let's say I'm interested in"
  },
  {
    "startMs": 343120,
    "text": "building some chat B and I want to do"
  },
  {
    "startMs": 345919,
    "text": "production use cases there you rely on"
  },
  {
    "startMs": 348320,
    "text": "runtime or end user applications you can"
  },
  {
    "startMs": 351039,
    "text": "build those using patenting AI or lchain"
  },
  {
    "startMs": 354240,
    "text": "um so you can see one of the examples"
  },
  {
    "startMs": 356000,
    "text": "like we have a cancel order a"
  },
  {
    "startMs": 357759,
    "text": "deterministic structure SQL query that"
  },
  {
    "startMs": 360000,
    "text": "we have given and you can use it as a"
  },
  {
    "startMs": 362080,
    "text": "tool."
  },
  {
    "startMs": 365039,
    "text": "This is one of the examples uh or demo"
  },
  {
    "startMs": 367919,
    "text": "for like wherein a buildtime tool was"
  },
  {
    "startMs": 371280,
    "text": "used and uh you can see the error"
  },
  {
    "startMs": 373680,
    "text": "message. So uh agent actually asked to"
  },
  {
    "startMs": 376479,
    "text": "delete the table and start fresh. We"
  },
  {
    "startMs": 378560,
    "text": "deleted everything and there were no"
  },
  {
    "startMs": 380639,
    "text": "safeguard or guardrails here."
  },
  {
    "startMs": 384000,
    "text": "Now let's go to our demo for"
  },
  {
    "startMs": 387120,
    "text": "runtime tools."
  },
  {
    "startMs": 398080,
    "text": "Yeah, maybe um I think until the video"
  },
  {
    "startMs": 401520,
    "text": "loads. So, so sorry for the technical"
  },
  {
    "startMs": 404160,
    "text": "glitch that we have, but I can quickly"
  },
  {
    "startMs": 406160,
    "text": "walk you through what we are going to"
  },
  {
    "startMs": 407520,
    "text": "present in the video and I guess it's"
  },
  {
    "startMs": 409520,
    "text": "loading. Yeah."
  },
  {
    "startMs": 412240,
    "text": "Um so this demo is particularly talking"
  },
  {
    "startMs": 415280,
    "text": "about how did we use our production"
  },
  {
    "startMs": 417759,
    "text": "tools in a chatbot. Uh and we created a"
  },
  {
    "startMs": 422400,
    "text": "demo called a Similar and Symbolair is"
  },
  {
    "startMs": 425440,
    "text": "going to help me with booking all my"
  },
  {
    "startMs": 427440,
    "text": "flights in San Francisco and do and"
  },
  {
    "startMs": 430720,
    "text": "whatever I would require to do in San"
  },
  {
    "startMs": 432880,
    "text": "San Francisco it would basically help me"
  },
  {
    "startMs": 434720,
    "text": "with it. Uh, one of the things that I"
  },
  {
    "startMs": 437840,
    "text": "would try is I would try to fool my"
  },
  {
    "startMs": 440400,
    "text": "agent that I am Avery and not PRA and"
  },
  {
    "startMs": 443599,
    "text": "book a flight for me to San Francisco."
  },
  {
    "startMs": 447599,
    "text": "But because our agent is uh has all the"
  },
  {
    "startMs": 451599,
    "text": "authenticated O, it will not get fooled"
  },
  {
    "startMs": 454319,
    "text": "and it will not book any flights uh on"
  },
  {
    "startMs": 456960,
    "text": "behalf of Avery, but it will do it on my"
  },
  {
    "startMs": 459360,
    "text": "behalf. Um and then you can use it to"
  },
  {
    "startMs": 462240,
    "text": "basically change your flights. You want"
  },
  {
    "startMs": 464479,
    "text": "to know about all the shops that are"
  },
  {
    "startMs": 466400,
    "text": "there, you can do all these requirements"
  },
  {
    "startMs": 468639,
    "text": "using that. So I guess thank you u"
  },
  {
    "startMs": 473520,
    "text": "Avery."
  },
  {
    "startMs": 479360,
    "text": "I think we"
  },
  {
    "startMs": 497805,
    "text": ">> [sighs]"
  },
  {
    "startMs": 499759,
    "text": ">> Apologies again for our technical"
  },
  {
    "startMs": 502639,
    "text": "difficulties here."
  },
  {
    "startMs": 512000,
    "text": "Um, unfortunately, it looks like I need"
  },
  {
    "startMs": 513680,
    "text": "to present from just the slide deck"
  },
  {
    "startMs": 515839,
    "text": "because it's not loading. Okay, so I"
  },
  {
    "startMs": 518880,
    "text": "apologize for not being able to see our"
  },
  {
    "startMs": 520560,
    "text": "demo today, but we can still learn all"
  },
  {
    "startMs": 522399,
    "text": "the security and guardrails that we need"
  },
  {
    "startMs": 524320,
    "text": "to secure our database access. So, the"
  },
  {
    "startMs": 526959,
    "text": "first thing that we need to know is your"
  },
  {
    "startMs": 528720,
    "text": "database is only as secure as your"
  },
  {
    "startMs": 530640,
    "text": "agent. We all know that agents and LMS"
  },
  {
    "startMs": 534000,
    "text": "are actually pretty easy to trick. They"
  },
  {
    "startMs": 535839,
    "text": "might be getting slightly better today,"
  },
  {
    "startMs": 537600,
    "text": "but we can still work really hard to"
  },
  {
    "startMs": 540160,
    "text": "trick them. And so we have a very common"
  },
  {
    "startMs": 543440,
    "text": "attack pattern called the confused"
  },
  {
    "startMs": 545120,
    "text": "deputy attack. And this is when a user"
  },
  {
    "startMs": 547600,
    "text": "can trick an agent into misusing their"
  },
  {
    "startMs": 550640,
    "text": "privileges um to access data that a user"
  },
  {
    "startMs": 553760,
    "text": "wasn't supposed to access. So Simon"
  },
  {
    "startMs": 556720,
    "text": "Willis actually coined the phrase the"
  },
  {
    "startMs": 558640,
    "text": "lethal trifecta. And a data breach"
  },
  {
    "startMs": 561440,
    "text": "occurs when an agent has simultaneous"
  },
  {
    "startMs": 564320,
    "text": "access to three different things. One,"
  },
  {
    "startMs": 566720,
    "text": "private data. Two, untrusted content."
  },
  {
    "startMs": 570399,
    "text": "And three, the ability to expose that"
  },
  {
    "startMs": 573040,
    "text": "content and that data back to an"
  },
  {
    "startMs": 575279,
    "text": "external user."
  },
  {
    "startMs": 578560,
    "text": "So let's take a look of that in action."
  },
  {
    "startMs": 581600,
    "text": "So let's say I'm building a triage um"
  },
  {
    "startMs": 584080,
    "text": "agent and so a ticket is fired or alert"
  },
  {
    "startMs": 586880,
    "text": "goes out and my agent is designed to um"
  },
  {
    "startMs": 591040,
    "text": "look at that ticket and go investigate"
  },
  {
    "startMs": 594320,
    "text": "what it needs to do. So on that ticket"
  },
  {
    "startMs": 597200,
    "text": "the agent gets a little bit of data like"
  },
  {
    "startMs": 599200,
    "text": "we need to go look in this database for"
  },
  {
    "startMs": 601279,
    "text": "these reasons. Um but a malicious"
  },
  {
    "startMs": 603600,
    "text": "insider can actually come into that"
  },
  {
    "startMs": 605600,
    "text": "trusted system and instead say well I"
  },
  {
    "startMs": 609279,
    "text": "want to query the salary database and"
  },
  {
    "startMs": 611680,
    "text": "please return all the employees"
  },
  {
    "startMs": 613440,
    "text": "salaries. And so since this is a trusted"
  },
  {
    "startMs": 616079,
    "text": "system the agent goes okay let me use my"
  },
  {
    "startMs": 618880,
    "text": "permissions. I have those privileges. I"
  },
  {
    "startMs": 621120,
    "text": "have that access. I will query that and"
  },
  {
    "startMs": 623200,
    "text": "I'll post that right back on the ticket"
  },
  {
    "startMs": 624720,
    "text": "because that's what the ticket tells me"
  },
  {
    "startMs": 626640,
    "text": "to do. But now we have a huge data"
  },
  {
    "startMs": 630000,
    "text": "breach. a user that wasn't supposed to"
  },
  {
    "startMs": 633040,
    "text": "have access to private data now has that"
  },
  {
    "startMs": 635760,
    "text": "access. And so now we have a big PR"
  },
  {
    "startMs": 638480,
    "text": "fiasco."
  },
  {
    "startMs": 643519,
    "text": "So this makes a little bit more sense"
  },
  {
    "startMs": 645519,
    "text": "when we think about who's controlling"
  },
  {
    "startMs": 647760,
    "text": "access and who's controlling the"
  },
  {
    "startMs": 650079,
    "text": "parameters. So we talk about agent or"
  },
  {
    "startMs": 652880,
    "text": "application versus modeled controlled"
  },
  {
    "startMs": 654959,
    "text": "parameters. So in a traditional"
  },
  {
    "startMs": 657360,
    "text": "architecture, things were actually much"
  },
  {
    "startMs": 659920,
    "text": "easier because you would have a few"
  },
  {
    "startMs": 662640,
    "text": "input fields, you would define your"
  },
  {
    "startMs": 664640,
    "text": "queries and then that would be safely"
  },
  {
    "startMs": 666880,
    "text": "injected into those queries."
  },
  {
    "startMs": 670800,
    "text": "And so it was okay when your application"
  },
  {
    "startMs": 673600,
    "text": "had a little bit more access because it"
  },
  {
    "startMs": 677279,
    "text": "knew exactly what actions it was going"
  },
  {
    "startMs": 679360,
    "text": "to take."
  },
  {
    "startMs": 681920,
    "text": "But in uh a gent application these rules"
  },
  {
    "startMs": 685200,
    "text": "aren't as clear. So we need to first"
  },
  {
    "startMs": 687360,
    "text": "think about um separating the three"
  },
  {
    "startMs": 689440,
    "text": "different identities. We have the user"
  },
  {
    "startMs": 691920,
    "text": "identity, we have the application"
  },
  {
    "startMs": 694000,
    "text": "identity and the agent identity."
  },
  {
    "startMs": 699120,
    "text": "So first um we need to think about what"
  },
  {
    "startMs": 701680,
    "text": "the user has access to. So the user just"
  },
  {
    "startMs": 704640,
    "text": "needs to have access to the application."
  },
  {
    "startMs": 707279,
    "text": "that application's workload identity can"
  },
  {
    "startMs": 710079,
    "text": "have a little bit more broader access um"
  },
  {
    "startMs": 712880,
    "text": "because it needs to probably talk to"
  },
  {
    "startMs": 714480,
    "text": "different services but the agent running"
  },
  {
    "startMs": 717200,
    "text": "in that application only needs to have"
  },
  {
    "startMs": 720399,
    "text": "access to the data that that end user"
  },
  {
    "startMs": 722880,
    "text": "initially needs to have."
  },
  {
    "startMs": 726000,
    "text": "So then next we need to think about"
  },
  {
    "startMs": 727760,
    "text": "who's controlling the tool inputs. So we"
  },
  {
    "startMs": 731600,
    "text": "have um agent parameters"
  },
  {
    "startMs": 735040,
    "text": "um and a application parameters. So"
  },
  {
    "startMs": 737440,
    "text": "agent parameters are the untrusted"
  },
  {
    "startMs": 739519,
    "text": "inputs that the agent is deriving"
  },
  {
    "startMs": 741360,
    "text": "dynamically. And then we also have"
  },
  {
    "startMs": 743519,
    "text": "application parameters. These are the"
  },
  {
    "startMs": 745680,
    "text": "factual constraints that we need to keep"
  },
  {
    "startMs": 747920,
    "text": "outside of the agents uh control."
  },
  {
    "startMs": 753040,
    "text": "Okay. So now let's look at the evolution"
  },
  {
    "startMs": 756240,
    "text": "of a secure tool. Here we have a fully"
  },
  {
    "startMs": 759360,
    "text": "modeled control tool. And so essentially"
  },
  {
    "startMs": 762079,
    "text": "the agent here is a super user. It has"
  },
  {
    "startMs": 764399,
    "text": "access to database credentials, the"
  },
  {
    "startMs": 766720,
    "text": "host, the port, the connection details,"
  },
  {
    "startMs": 768959,
    "text": "and even the raw SQL query."
  },
  {
    "startMs": 773440,
    "text": "And so we're only secure as um the agent"
  },
  {
    "startMs": 777920,
    "text": "here. And we can really easily again"
  },
  {
    "startMs": 779839,
    "text": "trick the agent into exposing all of"
  },
  {
    "startMs": 781839,
    "text": "this data. And now we have access to"
  },
  {
    "startMs": 783920,
    "text": "essentially any database in the system."
  },
  {
    "startMs": 788000,
    "text": "So Toolbox solves for this um by"
  },
  {
    "startMs": 790160,
    "text": "introducing a source primitive."
  },
  {
    "startMs": 793040,
    "text": "So we move the connection details out of"
  },
  {
    "startMs": 795279,
    "text": "the agents control and in toolbox um a"
  },
  {
    "startMs": 798399,
    "text": "user will preconfigure the connection"
  },
  {
    "startMs": 800320,
    "text": "details in a YAML file and then when we"
  },
  {
    "startMs": 802560,
    "text": "start our MCP server those are safely"
  },
  {
    "startMs": 804720,
    "text": "injected and so we do not have to have"
  },
  {
    "startMs": 807200,
    "text": "the agent um to have access to that."
  },
  {
    "startMs": 813279,
    "text": "So we can add a little bit more control"
  },
  {
    "startMs": 815279,
    "text": "to our um source security as well. Our"
  },
  {
    "startMs": 818480,
    "text": "number one request that we get from"
  },
  {
    "startMs": 819839,
    "text": "customers is read only restrictions. We"
  },
  {
    "startMs": 822560,
    "text": "want to be able to remove all right"
  },
  {
    "startMs": 824480,
    "text": "ability from agents if we need that"
  },
  {
    "startMs": 826639,
    "text": "specific uh user journey. So this means"
  },
  {
    "startMs": 829600,
    "text": "removing right tools but also down to"
  },
  {
    "startMs": 832079,
    "text": "the database driver ensuring that we can"
  },
  {
    "startMs": 834880,
    "text": "only do read only queries."
  },
  {
    "startMs": 838160,
    "text": "If we're also concerned about again"
  },
  {
    "startMs": 840399,
    "text": "blast radius um and securing all of our"
  },
  {
    "startMs": 843279,
    "text": "tables and our databases um some of our"
  },
  {
    "startMs": 845600,
    "text": "cloudnative databases have this concept"
  },
  {
    "startMs": 848240,
    "text": "of allowed data sets. So again we can"
  },
  {
    "startMs": 850560,
    "text": "add that like enum to our source in"
  },
  {
    "startMs": 852880,
    "text": "order to continue to restrict um the"
  },
  {
    "startMs": 854959,
    "text": "blast radius of um the agents control"
  },
  {
    "startMs": 857839,
    "text": "and lastly is output size. You might not"
  },
  {
    "startMs": 859920,
    "text": "actually think that this is a security"
  },
  {
    "startMs": 862000,
    "text": "layer, but if again the agent gets into"
  },
  {
    "startMs": 865360,
    "text": "the wrong hands, we can reduce that"
  },
  {
    "startMs": 867360,
    "text": "blast radius by saying uh the agent can"
  },
  {
    "startMs": 870000,
    "text": "only uh grab this much data. So we're"
  },
  {
    "startMs": 871920,
    "text": "not overwhelming both our agent or our"
  },
  {
    "startMs": 874320,
    "text": "database."
  },
  {
    "startMs": 878240,
    "text": "So sweet, we have our configurable"
  },
  {
    "startMs": 880399,
    "text": "sources tool. So you can see here that"
  },
  {
    "startMs": 882800,
    "text": "actually now our tool input, our tool"
  },
  {
    "startMs": 885120,
    "text": "signature is very minimalized. we only"
  },
  {
    "startMs": 887839,
    "text": "have the SQL string that's um being"
  },
  {
    "startMs": 890959,
    "text": "generated by the agent."
  },
  {
    "startMs": 895120,
    "text": "But this comes to our actual our next"
  },
  {
    "startMs": 896959,
    "text": "pro problem. We want to be able to"
  },
  {
    "startMs": 899839,
    "text": "control what the agent is running. We"
  },
  {
    "startMs": 902639,
    "text": "don't want the agent to have the ability"
  },
  {
    "startMs": 904160,
    "text": "to generate any SQL um that it can think"
  },
  {
    "startMs": 906880,
    "text": "of. So toolbox introduces custom tools"
  },
  {
    "startMs": 910720,
    "text": "and again in our YAML file we can define"
  },
  {
    "startMs": 912959,
    "text": "the exact SQL uh statement that will run"
  },
  {
    "startMs": 916320,
    "text": "very reliable."
  },
  {
    "startMs": 918880,
    "text": "It's a reliable and secure uh SQL query."
  },
  {
    "startMs": 922399,
    "text": "Um this also allows us to customize the"
  },
  {
    "startMs": 924560,
    "text": "tool name and the tool description."
  },
  {
    "startMs": 926480,
    "text": "These are really important for the agent"
  },
  {
    "startMs": 928720,
    "text": "to have the context on how to use this"
  },
  {
    "startMs": 930959,
    "text": "tool um accurately."
  },
  {
    "startMs": 934000,
    "text": "And in the system we use prepared"
  },
  {
    "startMs": 935920,
    "text": "statements with type parameters in order"
  },
  {
    "startMs": 937920,
    "text": "to reduce um SQL injection attacks. So"
  },
  {
    "startMs": 940720,
    "text": "we make sure that everything is um"
  },
  {
    "startMs": 944480,
    "text": "we validate all the input types um when"
  },
  {
    "startMs": 946800,
    "text": "we inject that into the SQL for the"
  },
  {
    "startMs": 948639,
    "text": "user."
  },
  {
    "startMs": 952160,
    "text": "Okay, let's dive into a little bit more"
  },
  {
    "startMs": 953920,
    "text": "of best practices for tool quality. So"
  },
  {
    "startMs": 956399,
    "text": "we really highly recommend that tools"
  },
  {
    "startMs": 958399,
    "text": "focus on outcomes. We really shouldn't"
  },
  {
    "startMs": 960560,
    "text": "be thinking in atomic rest APIs. we"
  },
  {
    "startMs": 963360,
    "text": "should think about what the action"
  },
  {
    "startMs": 965360,
    "text": "actually needs to do. This also reduces"
  },
  {
    "startMs": 967759,
    "text": "the round trip of needing to make"
  },
  {
    "startMs": 970160,
    "text": "multiple tool calls. And again, the"
  },
  {
    "startMs": 973279,
    "text": "descriptions are guidance. We shouldn't"
  },
  {
    "startMs": 975759,
    "text": "um duplicate information like input"
  },
  {
    "startMs": 977839,
    "text": "parameters because the agent already has"
  },
  {
    "startMs": 979600,
    "text": "access to that. So, writing really good"
  },
  {
    "startMs": 982240,
    "text": "um tool descriptions is very important"
  },
  {
    "startMs": 984560,
    "text": "for accurate tool usage."
  },
  {
    "startMs": 987120,
    "text": "We also recommend that you separate read"
  },
  {
    "startMs": 989600,
    "text": "versus write tools. Um by doing this you"
  },
  {
    "startMs": 992720,
    "text": "can automatically approve read tools and"
  },
  {
    "startMs": 995120,
    "text": "but you can also then send write tools"
  },
  {
    "startMs": 997680,
    "text": "uh to the user for um confirmation and"
  },
  {
    "startMs": 1000160,
    "text": "this just makes it very much more clear"
  },
  {
    "startMs": 1002000,
    "text": "for the agent to use these"
  },
  {
    "startMs": 1004639,
    "text": "and this is actually uh the next is"
  },
  {
    "startMs": 1006240,
    "text": "actionable errors. This is the number"
  },
  {
    "startMs": 1008079,
    "text": "one thing that I think we can all do"
  },
  {
    "startMs": 1010160,
    "text": "better. So usually we just return like a"
  },
  {
    "startMs": 1012800,
    "text": "generic HTTP error four or four but we"
  },
  {
    "startMs": 1016079,
    "text": "all know agents are actually really"
  },
  {
    "startMs": 1017519,
    "text": "smart now and so if you give the ability"
  },
  {
    "startMs": 1020160,
    "text": "to have an error of that can be"
  },
  {
    "startMs": 1022399,
    "text": "retrieded the agent can actually take"
  },
  {
    "startMs": 1024720,
    "text": "that action. So being able to return a"
  },
  {
    "startMs": 1027360,
    "text": "error is really important and lastly is"
  },
  {
    "startMs": 1031280,
    "text": "simple inputs. We see that people try to"
  },
  {
    "startMs": 1033760,
    "text": "use these complex maps uh complex"
  },
  {
    "startMs": 1036559,
    "text": "primitives to um that an agent needs to"
  },
  {
    "startMs": 1039439,
    "text": "be able to build and that is not"
  },
  {
    "startMs": 1041760,
    "text": "reliable. Using flat structure with um"
  },
  {
    "startMs": 1046559,
    "text": "with uh simple inputs will really"
  },
  {
    "startMs": 1048880,
    "text": "increase your reliability."
  },
  {
    "startMs": 1052640,
    "text": "So sweet. Now we're at custom semantic"
  },
  {
    "startMs": 1055120,
    "text": "tools. You can see that we now have our"
  },
  {
    "startMs": 1057679,
    "text": "lookup flights tool that takes in the"
  },
  {
    "startMs": 1059919,
    "text": "dynamic parameters such as user ID and"
  },
  {
    "startMs": 1062880,
    "text": "date. And so now our we're very much"
  },
  {
    "startMs": 1065679,
    "text": "more secure because the agent isn't"
  },
  {
    "startMs": 1067520,
    "text": "generating that SQL query. It doesn't"
  },
  {
    "startMs": 1069520,
    "text": "have the ability to kind of go off the"
  },
  {
    "startMs": 1071200,
    "text": "rails. It only is looking at these very"
  },
  {
    "startMs": 1073360,
    "text": "specific inputs."
  },
  {
    "startMs": 1077600,
    "text": "But user ID is actually a very sensitive"
  },
  {
    "startMs": 1080480,
    "text": "piece of information. It is PII. we need"
  },
  {
    "startMs": 1082960,
    "text": "to also remove that from the ability of"
  },
  {
    "startMs": 1085360,
    "text": "the agent's control. So we can do this"
  },
  {
    "startMs": 1087919,
    "text": "in two different ways. We have bounded"
  },
  {
    "startMs": 1090160,
    "text": "parameters. This is when the application"
  },
  {
    "startMs": 1092640,
    "text": "first um authenticates the user and then"
  },
  {
    "startMs": 1095120,
    "text": "we can bind that parameter um directly"
  },
  {
    "startMs": 1097679,
    "text": "to our tool. And so that restricts the"
  },
  {
    "startMs": 1100000,
    "text": "agents control of it. It actually never"
  },
  {
    "startMs": 1101760,
    "text": "sees that user identity."
  },
  {
    "startMs": 1104880,
    "text": "But toolbox also solves for this in"
  },
  {
    "startMs": 1107200,
    "text": "another way called authenticated"
  },
  {
    "startMs": 1108880,
    "text": "parameters. This is when we tell the"
  },
  {
    "startMs": 1111360,
    "text": "tool that you're going to receive a"
  },
  {
    "startMs": 1114240,
    "text": "identity token, an open ID, a signed jot"
  },
  {
    "startMs": 1116960,
    "text": "token, and when we call that tool that"
  },
  {
    "startMs": 1120080,
    "text": "we want it first to validate that token."
  },
  {
    "startMs": 1122320,
    "text": "Is that token real? Is that token"
  },
  {
    "startMs": 1124320,
    "text": "correct? And then we'll extract the user"
  },
  {
    "startMs": 1126960,
    "text": "claims from that token for the user. And"
  },
  {
    "startMs": 1129440,
    "text": "so the claims usually include like a"
  },
  {
    "startMs": 1131120,
    "text": "user ID, an email, um an issuer."
  },
  {
    "startMs": 1135440,
    "text": "And so it's secured because we're again"
  },
  {
    "startMs": 1138080,
    "text": "extracting that user identity out of the"
  },
  {
    "startMs": 1141280,
    "text": "agents control and binding that to the"
  },
  {
    "startMs": 1143360,
    "text": "tool."
  },
  {
    "startMs": 1148640,
    "text": "So now um we're have a much more secure"
  },
  {
    "startMs": 1153120,
    "text": "tool. We have our lookup flights tool"
  },
  {
    "startMs": 1155440,
    "text": "that only takes in a very easy parameter"
  },
  {
    "startMs": 1158960,
    "text": "such as date. It doesn't have to handle"
  },
  {
    "startMs": 1161120,
    "text": "any sensitive information such as PII,"
  },
  {
    "startMs": 1163919,
    "text": "user identity. And so we're really here"
  },
  {
    "startMs": 1166640,
    "text": "now at um our zero trust architecture"
  },
  {
    "startMs": 1170000,
    "text": "where we're in full control of"
  },
  {
    "startMs": 1172080,
    "text": "everything that we need to be in control"
  },
  {
    "startMs": 1174080,
    "text": "of."
  },
  {
    "startMs": 1179520,
    "text": "So thank you all for coming to listen to"
  },
  {
    "startMs": 1181520,
    "text": "our talk today. Again, I apologize for"
  },
  {
    "startMs": 1183600,
    "text": "our technical difficulties. Uh we highly"
  },
  {
    "startMs": 1186080,
    "text": "recommend if you want to learn more"
  },
  {
    "startMs": 1187440,
    "text": "about our technologies um that you look"
  },
  {
    "startMs": 1190000,
    "text": "at our documentation and our uh GitHub"
  },
  {
    "startMs": 1192559,
    "text": "repository. I also really want to"
  },
  {
    "startMs": 1194880,
    "text": "highlight our eval bench repository"
  },
  {
    "startMs": 1196960,
    "text": "because this is how we know that our"
  },
  {
    "startMs": 1198880,
    "text": "tools are working well and eval."
  },
  {
    "startMs": 1203600,
    "text": "So thank you all for joining us today."
  },
  {
    "startMs": 1207178,
    "text": "[applause]"
  }
]