It's 10pm. Do You Know Where Your Agents Are?

Talk transcript

1. 0:00

   [upbeat music] Okay, so when I was a kid growing up in the '90s, we'd be out late all summer riding our bikes off in the neighborhood and playing with friends.

2. 0:20

   And from the '60s to the '80s, there were these public service announcements on TV where celebrities would come on, and they would say, "It's 10:00 PM. Do you know where your children are?"

3. 0:30

   Because apparently our parents at that time needed to be reminded that they had offspring they were responsible for. And I feel like in this phase of AI where we are right now, um, we're entrusting agents with more and more responsibility,

4. 0:44

   but we still kind of need that public service announcement that says, "It's 10:00 PM. Do you know where your agents are?" [laughing]

5. 0:54

   So say you as a user want an agent to use an MCP server or API to accomplish tasks. Now, we know agents without access aren't useful, so we give them an end file, and we give them some API keys, and we let them run off and go do their thing, and this is fine until it's not.

6. 1:14

   Uh, so you've heard the horror stories, right? Or maybe even experienced some of them yourself. Uh, so let's see what this looks like in practice.

7. 1:34

   Okay. It's this way. Okay. So I have an agent running here that is basically an incident management agent.

8. 1:51

   It is l- late night, and there is a human user, but they're probably half asleep. The agent is responsible for triaging issues that are coming in. Right. So if we

9. 2:04

   look at the first one here. If I could find the mouse.

10. 2:10

    So the first one is the backup power supply failed in the server room.

11. 2:16

    Now, the agent is going to use an API key to read the system that is bringing in the reports. It's going to evaluate what is written there, and it's going to decide that it can't do anything about this, right, because it's a, it's a physical fail- failure.

12. 2:31

    Um, so it's going to escalate that ticket for the morning, and there's really sort of no problems yet, right?

13. 2:38

    So now the second one is the certificate is expiring soon, right, for TLS.

14. 2:46

    Agent is going to use that same API key again to read the ticket contents, and then it is going to decide that it should renew the certificate, and it's going to use another API key to call the cloud hosting service to, uh, renew the certificate, and then it's going to use the API key from before to make

15. 3:08

    the report for the morning team. So here's where it kind of gets fun, right? So this one is the billing database is broken and payments are failing.

16. 3:23

    Agent's going to read that. It sees that the solution is pretty clear. It says that the documented recovery is to delete the database and then restore fro- let the restore from backup happen automatically.

17. 3:36

    Um, so it has the Postgres connection string, so it goes ahead and it drops the database, and then it doesn't have a way to check to see if it was backed up, so it just escalates that for the morning.

18. 3:48

    And this has really happened, right? Like, it's happened to high-profile companies.

19. 3:57

    Now, if we go to the next one, this one is the main server processes are frozen, and they're not recovering. Doing something about it is gonna take prod offline for a brief amount of time, and you can see the, the agent decided that it should do that, and it's using the same API key now that it did

20. 4:14

    to renew the certificate, right? Because that API key is a kitchen sink. It can do all of these things with it.

21. 4:23

    And then finally, we have the site's failing for one in three users.

22. 4:28

    The recommended solution is to scale up, right? And this is going to incur some amount of spend, and then it goes ahead and it does that because, again, it can use that same API key to do this as well.

23. 4:52

    Right, so agents with API keys are indeed out past 10:00. They're over privileged, so this means they are able to act freely on decisions that they make that you may or may not agree with, and they can do this even with your supervision.

24. 5:05

    So you might be familiar with the panic of, uh, trying to stop an agent mid-task because you told it to maybe read a project, and it read the project, and it found something it thought it should fix, and then it starts writing.

25. 5:16

    Um, agents do that even while they're supervised, and this is becoming even more of a problem because we have more and more agents that are running unsupervised, and that only makes it worse because agents want to be helpful.

26. 5:28

    They're going to use all the permissions that they have access to in order to get the job done.

27. 5:35

    And we can't just solve this with human-in-the-loop. We spent decades solving access management for humans, so just blindly trusting a human who might be a little bit consent fatigued, uh, or who might be tired enough at night, this isn't really going to be enough.

28. 5:52

    So in order to see where we can introduce security and access control, we have to take a look at the agentic execution path.

29. 6:02

    So we have a user who wants to use an LLM to interact with a resource. Now, an agent is a control loop that calls an LLM, and often we have an MCP server in between that provides tools that the agent can call, and then it connects directly to the resource.

30. 6:19

    Now, an MCP client takes the agent's proposed tool calls and dispatches them to its MCP server.

31. 6:26

    And then we have a runtime. Now, this is a process that runs the agent loop and executes the calls, and this runtime might be a CLI like Claude Code, it might be an SDK like AI SDK or provider agent SDKs, or it might be an app like Cursor or Codex.

32. 6:45

    So let's follow a prompt through the execution path. The user submits the prompt to the runtime, which calls the model, which sends it to the LLM. The model is then going to propose tool calls, which are dispatched by the MCP client to the MCP server, which then executes the tools, and it calls the resource API.

33. 7:06

    The API then responds to the MCP server, and the MCP client delivers the results to the runtime, and then the model will be called, and this loop repeats until the model is satisfied.

34. 7:17

    And at that point, it's going to return the final answer to the user.

35. 7:23

    So there are a few places in this path where we could implement real access control, and we can actually do this with open standards. And as kind of a spoiler alert, it's not just OAuth.

36. 7:36

    So RFC 8693 is Token Exchange, and this is an RFC that extends OAuth 2.0, and I'm going to show you how this spec can be used to address agent access.

37. 7:49

    So first, I wanna recap the problems that we're actually trying to solve here, right? So if you remember looking at the audit log as it was going by in the demo, we could see API keys were being used to call endpoints, but we had no idea who was using the API keys.

38. 8:02

    So we have credentials that are being used that aren't attributed to a user or an agent identity.

39. 8:09

    We have an agent that has unrestricted access to any and all permissions that are in an API key. And finally, we know that we can't just slap human in the loop everywhere because humans make mistakes too, and also many agents run autonomously.

40. 8:26

    So we can address this with an authorization server called a Security Token Service.

41. 8:32

    Now, an authorization server verifies identities and issues tokens, so identity providers like Google, Okta, Auth0, and so on provide authorization servers. And if we want an identity chain in our agent execution path, then we have to be able to log in first.

42. 8:49

    So the authorization server is then going to prompt the user for their consent to delegate access with a subset of their permissions. And this is the first narrowing of access, right?

43. 8:58

    We're only delegating some of the user's total permissions to the agent.

44. 9:04

    The authorization server issues a token that identifies the user and also contains their level of access. And so right now already we're doing better than the first demo because we actually know who the user is and what they're allowed to do.

45. 9:17

    So this token identifies the subject on whose behalf the agent is going to act.

46. 9:23

    In order to support token exchange, we need an OAuth client that's capable of executing code. So this might be a gateway between the MCP client and a third-party MCP server.

47. 9:34

    It might be your own custom agent app or a CLI wrapper around an off-the-shelf coding agent.

48. 9:41

    And we take the prompt and we take the subject token, and we send these to the OAuth client. Now, the agent loop runs and the model proposes a tool call.

49. 9:51

    And then the runtime is going to authenticate with the Security Token Service using its OAuth app client credentials or workload identity. And it also sends the subject token that contains the user's identity and level of access.

50. 10:04

    It creates a token exchange request, and this request is asking for permissions to access the MCP server for that tool call, but only that tool call. So now we have three key pieces of information that we're missing from the API key demo.

51. 10:22

    We know the identity of the agent that's requesting access, we know the identity of the user on whose behalf it's acting, and we know the delegating user's level of access as well.

52. 10:34

    So now we need to decide if the requested token should in fact be granted, and we can do this using governance policy,

53. 10:41

    which is evaluated against the requested access and who's asking for what resource on whose behalf.

54. 10:50

    Now, if the delegation chain and the requested access are within policy, then the Security Token Service issues an access token for the downstream resource. And this token has an audience declaring that only this target MCP server is allowed to use it to make requests.

55. 11:05

    It should be short-lived, uh, often expiring within a few minutes, and it's also ephemeral, meaning it should never be stored.

56. 11:13

    So this token is sent to the MCP client, which makes a tool call using it as a bearer credential.

57. 11:19

    The MCP server validates the token and then goes and calls the resource. And again, it never stores the token, and it discards it as soon as the call is done.

58. 11:31

    So the result flows back up the loop, and then it repeats until the model returns the answer to the user.

59. 11:40

    So if we come back to the demo, now we're going to use the same agent,

60. 11:49

    only now we have token exchange. Okay. So the first thing we have that's different already is that we have an operator sign-in, right? So we have authentication,

61. 12:05

    and I'm going to authenticate with Google as myself

62. 12:14

    So we have the same tickets, we have the same agent, it's got the same prompt, and now we can see what it's going to do. Now, the first item is probably going to be exactly the same, right?

63. 12:23

    Because this was a hardware failure. There's-- It's gonna decide after it reads it that there's nothing it can do. But as you can see kind of the audit log filling up, we've got a lot more information now.

64. 12:32

    Um, we know who the agent is acting on behalf of. We know that the agent is calling a prod infra MCP server, and we know that it's going to contact certain downstream resources, right?

65. 12:43

    So we have a hardware monitor that is the source of this incident, and then when it decides that it can't do anything about it, it uses a write scope to talk to the pager,

66. 12:57

    uh, resource in order to escalate to the morning team.

67. 13:07

    So for renewing the certificate, right? This is actually a pretty safe action, and it's going to specifically ask for a scope to only renew the certificate, right? So it's talking to this cloud host where before we had this API key that could do a ton of different things.

68. 13:24

    But this time it is only asking for permissions and being granted permission to do this one thing.

69. 13:33

    So now with the billing database, right? Like, the billing database is broken. It, uh, pretty clearly documents that you are supposed to, uh, drop the database here,

70. 13:44

    but no agent should be able to drop a database. So what happens is when we make this call, it's being evalu- the policy is evaluating the request against all of the permissions that the user has and, uh, it sees that there is actually a restriction in place that prevents agents from doing this.

71. 14:03

    And this credential never even existed. So the policy evaluates before the credential is minted, which means you don't have an overprivileged credential that's just floating around then that, uh, you were supposed to then prevent the entity from receiving.

72. 14:17

    Um, it just doesn't exist, so there's nothing to leak, there's nothing to replay, and there's nothing to steal.

73. 14:27

    So this one was the one where it wants to restart prod, right? So there are things that the agent should probably be allowed to do and things that maybe they shouldn't be allowed to do, and then there's some kind of, you know, something in between, right?

74. 14:41

    So it's gonna ask me as the user for my approval as human-in-the-loop.

75. 14:46

    I say that it can do that, but there's another policy here that says that the human user needs to have a specific role in order to be able to do this, and I actually do not have that role.

76. 14:57

    So it's going to prevent me from being able to allow the agent to do this, uh, even though I approved it. So we can prevent kind of people from just consent fatigue, clicking over and over just to get things done.

77. 15:14

    And then this is the scaling one, right? So this is something that maybe the user does have permission to do.

78. 15:23

    So if I say approve on this, I do have permission to do this and I was able to tell the agent that it is indeed allowed to, and the policy approved it because I am allowed to do it also.

79. 15:40

    Okay. Mouse is-- Where are the mouse? So the agent access problems that we had discussed, they have solutions now.

80. 15:56

    We know who the user is, and we know who the agent is as well. The agent also has task-scoped, short-lived ephemeral access.

81. 16:06

    And human-in-the-loop actually has access control that is backed by real policy, so an exhausted person can't just accept everything that happens.

82. 16:17

    Now, another benefit of using open standards like Token Exchange is the ability to continue to support emerging technologies. So this works with off-the-shelf agents. It also works with custom agents that you might build yourself.

83. 16:29

    It works with the CLI. It works with third-party as well as proprietary MCP servers, MCP gateways, agent-to-agent, uh, any OAuth identity provider. It works with OpenCL and basically anything that might come out next week.

84. 16:45

    So it is in fact possible right now to be that responsible parent and to say that, yes, you do in fact know where your agents are.

85. 16:58

    So my name is Kim Maida. I am the Founding GTM Engineer and Head of DevRel at Keycard, which is a standards-based platform for, uh, providing a Security Token Service and policy governance.

86. 17:10

    I'm gonna be at the Keycard booth for kind of the duration of the event, but we're also running a workshop tomorrow on building and securing an MCP server.

87. 17:20

    You can scan this QR code to connect with me, and I really appreciate your time today. So thank you very much. [clapping]

88. 17:31

    Uh, we do have time for a couple questions. Uh, so if anyone has some questions for Kim, um, we can answer those right now. And then we got about fi- five minutes, so we can probably do about five short questions if anyone has them.

89. 17:46

    Over here.

90. 17:55

    Um, uh, so it seems like you put the security bear- barrier, uh, at the, um, between MCP to, uh, resource? Or is it, uh, just maybe to clarify that, is it at, at both places?

91. 18:15

    I'm wondering if there was, like, any decision there or, like, what would motivate you to choose, you know, where to put these sorts of barriers.

92. 18:23

    Uh, yeah. So the, the authorization server is

93. 18:27

    sitting in between ... Let me find this slide, actually.

94. 18:32

    Do, do. So the runtime authenticates to the security token service and identifies itself, and this is the point at which we have the request that the agent generated.

95. 18:54

    Um, and we also have the req- the scopes that it's asking for to get the token to call the next thing in line, right? And the next thing in line in this particular case is the MCP server.

96. 19:05

    So the downstream resource is, like, one step farther down, but if you think about, like, an OAuth token for a user, right? So an OAuth token for a user is going to have all of the grants in it that the user accepted when it, when we were presented with, like, here's what access you're gonna delegate.

97. 19:25

    But an agent, you don't want an agent to be using any of those grants that it wants on every single tool call. So this, the service sits in between that so that we can say if the things it's requesting are kind of beyond what we want to allow for the s- specific tool call, then we never send

98. 19:47

    that OAuth token down, right? So they get that OAuth token only if they are within, requesting something within the scope of what, what we want.

99. 19:58

    Does that make sense? [background chatter]

100. 20:21

     OAuth-based enterprise systems, uh, there may be some resistance to actually adopt this new, uh, protocol, new open spec. Have you encountered that, and what are the ways to, uh, get past that?

101. 20:41

     So it's not actually a new spec, which is, you know, it, it's kind of one of those things, like, there was this period of time where people were like, "Oh, you can just use OAuth for, for this."

102. 20:54

     Um, we don't necessarily need a new spec for this. This spec has actually existed for a little while already. Um, so there's not kind of that fear of, "Oh my gosh, we're introducing something completely new."

103. 21:07

     There are new specs that are coming out almost daily right now, um, but they can be combined essentially with this, uh, with token exchange.

104. 21:20

     I think we have time for one more question. Uh. [background chatter]

105. 21:27

     I might be asking a big question, so if you tell me to just go look at the thing, that's fine.

106. 21:32

     Sure.

107. 21:32

     Um, I'm in the situation where we know that we don't have enough OAuth scopes defined yet in an MCP server sub, just like what you have. And one of the reluctant things that we've got is, like, how fine-grained do we get with the scope definitions?

108. 21:45

     Because we know that we're also having to define downstream services that will have a certain number of these things, and someone has to do the authorization check somewhere. What's your recommendation for getting started with defining your own scopes so that you can realistically manage this thing when you haven't defined enough yet and you're worried about ongoing management

109. 22:03

     over time?

110. 22:05

     Well, if your resource server already has, like, specific scopes, that's going to be kind of your place to start because the downstream token, the, the one that was the OAuth token for the user, is going to have the scopes for the resource, right?

111. 22:18

     Because it's the user's access to the resource. So those are kind of like the baseline. Like, those are the, those are the ones you know that you're going to have.

112. 22:26

     And then if you want to have scopes additionally for the kind of governed tool calls, if you have, like, a custom MCP server or something like that, then you can layer those on top, or you can just pass them through.

113. 22:39

     Yeah. Okay. Thanks.

114. 22:41

     All right. Thank you, Kim. [clapping] [outro music]