Why Are We Still Governing AI Agents Like Software? - Tim Tyler, Gravitee

Uh, before we get started, uh, my name's Tim, and I'm with Gravitee AI manage- uh, uh agent management. Uh, before we get started, I'm sorry, somebody back at the office asked me to put this up real quick, so bear with me. Uh, we're hiring right now, so I'm gonna put up a job description, and if you know anybody, it's in marketing operations. Um, basically, no ID required. You can just come in, no docs. Uh, there's no manager or supervisor for this position. It's got a really obscure job

description. You have ac- access to key systems, but don't worry, they're all shared passwords. Uh, and you will be expected to work twenty-four/seven, but nobody's checking on you, and we won't fire you if you get anything wrong. I see some smiles and chuckles. Everybody raise your hand if you would hire somebody by this job description. See, nobody raised their hand, but we're already hiring this new digital workforce with AI agents. And if you aren't,

then others are, and they're probably your competition. So today I'm gonna talk about why we're still governing AI like software, where we look at it in, uh, as a, at a different perspective of looking at it as a digital workforce. Of course, you need all of the software aspects. You need all of the protocols, the standards, so forth. But they're not just necessary. We need something more. We need to look at it in a different way. I had somebody meet me at a show and said, "I'm t-

I'm, I'm head of AI at my organization, and I sit between... I go and talk to, uh, my CTO, and they say, 'Well, it's not really an, a s- a software application.'" And he said, "Okay. So I went to HR and they said, 'Well, that's a software application.'" So we've done this before. This is how we started to look at this. Over the last century, we have been, as enterprises, looking at how we govern the employee workforce, the humans. So we said, "Well, what if we

just take that and apply that onto AI agents and look at them as a digital workforce?" So we call this agent accountability, and there's a framework behind this, and I'm gonna talk to you about the principles at the top of the framework, but there's controls for actually auditing to this framework as well. So we break it down into three stages. We start with the assignment stage. This is what's es- uh, essential to establishing the agent's presence in the organization. So we start with who are you? So if... when you start with a

company, first they gotta know who you are. You gotta get into the registry, into their system. And usually, if it's, like, a secure organization, like many of you are in finance, you have a secure organization. To get into the building, you have to start with a ID badge. And if your ID badge, like mine over at Verizon, if your, uh... When I started with them and had to get in the building, if your ID badge was like mine, it competed with your DMV photo for the worst picture of your life. So

we need to give the agent an identity. Who are you? That identity needs to be one unique ID, needs to be in a known shared registry that's one single point of truth. And that identity is what everything else in this framework is gonna point to. Now, the agent doesn't just work alone. The agent is part of a team. It's a team even if you're using an agent just between you and you're utilizing an agent, it is a team of two. And

if it's part of a team for a specific role or task or that, uh, is a team for, um, uh, a larger team that's a department or just the fact that this agent is in your organization as part of that team. So it needs to get context on the goal, the decisions, the history of that team, and it should be giving context back into that team so the humans and other agents can know what it's doing and what, what, what is its learnings.

And that means it's a teammate and not a digital clone. A lot of people think, "Oh, I'm just spinning up something here with Clyde, and it's doing things on my behalf. Therefore, it is me." That's dangerous. So the next step is we need to look at you've got your identity, but you need to look at who's the, my boss, right? Who is responsible and accountable for what I do in the organization, and who's gonna help me to understand what I'm supposed to do? So we need to have... the agent needs a sponsor who answers for you.

So our Agent 409 that we've already given that identity to is now going to have an accountable party that is a named person. Sven is gonna be the accountable party. Sven defines what Agent 409's purpose and so forth. We'll get into that. H- he owns what 409 costs and what it spends, and he gets the call if anything goes wrong. So there's one name that starts a line of accountability. That line of accountability we already have in

our, our hierarchy of our org chart. But the... it starts with somebody where we enter that. The way I like to think of a line of accountability or even the sponsor role is when my kids were young, we had a pool, and we had a rule that there was always somebody that was accountable for those kids in the pool, so that we'd know if they were drowning, right? And so we had a rule that if I, I was responsible for the kids, that if I needed to go to the bathroom or something, that I would look at some other

adult that I trusted to be with the kids. I would look them in the eye, make eye contact, and say, "Do you have the kids?" And they would reply, "I have the kids." We did that for years, and that's essentially Sven is saying here, "I have the agent." So now we've got our assignment. We know who the, and who is the agent and wh- who's responsible for it. So now we need to align to what is this agent's scope, what is it supposed to do, and how it's supposed to do it. So basically, what are you supposed to do in this job? So we've

all had that job where you started and there was really no job description or a really obscure job description, and you just figured it out and kind of checked in with people and said, "Is this what I'm supposed to be doing?" Um, that doesn't work for agents. They, they're better at give them what are you here to do, a role. So that job description is a purpose, responsibilities of how to accomplish that purpose, and some kind of budget. These are just high level. Of course, a role goes way deeper than this, and that's what the controls on the framework help to define.

But just to give an idea what is the role, these are the types of things we're putting into it. And beyond what it's for, how to do the job, it's also what success looks like, and we'll talk about that in a minute. So there's two types of definition for the role. We've got onboarding versus runtime scope. So a lot of what I just mentioned is that onboarding, what's the purpose and so forth. It's set once, it changes rarely unless we've got to take some recourse. Versus mission, and

this is a word that I'm pulling from some of those tech specs. Mission is like in the new Auth. It's what's happening at this very moment at every time it's making a call, for instance, to an MCP tool. So the, the agent actually defines its mission, but Sven always defines its role.

So we've got, we've got a role what we're supposed to do, but now we need a way to do it. So we need to know what access and authorization we can have to, to what data, to what tools to get the job done. So we're gonna give the agent authority. What can you see and what can you do? So sponsor-- the sponsor, Sven, decides which doors. He's the one that gives the keys. Now, it might be through an IT department or a particular team that runs an MCP

server or a database or whatnot, but Sven is the one that has to open it up for the agent because he's accountable. And this is where we start to set those boundaries and escalations are defined here on what, what, how, what happens with that authority. So let me take a minute and talk about that authority in scope for a second. We always work to the lowest scope. What I mean by that, instead of going through specific scenario that could get confusing, I'm just gonna call the access authority

for Sven. He has access to A, B, C, and D. That could be information he's allowed to see. That could be a tool he's allowed to access, but he has that scope. Agent 203, notice because we have identities I can call out, this is not Agent 409, this is Agent 203. Agent 203's job is to converse with, with Sven, uh, about projects, and Agent 203 has a scope of C and D. And then Agent 409 has D, E, and F. So to get... So when Agent 203 calls on

behalf of Sven delegating a task, Agent 203 calls to 409. 409 can actually go to D, E, and F to get the job done, but it can only return D back up the chain here to Sven, because only D, Sven, 203, and 209 all have access to. So the, the scope of authority never grows beyond what Agent 409 has been given. It always is constricted by what is that,

that lineage. Never assume alignment, meaning we have inherited, uh, uh, alignment here with the model. For instance, if you are using a large language model, built into that is its own alignments, right? Its own rules and its own, uh, definitions. As we know, the hyperscalers, the frontier models have just said they're misaligned. We're gonna tell everybody they're misaligned. So we know that we don't have control over that, and that's locked in from our

perspective. Also, that team context I was talking about, it's coming from somewhere else. So we don't have-- Sven doesn't have the capability to go change that very easily. And then there might be outside agents that are accessing Agent 409 from other parties. As we grow this whole agentic era, we're gonna see this more and more. An outside agent kind of needs to be more like a contractor coming to work in your building. You give it an identity, you give it, uh, authorization, when it's able to access and so forth. But from the perspective of 409, we need to

think that that con-- that, uh, alignment, that context coming in is not something Sven controls. Opposed to Sven can control what is defined for 409, and so we s- he sets up a set of rules that are deterministic. That's the floor. That's kinda like FGA policies and so forth, where you can define specific things. But then you have intents that are probabilistic. So he intends for something to get done, and how do we measure that? Oh, that's nice.

And then, uh, so the inherited alignment, it can change on us without our control even though we can't change it, and the defined alignment becomes a contract for 409. So how do we actually enforce that contract? Well, that's with the administration stage here. So with the administration is where we start to enforce everything before. So if, as an employee, we think we have trust, but also our enterprises and organizations make sure that they verify that trust. So they have

things like, uh, you know, human OS and so forth, the, uh, roles that are making sure that our devices have a mobile device management on it, that they're checking what we're doing, and, uh, we have people checking in with us. We have supervisors that are watching that we're on ali- uh, aligned to what we're working on. So we need to look at that for agents as well, but one thing I wanna call out here is that behaving, that's what we talked about with agents. We say like, "Are they out of scope? Are they accessing things they're not supposed to?" That does

not always, uh, equal succeeding. So we need to think of behaving we defined in authority and succeeding we defined in the role. Are you actually succeeding at what we want you to do? So the first thing we need to do in administration is we need to know if you're doing what you're supposed to do and you're staying within those, those, uh, guardrails of authority. So we set oversight. Are you doing what we expect? Now, that's not a single metric, it's actually a scorecard. So

on the authority side, we have those more deterministic rules like s- are you staying within inside the scoped authority? Are you, uh, you're only appro- approved for, uh, up to $40,000 invoices to be dealing with, something like that, very clear rule and defined. But then on the role, we tend to lean into things that are more that, those, uh, intents. So is your invoice match accuracy up to the par we want? Is your reconciliation time, the cycle time, uh,

within scope? Yes, you can put rules to these, but more and more as we start to define these, we start to look at them and say, we, we need to look at them as a scorecard. So... And the weights will come from the definitions in role and authority. So again, rules are checked, intents are judged. So rules, like I said, these types have never approves a payment, writes only to the ledger, versus the intents are probabilistic and it's like the correct PO was matched, a proper vendor, uh, project matched. And an intent evaluation should

be based on a confidence level. So if we look at that, well, who's actually looking at that? The agents need micromanagers. We as people, sorry, do not like micromanagers, right? We don't want that manager looking over our shoulder. But agents need that. And so we have a micromanager looking at what's going on with the model. We have a manager looking at what's going on with the context coming into that agent. And then we also look at what's the work product coming out. And so there's all

sorts of things we should be looking at in the oversight constantly, and that's why I have these as little guardian agents around here that are actually applying that at machine speed because if you wait for a quarterly review like we do with humans, you are about forty thousand, four hundred thousand, four million decisions too late to have that discussion. So what happens when things start to go awry? We need to take action. Well, if you think about it, if we do something wrong, the first step isn't, "You're fired." Hopefully it's not.

So you, you're gonna d- you're gonna, you know, your supervisor's gonna sit down and say, "Hey, you know what? You misunderstood that. This is what we want, a correction." But if you keep doing it, especially if you're out of your authority, we're gonna start restricting that authority. And then if you keep doing it, we're gonna escalate, and then you're out of here. So we also need recourse, the same kind of model for agents. What happens if you don't do what we expect? So we look at it as

a deterministic grade, uh, when we're talking about like intents, uh, for the probabilistic judgment. We look at like if, if we're pretty confident, then we can take a certain action. If we're low confidence, we can take another. That allows us in the definitions to be-- to set these steps, and the steps allow action. So for instance, if I'm looking at the intents and I've got a confidence score going, I can look at like if I'm ninety percent pretty confident, then I might just redact

content if, if it's out of, out of alignment. If it's less than fifty percent, I need to escalate to a human in the loop. If it's below twenty percent, block it. We're not, we're not confident at all. So by putting in those steps, those stops, we can start to turn that probabilistic into something that can have deterministic, uh, actions that can be, uh, acted on in recourse. But that's opposed to something like you accessed something you're not supposed to access, and, uh, so we restrict authorization at the moment,

or suspend access if you've done it five times today, or su- suspend the whole operation of the agent if it's been doing this consistently for days, so that we-- when we look at a zero trust, we start heavier at first, then we lighten with repeated success. So this is not something you just kind of set and then forget right away. It's something that you wanna be looking at and evolving, and again, at the machine speed is where you can do this practically with agents. So we have the police, the rule, we need a

cop. That's where we've got something like guardian agents, AI gateways, guardrails, where they're doing restriction, prohibit, and suspension at machine speed. And then Sven's job, what we're good as, uh, at as humans, is we come in with our judgment to be able to say, "You know what? If I tweak this role, if I tweak this authority definition in that alignment stage, then I can get a better result. I get more success out of the agent, or I get it to behave better." So Sven adjusts those rules and authority as a coach,

if you think about it that way. So we have assignment, we need to name every agent. We have alignment, we need to bound every action. And we have administration, we need to trace every outcome. That's agent accountability. That's how we reach agent accountability. This was built from the history of humans, tailored for agents. Like I said, there's a, a framework behind this with controls that I welcome you to take a look at, uh, that, uh, you can audit in your

organization, and then you can apply those technical solutions that I talked about as the methodology to achieve. So you can get started today by giving your agents an identity. And one thing I ask when I give this talk is if everybody can do me a favor, give it the worst picture of its life. Thank you very much. I'm Tim Tyler, uh, and, and, uh, feel free to go to agentaccountabilityframework.org or scan the QR code if you want my landing page, and you can check out the, uh, the agent accountability page or, uh, Gravitee

if you want more information. We're also downstairs. Thank you.