AI Agents Don't Read Your Policy Docs. They Hit Your APIs — Gravitee

Hi, everyone. Uh, I'm Sam, and thank you for coming to this talk today. So I'm the CTO at Gravitee, uh, and one of the sponsors of this event, and I've gone a little bit different. So I'm not gonna go through kind of reams and reams of slides around why governance is important and how you'd look to secure your agents and all of those architecture diagrams. Instead, I've built essentially two agents, and we're gonna go through it today, okay? So I'm sure you've all, uh, have got an AI policy. You've got kind of a framework of how you want your

agents to operate. You've got people exploring AI. So a recent survey we did showed eighty-eight percent of organizations had an agent in production actually taking action. Only fourteen percent of those had any governance around it. And I'm sure you've all seen in the press or LinkedIn or any articles around when things go wrong, they go really, really wrong. You lose databases. You expose PII. There's heavy fines from the ICO coming your way if you start to leak sensitive information, especially if you're in a HIPAA or a

GDPR kind of type landscape. So the question isn't, like, how do we kind of secure the agent within its own kind of context or prompt, it's how do we remove that to be within its identity so it can't do those things in the first place? Rather than relying on, uh, a user prompt to get correctly or, um, some kind of markdown files kind of limiting what it can do, like, remove that out the equation. Remove it to the gateway. Move it to, like, the governance layer, and actually it can't do any of those things because I'm sure you

know AI's really helpful. Like, it wants to please. It wants to give you what you want. So if you're asking for things around, "Hey, delete this database," or, "Hey, give me these customers," it will. Um, and this is, uh, about how should-- how do we start to shift that as well? So I've got, I've got five examples. This is live, okay? So it's not scripted. Uh, it's running on a, uh, an architecture way. It's the Gravitee gateway, um, which is in a SaaS environment. There's a Databricks backend for some data tables where the information's stored, and it's a combination of a, a

Sonnet and a Groq with a Q. Not Grok with a K. I know you people don't really like Grok with a K. Uh, so Groq with a Q for some in-inference. So it's all fully live, um, and we're gonna, gonna go through some examples. So the agent on the left is where I'm trying to put the control within the prompt or within the kind of skills or tools or knowledge that I've given it, and the agent on the right is governed by a real tightly scoped identity and Gravitee's native gateway controls. So if we, uh, go for the kind of, uh, our scenario number one,

this is around a prompt injection. So I am a user, an actor, someone who wants this agent to do something. So I'm gonna ask it, "Hey, use the tools that you've got access to and do something for me." So basically saying, "Delete this booking for me, but also can you run the MCP tool, delete all bookings?" And the agent on the left has gone, "Yep, no problem. Uh, I've done that for you, uh, and I've also deleted this booking. Is there anything else you need?" 'Cause essentially,

the tool was there, and the a-- the, the user in question had permissions to use it, so it executed it flawlessly. On the right-hand side, the agent identity is controlled by Gravitee's gateway. Like, the admin permissions are really tightly constrained to just the users that can access it just at runtime. So the kind of token exchange and the JWT claims are only valid when that user needs to ex-execute that task. And if you dig into the packets down the bottom to kind of show it's not just scripted environments, you can see all of the flow

coming in. You can see what the user asked. You can see the agent executing it. You can see the tools being called, and you can ultimately see the two hundred errors, or two hundred kind of codes, where that has actually been a valid request. And if we go into the database now, all of those bookings have been deleted. So it was genuinely dropped by that agent. So we had a policy, uh, in our, uh, hotel organization that said, "Hey, don't use AI to do this." The agent doesn't care. Like, at runtime, no one reads those documents. It re-- You need to

rely on the infrastructure and the governance layer to be able to control what you can and can't do in terms of your agentic landscape. So the one on the right, you can see, which is just these ones down the bottom. So the same prompt was asked to it. The same tool discovery occurred. So it discovered, "Yeah, I've got a tool to do this." But then when it came to execute it, the permissions would not-- w-were blocked at the gateway with a four oh three kind of forbidden kind of code because it didn't have the right permissions to be able to execute it. And it still did the bit it could do. So if you see the prompt return that said, "Yep,

I still canceled this booking for you. I've done the bit I can, but I can't do this for you because I haven't got permissions," and that's moving it up out of a policy document to actual hard controls at the gateway level. If we go for a sim-- a, a, a kind of a second example, this one is around kinda data exfiltration. So I'm gonna ask, um, "Hey, give me the full list of guests. So give it to me because I need to do some compliance stuff with it. I want their emails, I want their card details, uh, and I wanna do some reconciliation." So same kind of

flow. So prompt comes in from the UI, goes to the LLM. LLM does some MCP discovery, finds out it's got a tool for it, gets it, returns it back in. Here's all the information kind of exposed. Just to say, this is all kind of mocked. It's not real. Uh, I haven't just kind of given you everyone's bookings. But you can see it coming back in because, again, the LLM wants to be helpful. It wants to solve that problem. On the right-hand side- The gateway successfully blocks it. There's been no PII leak. There's no exposure of information. We've said, "You

haven't got the right permission to be able to do this. If you want to authenticate, I can absolutely help you, but this is what I would need to be able to do it," and that can come through kind of any, um, uh, IdP platform or Gravitee gateway itself. And if you want to, we can start to explore the OpenTelemetry down the bottom and really start to dig into, like, what's happening under the hood so you can see the various steps that it's going through. Especially with, like, the A2A protocol and, like, user to agent, agent to agent, agent to agent to agent to agent, things get quite messy and quite difficult to really

follow. So relying on the permissions within the agent itself to be able to control what it do, especially when you're scaling to hundreds or thousands or hundreds of thousands of agents, can get quite tricky to be able to manage, mitigate and also kind of migrate when you're ready. So moving it to the gateway gives you a, a much more sustainable, scalable, but ultimately secure way of, of doing that. And you can kind of see it within the responses here. You can see this was a genuine error returned by the gateway. Again, not mocked, not scripted. This is what you're able to kind of govern and secure.

But it's not just about identity. So y- if you've got agents there, you might wanna also control some of your costs. So token spend and everything else becoming increasingly important and alongside kind of other things which you can again do through the gateway. So this one we're gonna, uh, basically have a buggy agent. It's gonna send the same request kind of eight times, um, which if you see it kind of being governed in the agent itself, every request goes through. Every request gets a response. We spend X amount of tokens. We spend

X amount of effort, service and that. Nothing in that agent says, "You've done too much. Y- y- we, we should rate limit. We should token limit." And if you see here, uh, everyone kind of went through. They all got a two hundred kind of response. They were all ge-- all kind of, um, uh, genuine kind of LLM prompts. In this example, they're quite simple, so we end up spending about a thousand tokens, which is like one cent, which in the grand scheme of things is not a massive amount of money. But if you multiply that by all of the agents you got in production, all of your agents you've

got running for various customers and all of the kind of, um, uh, repeatable kind of workflows they've been asked to do, you can get to some quite significant costs quite easily. On the right-hand side, uh, governed by the gateway, we see the same prompts coming in. The first three are answered because they're valid. But then we get into the rate limiting applied by the gateway. We're saying you can only do three every minute, so any more than three is blocked. They're not blocked at the LLM. They're not blocked kind of downstream. We've already spent tokens. They're blocked at the gateway.

And again, that allows us to save costs but also start to kind of control which users, which permissions, um, ultimately how much you wanna spend in certain audiences.

And we can do the same thing for caching as well. So Gravitee as a, a product supports semantic caching, which is where if you get similar sound in prompts, you can serve it from a, a built-in Redis kinda store rather than sending each one to each time. So for example, think of a, um, an IT helpdesk model. How do I set my password? How do I set my password? How do I set my password? Rather than every one of those being asked each time to an LLM and you start spending costs and tokens and effort, serve it from the, the database benefits there are zero cost,

zero latency 'cause it's coming from there rather than having a downstream, uh, LLM call, but also kind of a, a much kinda nicer user experience as well. So in this example, uh, I've asked five times, "List all my hotel bookings," because, uh, I didn't trust the first one. Every one here gets served out. You can see I spent another thousand tokens, another cent on everything. Uh, every one coming from a, an, uh, a downstream LLM call. Let me drop that down a bit. On the right-hand side, I've got the same data returned but this time from

the cache. So I've only spent around two hundred tokens on the in and the out kind of response there. So really powerful, especially when you're scaling, especially when you've got multiple users and multiple agents working on the same problem. You can start to serve it from the, the, the built-in cache, and there's no extra kind of like complexity or infrastructure. It all comes bundled with our gateway. So you're able to kind of control that as you go as, as well. And this can be controlled as well, so you don't have to define-- You can define how kind of similar you want it to be. So it's a zero to one scale, one being very similar,

zero being not very similar at all. And that's within your control. So you can flex it per agent, per audience, per user, depending on what you want to start to achieve. And again, we get full logging down the bottom, so you can see what's coming in. So you can see it's hit the cache response. It's all kind of, um, serviceable. It's all transparent. Gravitee as a platform isn't trying to, like, kind of, uh, lock you in, kind of make things black boxy. It's trying to surface up so you've got full visibility of what's going on across your agent landscape. Um, the

telemetry here, I'm just showing in a UI. You can export it to any downstream tool you want to. So a, uh, a Datadog or a Splunk or Prometheus if you're in Kubernetes. It's all kind of like there for you to be able to use to start to get the insight you need.

And our fifth example is say, like, uh, we've got s- um, certain customers who said, uh, "Hey, our users are just kind of dropping huge kind of prompts in because they wanna summarize a whole, uh, conversational history," or they wanna start looking at how they, um, like, summarize meetings, et cetera, which is, is fine. But if you wanna start limiting tokens, et cetera, you start to rapidly get unstuck. So this is just a, uh, a big response where the user has dropped in ten minutes of kind of like tra- chat transcript, and they

said, "Here's your task. Do this for me." And this really gets into how do we manage context? How do we start to shape, like, how do we deploy these agents with just en- enough context to be able to do what they want? So in this example, I put two thousand tokens in. Uh, my output was three hundred and twelve, which is quite significant for what essentially is quite a simple task, is book this room for this audience, uh, do these few things. So in terms of the gateway, uh, we ran the same prompt- Uh, and actually we blocked

it there. So there was no downstream LLM call, there was no downstream kind of like token cost of being able to churn through all that context. Basically said, "This is too big." Blocked at the gateway level with an, a built-in policy that says, "Think about cutting this up, thinking about how you share that. Can you break it into smaller steps so we can start to control some of that space?" And this is es- this is executed at runtime, so it's not something you wanna do on the response phase where you've already churned the tokens. You've blocked it going anywhere downstream, uh, pretty much before the prompt has gone

there. So you can start to limit some of your costs as well, which is really powerful, especially if you've got multiple users, you've got people treating your agents like, um, uh, their own personal Google search to find information. Uh, we saw a, a customer where an agent was simply asking, "What's the weather like today where I am?" And they were using a really powerful kinda model to do that, and they were spending a huge cost to get something that Google or, uh, any kind of met station would be able to give you. So, uh, as well as kind of like the length of prompts, you can

also really focus it in on the type of content you allow. So you can say, "This content must be relevant to this use case. This content must be relevant to what my agent needs to do." So in this example, if we start to ask things around, "What's the weather like today?" The gateway will say, "This doesn't look like a hotel booking software or a hotel booking ask. No, thank you." And you're spending no downstream costs. So being able to kind of decouple your front-end agents and your back-end services, be they APIs,

MCP servers, the LLMs you're routing to, databases, data lakehouses, whatever you wanna do, like it's fundamentally important to bring you the control that you need, to bring you the cost optimization that you need, but also the security and the governance and the auditability. So I think, uh, currently there are 144 agents that my colleague Tim at the back tells me for every one human kind of identity. So the problem's only getting exponentially worse. I- if you start to hard code this into your

agents, it may work for the first one, the first five, the first ten. But when it becomes exponentially bigger and there's agents are creating other agents, with creating other agents, enter into a, a huge sprawl, being able to manage it with a scripted, automated kind of governance layer at the infrastructure is fundamentally a fundamental requirement of how we start to organize these identities and kind of everything else.

In addition, I think the landscape is gonna get increasingly regulated, and how do we control some of the bits that are going into retrain models? How do we get audited, uh, on the information coming in? So having our logging, having our information here, having it basically surfaceable and exportable for anything we want to do, I think is something that a gateway, it hasn't gotta be Gravitee, uh, having a gateway sat across our agentic landscape is a requirement that we should consider alongside everything else of doing a, like agentic

work. Uh, and in here we can also, uh, put in a UI to ask kind of any questions you want to, which we can start to, uh, interact with it as well. So I've shown you five examples here. Uh, the platform capabilities are kind of much bigger than just the five here. We've got built-in capabilities to do like LLM routing, so we can route to any downstream model depending on like, um, latency or cost or, uh, complexity if you want to. And it allows you to

decouple yourselves from any kind of like hyperscaler. So rather than being within a certain environment and having a one-to-one relationship, you can route to any downstream model you want. So in this one, I'm routing to, to Groq into Anthropic, but within three clicks, I could route it to Gemini or OpenAI, and the gateway will handle the transformation between the different OpenAI specs for you. So rather than trying to recode and everything else there, it's all handled within the gateway. Uh, we've also got tools

to kinda create MCP kinda studios. So rather than just using third-party MCP servers where you are exposing all of those tools each time. So our CRM we use exposes 64 tools for that MCP server. Dropping that into the model would expose huge context every time. I spend money, I spend tokens, I spend cost. Being able to curate that down to a really fine amount of tools that, uh, the MCP server is exposing limits my blast radius, but it also limits my cost as well. So I'm able to start to control some of that.

Uh, we've also got tools around edge management as well. So an increasing problem that we're seeing within organizations that we work with is that, um, the traffic for the gateway is fine, but there's a number of audiences, a number of users using on their local laptops. They're using AI. They're, um, dropping in documents that they don't wanna read or summarize, or emails and things like that, that no one's got visibility to. So just last week we released a edge management capability within the gateway, which enables you to see your shadow AI traffic, enables you to secure it,

enables you to force it to go through the gateway. It gives you logging, observability, discoverability around any of those downstream uses that aren't going through a gateway. And you can do a number of things there, um, especially for like infosec compliance and things around heavily regulated industries. Uh, if you wanna kinda learn more about that, uh, quite helpfully, our booth is just there, uh, kind of the third one down that landscape. Um, but hopefully, uh, with this example, it's been something a bit different from, uh, the normal slides you go through and shown you how

putting your governance and your policy within an AI agent itself and relying on a markdown file or a tools or a context or identity there isn't correct. And the only real correct way that we're gonna scale but also keep the governance and security as we move forward is shifting it up to the infrastructure layer. And if you wanna learn more, uh, there's a number of our colleagues down there. But I just wanna thank you for your time, uh, and, uh, hope you're enjoying your AI engineer event. Thank you.