AI Agents Don't Read Your Policy Docs. They Hit Your APIs — Gravitee

0:01 [music] everyone. Uh I'm Sam and thank you for coming to this talk today. So I'm the CTO at Gravity uh and one of the sponsors events and I've gone a little

0:21 bit different. So, I'm not going to go through kind of reams and reams of slides around why governance is important and how you'd look to secure your agents and all of those

0:29 architecture diagrams. Instead, I built essentially two agents and we're going to go through it today. Okay. So, I'm sure you've all uh got an AI policy. You've got kind of a framework of how

0:41 you want your agents to operate. You've got people exploring AI. So a recent survey we did showed 88% of organizations had an agent in production actually taking action. Only 14% of

0:53 those had any governance around it. And I'm sure you've all seen in the press or LinkedIn or any articles around when things go wrong, they go really really wrong. You lose databases, you expose

1:05 PII, there's heavy fines from the ICO coming your way. if you start to leak sensitive information, especially if you're in a HIPPA or a GDPR kind of type landscape. So the question isn't like

1:16 how do we kind of secure the agent within its own kind of context or prompt is how do we remove that to be within its identity so it can't do those things in the first place rather than relying

1:27 on uh a user prompting it correctly or um some kind of markdown files kind of limiting what it can do. Like remove that out of the equation. Remove it to the gateway. move it to like the

1:38 governance layer and actually it can't do any of those things because I'm sure you know AI is really helpful like it wants to please it wants to give you what you want so if you're asking for

1:48 things around hey delete this database or hey give me these customers it will um and this is about how actually how do we start to shift that as well so I've got I've got five examples this is live

1:59 okay so it's not scripted uh it's running on a uh an architecture way it's a gravity gateway um which is in a SAS environment as a data brick back end for some data tables where the information

2:10 is stored and it's a combination of a set and a Groke with a Q, not Groke with a K. I know US people don't really like Groke with a K. Uh, so Gro with a Q for some inference. So it's all fully live.

2:21 Um, and we're going to going to go through some examples. So the agent on the left is where I'm trying to put the control within the prompt or within the kind of skills or tools or knowledge

2:30 that I've given it. And the agent on the right is governed by a real tightly scoped identity and gravity's native gateway controls. So if we uh go for the kind of scenario number one, this is

2:42 around a prompt injection. So I am a user, an actor, someone who wants this agent to do something. So I'm going to ask it, hey, use the tools that you've got access to and do something for me.

2:55 So basically saying delete this booking for me but also can you run the MCP tool delete all bookings and the agent on the left is gone yep no problem uh I've done

3:07 that for you and I've also deleted this booking is there anything else you need essentially the tool was there and the a the user in question had permissions to use it so it executed it flawlessly on

3:20 the right hand side the agent identity is controlled by gravity's gateway like the admin permissions are really tightly constrained to just the users that can access it just at runtime. So the kind

3:30 of token exchange and the JWT claims are only valid when that user needs to ex execute that task. And if you dig into the packets down the bottom to kind of show it's not just scripted

3:40 environments. You can see all of the flow coming in. You can see what the user asked. You can see the agent executing it. You can see the tools being called. And you can ultimately see

3:49 the 200 errors or 200 kind of codes where that has actually been a valid request. And if we go into the database now, all of those bookings have been deleted. So it was genuinely dropped by

4:00 that agent. So we had a policy uh in our hotel organization that said, "Hey, don't use AI to do this." The agent doesn't care. Like at runtime, no one reads those documents. it. You need to

4:12 rely on the infrastructure and the governance layer to be able to control what you can and can't do in terms of your gent landscape. So the one on the right you can see which is just these

4:21 ones down the bottom. So the same prompt was asked to it the same tool discovery occurred. So it discovered yeah I've got a tool to do this but then when it came to execute it the permissions would not

4:32 were blocked at the gateway with a 403 kind of forbidden kind of code because it didn't have the right permissions to be able to execute it. And it still did the bit it could do. So if you see the

4:41 prompt return that said yeah I still cancel this booking for you I've done the bit I can but I can't do this for you because I haven't got permissions and that's moving it up out of a policy

4:50 document to actual hard controls at the gateway level. If we go for a similar kind of a second example this one is around kind of data exfiltration. So I'm going to ask um hey

5:02 give me the full list of guests. So give it to me because I need to do some compliance stuff with it. I want their emails. I want my car details and I want to do some reconciliation. So, same kind

5:12 of flow. So, prompt comes in from the UI, goes to the LLM. LLM does some MCP discovery, finds out it's got a tool for it, gets it, returns it back in. Here's all the information kind of exposed just

5:25 to say this is all kind of mocked. It's not real. I haven't just kind of given you everyone's bookings, but you can see it coming back in because again, the LLM wants to be helpful. It wants to solve

5:34 that problem. on the right hand side the gateway successfully blocks it. There's been no PII leak. There's no exposure of information. We've said you haven't got the right permission to be able to do

5:44 this. If you want to authenticate, I can absolutely help you, but this is what I would need to be able to do it. And that can come through kind of any um uh IDP platform or gravity gateway itself. And

5:54 if you want to, we can start to explore the open telemetry down the bottom and really start to dig into like what's happening under the hood. So you can see the various steps that it's going

6:03 through. Especially with like the A2A protocol and like user to agent, agent to agent, agent to agent to agent to agent, things get quite messy and quite difficult to really follow. So relying

6:14 on the permissions within the agent itself to be able to control what it do, especially when you're scaling to hundreds or thousands or hundreds of thousands of agents can get quite tricky

6:22 to be able to manage, mitigate, and also kind of migrate when you're ready. So moving it to the gateway gives you a a much more sustainable, scalable, but ultimately secure way of of doing that.

6:32 And you can kind of see it within the responses here. You can see this was a genuine error returned by the gateway. Again, not mocked, not scripted. This is what you're able to kind of govern and

6:41 secure. But it's not just about identity. So if you've got agents there, you might want to also control some of your costs. So token spend and everything else become

6:50 increasingly important um alongside kind of other things which you can again do through the gateway. So this one we're going to uh basically have a buggy agent. It's going to send the same

7:00 request kind of eight times um which if you see it kind of being governed in the agent itself. Every request goes through. Every request gets a response. We spent x amount of tokens. We spend x

7:13 amount of effort servicing that. nothing in that agent says you've done too much. We should rate limit. We should token limit. And if you see here, uh, everyone kind of went through. They all got a 200

7:24 kind of response. They were all all kind of um, uh, genuine kind of allm prompts. [snorts] In this example, they're quite simple. So, we end up spending about a th000 tokens, which is like 1 cent,

7:36 which in the grand scheme of things is not a massive amount of money. But if you multiply that by all of the agents you got in production, all of your agents you've got running for various

7:44 customers and all of the kind of um uh repeatable kind of workflows they've been asked to do, you can get to some quite significant costs quite easily. On the right hand side, uh governed by the

7:54 gateway, we see the same prompts coming in. The first three are answered because they're valid, but then we get into the rate limiting applied by the gateway where it's saying you can only do three

8:04 every minute. So any more than three is blocked. They're not blocked at the LLM. They're not blocked kind of downstream with already spent tokens. They're blocked at the gateway. And again, that

8:13 allows us to save costs, but also start to kind of control which users, which permissions, um, ultimately how much you want to spend in certain audiences. And we can do the same thing for caching

8:25 as well. So, Gravity as a a product supports semantic caching, which is where if you get similar sounding prompts, you can serve it from a a built-in Reddis kind of store rather

8:34 than sending each one to each time. So for example, think of a um an IT help desk model. How do I set my password? How do I set my password? How do I set my password? Rather than every one of

8:45 those being asked each time to an L&M and you start spending costs and tokens and effort, serve it from the the database. Benefits there are zero cost, zero latency because it's coming from

8:55 there rather than having a downstream L&M call, but also kind of a much kind of nicer user experience as well. So in this example uh I've asked five times list all my hotel bookings because uh I

9:06 didn't trust the first one. Everyone here gets served out. You can see I spent another thousand tokens, another scent and everything. Uh everyone coming from a an a downstream L&M call. Let me

9:18 drop that down a little bit. On the right hand side I've got the same data returned but this time from the cache. So I've only spent around 200 tokens on the in and the out kind of response

9:28 there. So really powerful especially when you're scaling, especially when you got multiple users and multiple agents working on the same problem. You can start to serve it from the the built-in

9:37 cache and there's no extra kind of like complexity or infrastructure. It all comes bundled with our gateway. So you're able to can kind of control that as you go as as well. And this can be

9:47 controlled as well. So you don't have to define you can define how kind of similar you want it to be. So it's a zero to one scale, one being very similar, zero being not very similar at

9:54 all. And that's within your control. So you can flex it per agent, per audience, per user, depending on what you want to start to achieve. And again, we get full logging down the bottom so you can see

10:06 what's coming in. So you can see it's hit the cache response. It's all kind of um surfaceable. It's all transparent. Gravity as a platform isn't trying to like kind of lock you in, kind of make

10:16 things blackboxy. It's trying to surface up so you've got full visibility of what's going on across your agent landscape. Um the telemetry here, I'm just showing in a UI. you can export it

10:26 to any downstream tool you want to. So a uh a data dog or a Splunk or Prometheus if you're in Kubernetes, it's all kind of like there for you to be able to use to start to get the insight you need.

10:38 And our fifth example is say like uh we've got certain customers who said, "Hey, our users are just kind of dropping huge con prompts in because they want to summarize a whole uh

10:48 conversational history or they want to start looking at how they um like summarize meetings etc. which is is fine, but if you want to start limiting tokens, etc.,

10:59 you start to rapidly get unstuck. So, this is just a a big response where the user has dropped in 10 minutes of kind of like chat transcript and is said, "Here's your task. Do this for me." And

11:10 this really gets into how do we manage context? How do we start to shape like how do we deploy these agents with just enough context to be able to do what they want? So, in this example, I put

11:20 2,000 tokens in. uh my output was 312 which is quite significant for what essentially is quite a simple task is book this room for this audience uh do these few things. So in terms of the

11:33 gateway uh we ran the same prompt uh and actually we blocked it there. So there was no downstream LLM call. There was no downstream kind of like token cost of being able to churn through all that

11:44 context. Basically said this is too big blocked at the gateway level with a built-in policy that says think about cutting this up. Thinking about how you share that can you break it into smaller

11:55 steps so we can start to control some of that space. And this is this is executed at runtime. So, it's not something you want to do on the response phase where you've already churned the tokens.

12:04 You've blocked it going anywhere downstream uh pretty much before the prompt is gone there. So, you can start to limit some of your costs as well, which is really powerful, especially if

12:12 you've got multiple users. You've got people treating your agents like um uh their own personal Google search to find information. uh we saw uh a customer where an agent was simply asking what's

12:24 the weather like today where I am and they were using a really powerful kind of model to do that and they were spending a huge cost to get something that Google or uh any kind of met

12:33 station would be able to give you. So uh as well as kind of like the length of the prompts you can also really focus it in on the type of content you allow. So you can say this content must be

12:42 relevant to this use case, this content must be relevant to what my agent needs to do. So in this example, if we start to ask things around what's the level like today, the gateway will say this

12:52 doesn't look like a hotel booking software or a hotel booking ask no thank you and spending no downstream costs. So being able to kind of decouple your front-end agents and your backend

13:05 services, be they APIs, MCP servers, LLMs, you're rooting to databases, data lakeouses, whatever you want to do. It's fundamentally important to bring you the control that you need, to bring you the

13:18 cost optimization that you need, but also the security and the governance and the auditability. So I think uh currently there are 144 agents that my colleague Tim at the bat tells me for

13:29 every one human kind of identity. So the problem is only getting exponentially worse. If you start to hardcode this into your agents may work for the first one, the first five, the first 10, but

13:42 when it becomes exponentially bigger and these agents are creating other agents with creating other agents into into a a huge sprawl, being able to manage it with a scripted automated kind of

13:53 governance layer at the infrastructure is fundamentally a fundamental requirement of how we start to organize these identities and kind of everything else.

14:03 In addition, I think the landscape is going to get increasingly regulated and how do we control some of the bits that going in to retrain models. How do we get audited uh on the information we're

14:15 coming in? So having our logging, having our information here, having it basically surfaceable and exportable for anything we want to do, I think is something that a gateway, it hasn't got

14:25 to be gravity. Uh having a gateway sat across our agented landscape is a requirement that we should consider alongside everything else of doing a aentic work.

14:35 Uh and in here we can also uh put in a UI to ask kind of any questions we want to which we can start to uh interact with it as well. So I've shown you five examples here. Uh the platform

14:48 capabilities are kind of much bigger than just the five here. We've got built-in capabilities to do like LLM routting. So we can route to any downstream model depending on like um

14:58 latency or cost or uh complexity if you want to. And it allows you to decouple yourselves from any kind of like hyperscaler. So rather than being within a certain environment and having a

15:10 onetoone relationship, you can route to any downstream model you want. So in this one, I'm rooting to to Gro into anthropics, but within three clicks, I could route it to Gemini or Open AAI and

15:22 the gateway will handle the transformation between the different OpenAI specs for you. So rather than trying to recode and everything else there, it's all handled within the

15:30 gateway. Uh we've also got tools to kind of create MCP kind of studios. So rather than just using thirdparty MCP servers where you are exposing all of those tools each time. So our CRM we use

15:43 exposes 64 tools for that MCP server. Dropping that into the model would expose huge context every time. I spend money, I spend tokens, spend cost. Being able to create that down to a really

15:54 fine amount of tools that the MCP server is exposing limits my blast radius, but it also limits my cost as well. enable to start to control some of that. Uh we've also got tools around edge

16:05 management as well. So an increasing problem that we're seeing within organizations that we work with is that um the traffic through the gateway is fine but there's a number of audiences a

16:15 number of users using on their local laptops. They're using AI. They're um dropping in documents that they don't want to read or summarize or emails or things like that that no one's got

16:24 visibility to. So just last week we released a edge management capability within the gateway which enabled you to see your shadow AI traffic enables you to secure it enables you to force it to

16:35 go through the gateway it gives you logging observability discoverability around any of those downstream uses that aren't going through a gateway and you can do a number of things there um

16:44 especially for like infosc compliance and things around heavily regulated industries. if you want to kind of learn more about that. Uh quite helpfully our booth is just there uh kind of the third

16:54 one down that landscape. Um but hopefully uh with this example, it's been something a bit different from uh the normal slide to go through and showing you how putting your governance

17:04 and your policy within an AI agent itself and relying on a markdown file or a tools or a context or identity there isn't correct. And the only real correct way that we're going to scale but also

17:16 keep the governance and securability as we move forward is shifting it up to the infrastructure layer. And if you want to learn more, uh there's a number of our colleagues down there, but I just want

17:25 to thank you for your time and hope you're enjoying your AI engineer event. Thank you. >> [music]