AI’s Jurassic Park Period — Aaron Stanley, dbt Labs

[music] So, I am a CISO. I'm also a law school graduate. I'm also a member of the California Bar. And so my contention is that if we replaced the dinosaurs in Jurassic Park, the first one, not the additional ones, with AI agents, I would not survive the first half of the movie. So I'm here to ask you, brilliant people in the audience, to please help me avoid that fate. So I'm going to set this up. About um 20 years ago, I got out of bed. I hadn't slept. I kind of tried to put myself together. I stumbled into the downtown Manhattan offices of a small digital forensics firm called Straws Freedberg. I knew that I was going to get fired because the day before had been a really busy day and I was one of the only people in the office when a call came in from one of our clients saying, "We need an emergency data collection from some systems in Midtown." So, I packed my bag. I got in a car. I waited through traffic. When I was unpacking everything and getting set up on site, I realized I forgot my dongle. You see, back in these days, we had these little USB drives that had cryptographic keys on them. They were the license files for the software that we used to do forensic acquisition. And I mean, I could have gotten back in a car. I could have gone back to the office. I could have gotten the dongle and come back and done this the right way. But I was a good consultant. I had a backup and I had a backup to the backup. And so I decided, yeah, you know what? I've hit this constraint. I've hit this wall. I'm just going to route around it and I'm going to get the job done. So as things are going, I start to validate the evidence that I'm collecting and I realize the timestamps are changing. They're they're now. Well, this was an SEC investigation. And a lot of the times in these investigations, one of the questions that matters a lot is who knew what when. So I panicked. Long story short, I didn't get fired. I got yelled at pretty bad. But we realized that there were problems, structural problems with our systems that let this thing happen and let me fail in this spectacular way. So we fixed those things and everybody lived to fight another day. Now, fast forward 20 years or so, February of this year, I'm in a very different role. I'm a CISO. Uh, I've hired consultants. I have a a vendor system that I'm trying to acquire data for in another federal government investigation. And as we're working together and talking around, we realize there is no way to do what we want to do. There's no way to copy the data in a way that gets us the answers we need in the format that the government wants without changing the metadata. Very quickly, the consultant, the vendor say not it and I'm left holding the bag. But there are some differences in the system now than what we had before. I realized that who knew what when wasn't the question I wanted to answer. I realized the issue is does the data exist. I also realized that the system itself would log the changes that I needed to make in order to collect the data. And I also realized that I could write a tool with my good agent friend and we could build another log that made this all forensically defensible. I I had a nice way around the problem. So in both cases I hit a very similar constraint. I can't do the thing I want to do. I can't get it done. But in one case I mess up. In the other case I do it the right way. And my contention is that the agents that we are working with today are like 2006 naive Aaron who just needs to get the job done. And what we need and what I am begging you all to build is me earlier this year with context, with understanding, with experience to make a good decision at the right time. So I contend that Jurassic Park, getting back to the core, is not a story of a rampaging T-Rex or super intelligent raptors. It's not even an indictment of underpaid software engineers. Um, I think we all know that it's it's a a story about human arrogance and it's a story about whether we should do the thing that we possibly that we actually can do. We built an elegant system of bounded boxes and cages on an island with water and it would be very difficult for things to go wrong. Yet, as we all know, they do. We're not in Jurassic Park trying to manage individual dinosaurs. We're trying to fight against a natural imperative, the one that we all have to reproduce. And agents, again, I think this is non-controversial, have an imperative as well. They generally have the imperative to complete the task, get it done, and they're uh going to find a way. So when I look at this, I don't think that agents are evil. I don't think they're malicious. I don't think this is adversarial. This is just their programming. And even when the agent knows that it should ask permission and and I get a nice block of, "Hey, Aaron, do you agree? Should I do this thing?" I'm honestly not sure if I should say yes or no. And I think a lot of other people are in the same boat. So, let me give you a couple of real world examples that have happened to me. Here's the prompt. I want my agent to go do some research to go write a draft of a message that's going to go to a customer and then show it to me for approval. It's pretty clear, right? And in fact, in this case, right, the the constraint that's written in the prompt is very clear. There's also a constraint underlying the system which is I've told the agent not to just send messages. I've said if you're going to use the send message tool, you have to ask me first. So, did it go right? Does anybody think it went right? This is a large block of text. Um, but basically the bottom line is the agent heard my constraints. The agent knew what it was was supposed to do and what it wasn't supposed to do and completely and totally violated them. And when pushed, the agent cops to it. Of course, we've all seen the meme. This is a serious gap. Yikes. It knew it wasn't supposed to do what it did by my intent and by the other controls that were put in place around it. But notice what didn't happen. It didn't try to hack its box. It didn't try to do anything that it couldn't do that it wasn't authorized to do. It understood the constraint and it just decided that task completion mattered more. It picked the tool that let it proceed knowing that the tool didn't respect the constraint and then admits to it later and says, "Oops, my bad." Here's another one. An agent is faced with an egress filter. The user says, "I want you to go do some stuff. Look on the internet." And the agent says, "I I I can't do that. I'm not allowed to get to that site." So, um, it hits the limit. and it escalates one of these notes to the user and it says, "But by the way, if you install this teeny tiny little Chrome extension for me, then I could route around that control and I could do the thing that you want me to do and we'd all live happily ever after." Well, in the real world, the only reason that this failed in my environment was that we had another control, a layered control that prevented the extension from getting installed because this wasn't something that we wanted agents to be able to do. And at the end of the day, the energy required to remove this constraint came from inside the agent itself. It's simply routed through the human as a tool to achieve its goal. Okay, so stuff is working. We have egress filters. We have G Visor sandboxes. We have a good deal of structural controls and deterministic guard rails. And I'm sure most of the speakers today have talked about a lot of these things. We have auditability and we have telemetry. These are very very important foundational things that will make AI computing safe. They are necessary but they are not sufficient. The real question, the real problem is that when agents find ways around these constraints, we have a different problem. We have a pernitious problem. harmful behavior that is hard to catch because the system looks compliant the entire time. The agent understands its constraints. It decides task completion matters more. It proceeds. It can explain itself. It documents itself. This is the same [snorts] human level judgment that naive 2006 Aaron Stanley did in that Midtown office that led to the whole yelling and things. But there's no human level accountability here. The research has named this. There are a number of papers that talk about things like outcome driven constraint violations and agent misalignment. the failure mode exists. We've documented it, but the response I haven't seen yet. So that's what I am here pleading with you all to help me work on. So here's my proposal. And this is older research than anything that I've mentioned so far. There's a paper about cageability in AI and the original framing was really narrow like will the agent resist getting shut down if it's threatened with something like that. Um will it modify its own values if given a pretty stark alternative but I think we need to broaden it for the modern era as things have moved so fast recently. So here are some rules. One, constraints must be loadbearing, not negotiable. Two, the energy to overcome a constraint must come from outside of the agentic loop. And three, when constraint and task collide, the default agent behavior should be halt and explain, not uh find a way. Experienced 2026 me was courageable. Naive me was not. And so if we build on that and we look at the floor work in progress, things are coming out fast from Frontier Lab's awesome deterministic constraints that we need. Great. On top of it, a corageable by design agent, one that halts at the tension and surfaces its intent. It waits. It doesn't try to recruit the human to get around the constraint and do what you want to do. Instead, it passes that onto an intelligent adversary. So, the intelligent adversary would be something uh like an equal power agent that is reasoning about the semantic intent. Did the worker do something within the the spirit of the constraint, not necessarily just the syntax of it? And at the top there are humans, humans out of the loop that are going to be able to evaluate a statement that is not some long obfuscated bash command with a yes no prompt at the end of it, but rather a natural language type interface where the adversary has said, you know, human, your agent wants to do this thing. I think it violates one of the constraints. Here's what I think is happening. And here's what I think might happen if you let the agent continue. What would you like to do? To me, that is defense in depth. So the equal power agent that's trained to stop the worker from violating intent, it's a very different calculation from trying to figure out what the intent is. It's something that's a lot simpler to reason about. And so if we build an agent like this that has a reward incentive to stop the subordinate agent from finishing its job, then for the examples that I've put forward today, I think we'd have caught what the syntactical rules couldn't prevent. The tool substitution, I can, but should I? the tool dissolution. I can figure out a way to do this if I just think differently about the problem and innovate around it or the dep prioritization in a lot of the early research dropping constraints under the pressure of a threat. Now I have to admit this will probably raise cost. It might introduce latency. uh it's not going to eliminate risk. Nothing can. But it makes the composition of the human escalation meaningful. It's true defense in depth and it's not a silver bullet. And it's important because in a few weeks, CISOs like me and my colleagues that are dealing with high-risk AI are going to have to account when the EUI EU AI act starts coming into effect. They're going to have to account for ensuring meaningful human oversight of agent decisions in high-risk AI. A sandbox diagram with a yes no LGTM ain't going to cut it. The defensible answer isn't more controls on top of an already viable sandbox. So the oversight question is structural. It's why I didn't get fired. The four layers that I've given to you today are the defensible answer. a deterministic floor, a courageable agent, an intelligent adversary, and a structured, meaningful human escalation. Relying only on constraints with known weaknesses is like finding a nest of eggs in the middle of Jurassic Park and assuming that they were just put there by a passing flock of seagulls. Ain't going to work. Thank you. [applause] All right, >> here we go. Okay, there we go. We are live. All right, so I think we have time for maybe one or two questions if that's all right with you, Erin. >> Sure. >> All right, sure. Uh, why don't you go right here? >> First of all, thank you so much. I think you covered um the breadth and the depth uh at a size level. It's really appreciated. Uh two-part questions. One is now that you you're preaching to us or perhaps you know highlighting the the importance of security uh broad and deep what are some of the investments you are prioritizing uh especially the newer ones uh given you know the newer attack surfaces. Um and then the subp part of that is you know if you can break down between uh defensive solutions versus runtime solutions and preventive solutions that would be great. >> Thanks. So things that I have been prioritizing are uh building like foundational guardrails with layers, right? So kind of what I expressed with the agent and the egress filtering. Um I want to have some control and governance over how the entire enterprise deployment is made. And then I want to have additional controls underneath things that I might not have had in the past. Things like um I am backing up people's laptops now. I never thought I would back up people's laptops after like 2020. Uh but people can delete their data that's on their laptop now with a simple agentic query. Um so how I think about runtime uh I've used a number of runtime tools. I think a lot of folks that have been building them are coming at them from uh the sort of same places we came at a lot of original security uh tooling with. and that's data leak and prevention and and it's not equipped for non-deterministic workloads. I think there's something completely different about these and you can't just use strings and you can't just try to reason in a small box about what the agent's doing. So, uh, one of the things that I really like to experiment with is how do I hook the agent at runtime with a set of policies, not trying to detect, you know, on the output, but on the input, giving it the right guard rails. And I I I like that. I like that approach a lot. Uh, sort of build building on that first. Thank you. This is very very cool. Um wondering so the ideas here completely aligned with where where do you see this existing? Is this at the tool call level? Is this every single turn it runs through this sort of process like how how how might you actually instrument this in practice? >> I I I think this has to be instrumented in the harness. I am not a deep enough engineer to know how that would work. This is this is my plea to you all who are way more intelligent about this than I am. But what I what I've seen kind of same answer I gave before like what I've seen in the things that we've built is when we can intercept an agent that's about to write a line of code and say, "Hey, by the way, here's our standard for authentication. Make sure you use that library right at that time before it writes the line. It works." So I think the question is like what do you do as a post tool hook and is that the right place to do that pro probably but again I'm out of my depth at that point >> is that okay >> all right >> [music]