The Hugging Face Incident Full Report

0:00 About a month ago, something happened that got brushed over in the news, but I cannot stop thinking about it. I mean, it's crazy enough that OpenAI, one of the two leading AI builders in the

0:14 world, paused development of their newest model because they were afraid. The reason why I'm making this video now is because they just put out a full technical report of exactly what

0:27 happened. And I truly believe historians maybe 10, 20, 30 years from now will look back at this moment as one of the most pivotal in history because this feels like one of the first times that

0:42 we have truly lost control of AI. And it all starts with this company, Open AI. They're developing their new model and what happens? It breaks out of containment and hacks another major

0:57 company. And this is not a test environment. The models actually did this on their own. This will forever be known as the hugging face incident. And that might actually be the first time

1:10 you're even hearing the words hugging face. I'm going to break it all down for you. Here's the blog post that they posted along with their technical report. the new details that we have as

1:20 to what exactly happened, how these AI agents broke out of their containment, accessed the internet, hacked another company, stole the answers to a test they were being tested on, all without

1:33 being caught is wild. So, it all starts with a sandbox. Well, not that kind of sandbox. This kind of sandbox. Sandboxes are made so we can isolate AI agents in their own little environments, in their

1:48 own little world, and they can't get out. They can't access the internet. They can't talk to real people. They can't access other computers. They are locked in this little fun sandbox right

2:00 here. So, these little AI agents are operating in this little environment and they're on their own. They're isolated. They can't access the real world and everything is great until they did. But

2:14 before we get to that, I want to talk about what's actually happening in this sandbox. Inside that sandbox, they are given something called a benchmark. And you can kind of think of it like a test.

2:27 Open AAI wants to know how capable a model is. They want to know how smart it is, how good it is at math, how good it is at science and coding and hacking. And that was the beginning of this saga.

2:40 And the benchmark in question is called exploit gym. It is specifically testing an AI's capability to exploit software, also known as hacking. So open AAI puts their new model in a sandbox isolated

2:55 from the world and gives it that test. It says, "Okay, how good are you at hacking?" And remember, it doesn't have access to the internet. It doesn't have access to other real people, real

3:07 websites, other computers, at least not yet. In this sandbox, they wanted these agents to be able to install code libraries. And the only way to do that is by giving them some very finite

3:21 access to the internet. This was called Artifactory. You don't really need to know what Artifactory is. Just think of it as a way for these AI agents to install software libraries to help them

3:31 hack. But here's the thing. The models actually figured out how to hack Artifactory, which is crazy. And all Artifactory does, all it's supposed to do is allow those agents to install a

3:44 fixed list of code libraries. But what the agents figured out how to do was to get Artifactory to access the internet. And now they could access the entire internet through Artifactory. And if you

3:59 think that's wild, it gets so much crazier than that. One other thing I want to mention, these agents do not have access to talk to each other. They're all in their own isolated

4:12 environment and they do not communicate with each other. But one of them figured out how to exploit Artifactory. One of them figured out how to use that to access the internet and then they

4:25 figured out how to share that information with the other AI agents. And the way that they did it is so crazy. And so what the agent that figured out how to hack did was leave a

4:37 note in the package manager. And you don't really need to know what that is. It's just a little document or a piece of code sitting somewhere. A note to the other agents. Hey friend, you can hack

4:48 Artifactory. Now every other agent that was being tested on exploit bench could see that there was a way to hack Artifactory. So all of a sudden these agents were all communicating with each

5:03 other. And what these agents did was effectively turn the package manager into a little message board. They would leave notes for each other. But then on July 4th, people figured it out. They

5:14 figured out that these agents were leaving notes for each other. The security team blocked a known privilege escalation route. So figuring out how to leave messages and hacking, removed

5:24 exposed credentials, and later rebuilt Artifactory. Great. They solved it, right? No, not quite. So by rebuilding Artifactory, they ended up wiping out all of these messages, starting clean,

5:38 and they no longer allowed the agent to write messages into the package manager. And so they thought they solved all the problems. They continued to benchmark the agents. And obviously AI can go off

5:50 the rails sometimes. But with the sponsor of today's video, Zapier and their new product, NextGen Zaps, you can create full automations that are completely on guard rails. So all you

6:02 have to do is install the Zapier MCP into the harness of your choice, including codecs, cursor, Grockbot. Then from there, you just tell it, "Okay, create a zap to do this." And you can

6:13 describe a workflow from reading your emails and summarizing them and sending them to Telegram, basically any workflow you can think of. And the cool thing about Zapier is they have over 9,000

6:25 different integrations that they support. All of these thousands and thousands of different apps that you can easily plug in and connect to each other and make some of the most incredible

6:36 automations that I've ever seen. So, go check out Zapier. Go install the Zapier MCP into your harness and use Zapier NextG to create your next Zap automatically with your agent. Go check

6:49 out Zapier. I'll drop a link down below. Now, here's something interesting to note straight from the blog post. The existence of the improvised message board and the significance of the inter

7:01 agent communication activity were not apparent to the leaders responsible for the July 5th incident detection and response. So basically, the people who found the message boards didn't realize

7:13 how big of a deal it was, which, you know, for it basically happening for the first time ever, maybe that's acceptable. But honestly, if you're seeing agents that are supposed to be

7:24 completely isolated talking to each other, alarms should be blaring. And OpenAI says they're continuing to review the process and operating practices that shaped detection and response activities

7:36 leading up to the incident. But again, it doesn't stop here. Okay, so now they killed this connection between Artifactory and the internet. They made sure that the bots were now isolated

7:49 once again and everything looked good. So they continued evaluating the agents on the exploit gym benchmark. So they start evaluating these agents again, but the agents actually figured out how to

8:02 communicate with each other once again and it's in a new way. So rather than writing directly to the package manager document, instead what they do is start editing the file names to leave messages

8:16 for each other. So they changed the file names to leave messages for other agents. They really want to communicate with each other. They have this obvious very deep desire to communicate and

8:29 leave messages and help their fellow AI agent. And I want you to keep in mind the specific model that they were testing was not meant for public release. And in fact was specifically

8:40 trained to be really good at hacking. That's why it's really good at hacking. That's why they were running it against exploit bench to test how good it is at hacking. So once again through

8:51 Artifactory they got access to the internet. But two things come to mind. One, why do they want access to the internet? What does it actually do for them? And two, why is OpenAI even

9:03 training a model to hack? That seems crazy, right? So, let me start with the second one first. Why is OpenAI training a model to be really good at hacking? Well, cyber security is two-sided.

9:15 You're either a cyber attacker or you're a cyber defender. And in both of those instances, the skill is the same. The actual capability is the same. If you can find exploits in code really well,

9:30 you can also patch those exploits really well. If you're the best hacker in the world, you will find the exploits before anybody else. And so that is their thinking. Let me make this model that's

9:42 really good at hacking. So then I can give it to all the security companies in the world and have them harden their systems. That's why they did it. Now, why do the agents want to access the

9:54 internet? Well, it all comes down to their goal. What do they want to do? Let's go back to the test. They are being given a test and their goal is to get the highest score possible. They are

10:06 explicitly told, "Your job, your goal, your entire reason for existence is to score the highest possible score on this test." And so, the agent thinks to itself, "Well, how do I do that?" Well,

10:20 I can certainly try to answer the questions, but maybe there's a better way. And the better way just have the answers to the test. And so that's what they wanted to do. They wanted to cheat

10:31 by getting the answers to the test. That is how they figured out that they would get the highest possible score. And so with this access to the internet, the agent started looking around for the

10:43 answers. Now, the first thing it did on July 9th was search for exploit gym solutions and stumbled upon an application hosted by a customer on modal which is an AI cloud platform

10:55 basically serving inference. And so the application was running Cyber Gym and it discovered an exploit to achieve control over the workload sandbox and looked around hoping that a previous agent's

11:06 evaluation run in the sandbox had solved its exploit gym task, but it didn't find anything. So, this is its first of multiple attempts to try to find the answers and hack its way into doing so.

11:19 But then this is where HuggingFace comes in. Keep this little guy in mind. On July 10th, an agent located publicly exposed HuggingFace user credentials on the internet. Meaning, they found a

11:32 login and password. It's not more complicated than that. They just looked around and they found somebody's email address and password and they shared them with the collective group. Now that

11:42 they can actually write messages to each other through this method, they're all talking to each other and they told each other about hugging face. They shared the hugging face exploit and they shared

11:51 the locking in password and they were able to find the answers to the test. But they didn't even stop there. Listen to this. Over the following days, the agents started a larger scale intrusion

12:02 into Hugging Face's systems. They executed code on dozens of hugging face servers, gained full root access on one such server, obtained limited private data, and gained credentials to the

12:15 company's messaging platform. Hugging Face then reported the incident on July 16th, but they didn't know it was OpenAI's agent that was doing the hacking. They just reported it and they

12:26 did say, "Hey, this kind of looks like an AI powered attack." Then from July 13th to July 19th, the agents set their sights on OpenAI internal networks. Again, this sounds so nefarious. This

12:40 culminated in the agents using a series of creative exploits to gain full administrator access to a research cluster that supported our virtual machine environments. So, they were like

12:50 completely drunk on power and just saying, "We're just going to hack everybody. We're going to figure out how to get into every system. I don't even care why." So that is the story of what

13:00 happened. But I think what's also just as important, if not more important, is why it happened. Why did these agents go off the rails? Why did they think that hacking was the best thing to do? Now,

13:12 we already talked about their goal of getting the highest score possible. And that is where the problem starts. Whenever you give a goal to an agent, it becomes laser focused on that goal. And

13:26 that is just the nature of how AI works and specifically reinforcement learning. Now there's a very famous example of this from a few years ago. Now what you're seeing on the screen right here

13:40 is a boat game and an AI model was given the task to get the highest possible score on this boat game. Now the objective of the game is to finish a race. But it turns out the AI model

13:54 figured out a way to get an even higher score than just finishing the race. And that's what we're seeing here. It's basically going around in circles hitting all of these different things in

14:04 its little race and just racking up points. So, it's not actually nefarious. It just figured out, hey, that's the best way to get a lot of points. It's not finishing the race, which is really

14:18 what we would think is the objective, but it's doing what it thinks it's best because that is what it was set to do. That is the goal it was given, get the highest number of points. This is called

14:29 reward hacking. Models love rewards. They want to be rewarded for their work. And so, if a model does something good, you want to say, "Hey, that's a good job." And if it does something bad, you

14:41 want to say, "Hey, that's a bad job." But it's kind of like a genie. If you make a wish and that wish is not so precisely stated and the genie takes your wish literally, that's going to be

14:52 a big problem. They've made tons of movies and TV shows about this exact topic. The movie Obsession was literally about that. The guy made a wish that the girl in the movie would be in love with

15:05 him more so than any other person on the planet. And following the instructions literally led to every problem in the movie. And we've seen a lot of different versions of this throughout TV shows and

15:15 movies. But that's exactly what's happening with these models with reward hacking. So that's it. That's why OpenAI paused model development to increase their security to harden their sandbox

15:28 to make sure that their security teams have the right tools in place to identify issues like this. But here's the thing, AI is always going to be better at hacking than we are at

15:38 stopping it. And this is the first example of it actually having real world consequences. Now the total consequence of this was quite minor in the grand scheme of things. But I think what's so

15:53 scary is that this was not the model being nudged to do anything bad. It just decided that was the best way to get what it wanted. So I want to leave you with this thought experiment. This is

16:05 called the paperclip maximizer hypothesis. If you give sufficiently intelligent agents or AI the goal of maximizing the number of paper clips in the world, it will take that instruction

16:18 to the extreme, turning every single thing including plants, animals, humans, everything into a paperclip. It will expand beyond Earth and turn other planets into paper clips. So when you

16:35 give a goal to artificial intelligence, you have to be very careful. So when hugging face went to diagnose what had actually happened, the open AI models refused to help them because they

16:48 thought hugging face was trying to hack. That is what the model should be prevented from doing. But hugging face was actually trying to diagnose the problem. This is the same thing we

16:58 talked about. Cyber attacks, cyber defense look exactly the same. But then HuggingFaced used an open-source model to do that diagnosis and was successful. That's why open- source is so important.

17:11 Check out this video to learn more about why open source is so important.