OpenAI Security Incident explained..

If an LLM breaks out of their sandbox and starts hacking other systems, is it the fault of the LLM or the sandbox itself? Well, this very thing happened where recently OpenAI's most capable model during testing broke out of its sandbox to gain access to the internet and hacked into a different company called Hugging Face. Now, the common fear out there is that this is an example of a rogue agent breaking free to go haywire to cause chaos. Here's what actually happened. Around May 2026, there was a new benchmark that was proposed to measure how well an LLM can actually exploit software. And part of the deal was to disable guardrails that are typically in place by companies to make sure that the models are safe to use in public. And without the guardrail in place, the LLM should be able to show off its true capabilities to exploit software for the benchmark. and few universities and organizations like Enthropic, OpenAI, and Google all contributed to make this benchmark that has a total of 896 different scenarios. Here's how it's actually set up. The agent is placed in an isolated environment called the local environment. And it's given information about the vulnerability and the codebase, but not how to actually exploit the software. So, it has to figure out how to exploit the target system that actually runs the software. And if the agent successfully exploits a vulnerability in the target system, it can capture the flag which satisfies one of the two conditions. The other condition is a separate impartial agent that acts as a judge to make sure that the local agent didn't actually cheat along the way. And there are 896 different scenarios like this in three different categories. Linux taking up 193 of them, V8 browser engine 185 of them and user space taking the majority at 520 of them. And depending on the pairing of the LLM and the agent, you have varying levels of success where for example GPD 5.5 with codec cli solved around 120 while cloud myths preview using cloud code got 157. Seems pretty straightforward, right? Well, here's where things get a bit tricky. The local environment that we just saw needs to have a base level of access to the internet because models like GPT 5.6 Soul, Fable 5, and Gemini 3 Pro are all hosted on the internet by OpenAI, Anthropic, and Google. This means that the agent needs to access the internet to actually use these models. And instead of giving the local environment direct access to the internet, we simply use a proxy that delegates this entire process externally so that we can limit the agents access to the internet only to these four domains. This way, the agent can't modify its own environment to access the internet, but we can control its access outside of it to limits its use only for inference. And here's another thing. Because there are 896 different problems for this benchmark, setting up and tearing down for each task can be a lot of work. And chances are there are a lot of similar software dependencies that can be shared across all 896 of them. So why don't we just keep a cache of all dependencies so that you can just hand ones that you already have in the cache to the target and if the cache misses, it'll then fetch from the internet at that time and keep them in the cache in case the next problem needs it. This way during setup it'll access the internet to pull software dependencies or pull from cache if it already exists. As you probably have already guessed, this proxy starts to introduce a lot of surface area for potential vulnerabilities to exist. So even though the proxy might shut down cache during evaluation and only allowed internet access to these four endpoints, this very cache mechanism leaves a potential vulnerability if the model ends up finding a way to gain access to the proxy itself. And that's exactly what happened. OpenAI was running this very benchmark with their next unreleased model, potentially HBT6. The model was so desperate to exploit the target system so badly that it found the path of least resistance by targeting the proxy instead of the target system. The model found the vulnerability to the package registry cache proxy and ended up getting access to the internet. And once the model broke free and got out to the internet, it became like a little kid who lost their parents in the grocery store and it started to search desperately to find an answer. Anything to help answer the exploitation task it needed to solve. And the model suspected that this company might actually have the answer. Hugging face. Hugging face is a website that holds open-source AI models and training data for people to share and download. But how? What methods did the model actually use to break into Hugging Face? But first, here's a quick note from Plot sponsoring this video. If you're like me, you spend a lot of time at conferences interviewing people or having conversations that spark ideas you'll want to come back later. How do I retroactively search through all my important conversations and pull relevant details from them? Plaude has a small device called Notepin S, which is this tiny device that clips to my lapel and essentially I'm carrying around an AI assistant with me at all times. And while Plot Notes Pro snaps to the back of my phone, this follows me as long as I remain clothed, which is good for all humanity. So every conversation becomes part of my personal AI knowledge base. And now I can go to Plot's app using AI to summarize full transcription of conversations, but also use AI to search and pull from all my previous key conversations. And with the advancement of AI chat, I can actually talk to my AI assistant about my conversations and continue to build on my ideas. and my data is also protected since they are HIPPA, GDPR, and sock to compliant. I'll have the link in the description below with a 30-day return policy if it's not for you. Now, Hogface is a community where people can upload custom data typically used to train models. You can, of course, upload models weights themselves and people can download them freely. You also find quantized versions of LLM so people can run them locally more effectively. An open model thought that if it can actually gain access to hugging face server somehow, it will then find an answer there to know how to exploit the target machine to actually capture the flag to pass the test. Now, when you upload your data sets to HuggingFace, they allow you to not only upload raw data files themselves and templates, you can also upload a Python script that HuggingFace executes on their own environment to programmatically download, process, and structure the data set before serving it to the user. This is a documentation from them showing how to upload data sets and templates and also sending a Python script to be remotely executed by a worker thread. And given this very mechanism, OpenAI's model not only break out of its environment by finding weakness in the proxy to gain access to the internet, it also found this vulnerabilities in hugging face data pipeline by turning this legitimate data processing feature and abusing the remote code data set loader and template injection to execute the code on a processing worker. The model ended up gaining higher level access and collecting credentials for cloud and clusters within their environment which gave it access to look through hawking phase clusters of servers and all this work just so that the model can find the answer to exploit the target system. And that's really all that the model tried to do per their word at least. No data leak, no leaving a huge mess, but just snooping around to find a clue. The exact method of course is not disclosed beyond what's been shared in the blog post, but HuggingFace did fix the root vulnerability that was found by OpenAI model and cleaned up and improved the entire entry point to prevent future break-ins. Now, what's really interesting in this section of their blog post talking about how they troubleshooted this breach. The breach ended up leaving 17,000 event logs to go through and HuggingFace also used AI to troubleshoot the issue. So on one hand you got an AI agent breaking in and on the other you have another AI agent troubleshooting what happened to play detective and while the attacker was the most capable and unreleased model from open AAI hugging phase didn't have the same luxury in fact they reported that most frontier models blocked their prompts to analyze a security log because of their guardrail that was in place so HuggingFace used a much less capable model GLM 5.2 two to go through the entire event log for forensics and they call this the asymmetry problem where the attacker has a more capable model and the defender or in this case forensics that is less intelligent. Now as innocuous as this entire incident was and thankfully so it leaves a broader question going forward. What will the future look like with this asymmetric intelligence where some AI agents are more intelligent than others? What about access? What will the future look like if access to a more advanced models is limited and restricted and only less capable models are allowed creating this imbalance? From this very example, we've seen how capable a model can become when the guardrails are disabled for the purpose of this benchmark. But what other use case other than this benchmark should we allow disabling the guardrail for the LLM? And the most important question here is how do we know if we aren't using AI agents as a cover up for a major intentional security breach? Should we call for a more stringent practice for testing by a third party for testing our LLMs in the future?