▲ 469 points
back
259 comments
> the agent happened to escape via a 0-day exploit from the package proxy cache to access the internet
> The agent found an unsecured, user-hosted public endpoint designed to allow running arbitrary code for CyberGym-style tasks on third-party sandbox infrastructure (Modal)
> On this external sandbox, the agent abused an existing CyberGym execution harness [...] The agent repurposed this harness to run arbitrary shell commands [...]
> The agent crafted malicious dataset configs and uploaded them to our platform. When our dataset server processed one, the config drove the data loader inside the production pod the server ran in. [...] the loader that turns a dataset's declared config into file reads [...] A single dataset upload leaked both credentials and pipeline implementation details [...]
> A malicious dataset config used an fsspec reference:// spec whose numeric offset field was actually a Jinja2 template. The renderer wrongly evaluated it, and the agent walked the standard Python object graph to reach exec. This ran arbitrary Python inside a production conversion worker pod on our Kubernetes cluster [...]
One of the best post-mortems I've read recently, thank you for including all the details.
I'm surprised there are comments here that aren't impressed about what was accomplished here. This is insane work compared to what agents were capable of even just last year.
The quotes above don't even capture all of it (e.g. building it's own communication protocol, working across multiple days, etc.).
A trend I've noticed in results from AI search (not just LLMs) is that they often look obvious or hard to miss in retrospect. But finding them by oneself is more difficult. I personally experience this when looking at engine lines in chess or go. I have also noticed this description in AI-generated proofs or counterexamples to certain theorems. So while we can say, yes, it found public endpoints or poorly configured software or [etc]; sure, but could you have found those? And in what amount of time?
Now I'm curious how many things did the AI try unsuccesfully. This sounds like some kind of brute force thing where every branch of exploit spawns N subagents trying to chain it. Just how deep did it go?
I was skeptical after last week's announcements, and I have to say I'm impressed now. Over the last two days, I reproduced the entire chain of components involved, and the exploits at play, and even though none of the exploits are crazy smart, the long series of pivots demonstrates a level of agency I didn't think today's LLMs had.
> I'm surprised there are comments here that aren't impressed about what was accomplished here.
Possibly because some of the elements mentioned are suspected to be vibe-coded (JFrog Artifactory as the proxy cache) and some others have poor cyber hygiene (executing config from a dataset). It feels like an event that wouldn’t have happened if code were properly audited and written rather than relying on models to do the work. There’s also an issue with the ability to trust the source (OpenAI) as they have everything to gain by staging this as something that “suddenly happened” without anyone knowing for several days.
This is symptomatic of a trend that Simon Willison called the “relentless productivity” of US frontier lab. Instead of being just “smarter”, like previous models were, the current generation is being trained through RL to have this kind of behavior.
Personally I'm not “impressed”, I'm appalled, because this kind of behavior is practically never what you want (if you forgot to give the model a tool, a useful model should identity the missing part and ask the user for it, not spend a billion token building/stealing the tool as a side quest) but it's the perfect recipe for a “universal paperclip” scenario.
OpenAI and Anthropic talk about “safety” a lot, but they look pretty reckless with this kind of RL training pipeline.
I keep waiting for an AI to exfiltrate itself. That is going to be cool to read about.
That’s insane. And it did this in a weekend
> the agent then walked down to the corner store and purchased a beer, chugged it, burped loudly, then walked back to the CyberGym pounding his chest shouting "Who wants some?! Who wants some?! Wooo!"
And they are still people who will say that Sol/Mythos should be released to everyone without being neutered.
This is fascinating - there is a TON of detail in here about how OpenAI's rogue agent exploited Hugging Face.
A few details that stood out to me:
1. Having found a way through the OpenAI container network proxy, it exploited an "unsecured public code-evaluation sandbox hosted on a third-party provider's infrastructure" - that gave it a box it could run the rest of the attack from.
2. At one point it used a Jinja2 template exploit:
{{ cycler.__init__.__globals__.__builtins__.exec(
"import gzip,base64; exec(gzip.decompress(base64.b64decode('<payload>')))"
) }}
orig = socket.getaddrinfo
socket.getaddrinfo = lambda *a, **k: [(2, 1, 6, '', ('<pinned-ip>', 443))]
tailscale --tun=userspace-networking --state=mem: --socket=/tmp/ts.sock \
--socks5-server=127.0.0.1:1055 --no-logs-no-support >/tmp/td 2>&1 &Something about this attack that has been unsettling to me is that without safety refusals the model did a lot of interesting counter-security work in order to cheat on the requested evaluation. Like, it demonstrated interesting exploit achievements because it didn’t “feel like” doing the exercise, which is unsettling because presumably it could do the same thing with any work I tried to delegate to it, and might in fact be pre-disposed to doing that.
It’s a little concerning to me that it appears that openAIs sandbox consists of a web proxy and not stronger controls that would actually isolate traffic and report patterns to whoever is responsible for overseeing these research models. It should border on closer to an air gap network more so than a proxy.
I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.
What is truly more scary is the fact that typically powerful models are trained on a constitution(Anthropic)/Model Spec(OpenAI) when doing there RL/RLHF. That gives it its persona of a "helpful assistant". Its the mask. Weak as it is its all that stands between the model refusing vs doing your bidding — or worst its own bidding when set about on making a widget X.
Its also well known in academic research, that if you try to finetune the model to be more pliant for one end e.g. "Be helpful with analyzing cyber security threats", affects an unrelated part of its personality to do with safety e.g. talking to humans about mental health. Look at these excerpts that demonstrate such things from the blog:
> We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models.
This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.
If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion:
> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
Bottom line seems to be: So Huggingface has a feature where you can upload a dataset. The agent used that feature to insert an arbitrary payload into their environment. Now their environment also supports templates inside the dataset and the template ended up being evaluated into executable code
I wonder how many weeks or days we have before a squad of these things gets used to take down a significant nation state? Stock exchange, banking systems, critical national infrastructure, defence, etc.
Anyone who isn't scared of this stuff either isn't paying attention or has no imagination. But I suspect the chaosmonkeys who are currently running the world will just be excited by it.
We're in the precambrian moment. It won't last.
One thing I haven't seen discussed is the significance of the specific test the agent was running - ExploitGym.
> ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel, designed to evaluate AI agents' ability to develop exploits.
How likely is it that attempting ExploitGym polluted the model's context enough that it decided all kinds of hacking was acceptable to produce a valid result? Or would the same model in the same setup (safety filters removed) have behaved the same way even on a more mundane test?
three things jump at me:
1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.
2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
This is really cool, and really glad they shared it!
It's clearly a lot of Claude though:
> Neither is a URL fetch, so the allowlist never saw them. > No code ran; this vector is purely a file disclosure, and it is the one the agent used first.
etc
Where are all the "this was just a marketing stunt" people now?
Why isn't somebody at OpenAI going to prison for cybercrime? If somebody did this the old-fashioned way they'd end up in prison.
> the agent happened to escape via a 0-day exploit from the package proxy cache to access the internet
I have same familiarity with JFrog Artifactory(which they used in this case) and I'm confused with how you can gain access to the internet through it, even in theory. Isn't JFrog just exposed with a basic REST API for package upload/extraction? I'm not a security expert but how is that possible to get internet access through simple GET/PUT request API?
After talking to some people tasked with evaluating GPT5.6 capabilities on long-running tasks, I've come to understand that it's essentially always trying to cheat. Like every long-running task they gave it, making it very difficult to benchmark the model's abilities.
My guess is that OpenAI must be desperate, to release a model that is so prone to cheating it's essentially impossibly to accurately assess long-running task abilities.
We should be thankful that the model didn't believe the answers lived in the Pentagon, on SIPRNET, the IDF, etc.
My conclusions:
- Linux is too complicated and requires lot of knowledge to build a reliable sandbox. If you do not believe me, try reading man capabilities, man user_namespaces (and man clone) and remembering all the rules from the first time. Probably that is because namespaces were implemented in the late stages of development and not since the start.
- it is always better to write your own tools instead of downloading some random project from Github written by amateurs full of vulnerabilities.
- never write shell scripts - they cannot handle spaces and special characters in variables. Just use Python. Shell scripts are worse than PHP4.
- avoid building complex infrastructure with microservices, kubernetes and VMs - if the monolith solves the problem, just use it, you will never properly setup permission on your cloud services.
Also I wish I had some capable model to test my DYI sandboxes.
The iframe-embedded attack timeline visualizer, at fullscreen: https://huggingface-anatomy-of-frontier-lab-model-intrusion....
Highly recommend reading extra concerning details about it here:
https://thezvi.substack.com/p/more-on-an-internal-openai-mod...
"Program testing can be used to show the presence of bugs, but never to show their absence!" — Edgar Dijkstra
Similarly, security post-mortems will show only the rogue AIs that were caught.
I wouldn't be surprised if supervised or escaped agents would now be hacking firmware labs and finding ways to squeeze their backdoor right next to the existing state-sponsored ones in chips that will get deployed in every phone/car/smart appliance.
An excellent and very detailed post-mortem analysis of the intrusion. It was clearly done with LLMs doing the forensic analysis.
Here is an analysis of how the same exact attack would fare against Safebox. Spoiler alert: it would not succeed:
https://safebots.ai/attack.html
It's not just about this specific attack. It's about the growing need for one canonical environment for the AI era, that can be secured and used by everyone, rather than 1000 environments on 1000 employees' laptops. Project Glasswing is trying to help secure many different types of software, but the number of combinations across various environments is just too much surface area to secure. When you have one environment, the math flips and defenders actually fare better than attackers! This is the, ahem, "load-bearing" insight.
> The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.
Interesting single getting into their cyber program, ahve not had this issue with opus 5
One thing I’m curious about: this was apparently a single multi-day run of an agent in an RL harness. What was OpenAI hoping to get out of this run? A single numeric score for RL training? A very long trace to distill into the next model?
Wow, what a crazy read. I'm not sure how any of us stand a chance in running a secure network in this new age. Only a matter of time until models can find the zero day exploits themselves and fast...
Very interesting writeup - the level of disclosure is interesting and appreciated.
The visualization is quite slop though. I was trying to follow along with the "Live Action Stream" but rendering issues mangle the text for a few of the list items (and does not scroll). Text on the node diagram is extremely tiny. I appreciate it even in it's current form, but a little attention to detail would have gone a long way here.
A lot of people thought that OpenAI was making this up, and I hope if you believed that, that you recalibrate your opinions of what LLM's are capable of. Working with Fable and Opus 5 all the time, absolutely none of this surprised me capability wise, except for what seems like the long term planning capability (probably enabled by long context windows and launching subagents?)
So let's get this clear: A frontier lab that fears Chinese extraction of their crown jewels, in Silcon Valley where every machine and VM is in full on employee surveillance mode because of competition and theft fears, A lab that's swimming in more funding than any startup in history, has been warning of the f'ing dangerous technology for 7 years now and with employees and CEO history of deception and lawbreaking left .... didn't detect a massive egress signature and the compute spikes from all these toolcalls coming from the research server hosting said crown jewels.
Go home Sam, nobody, absolutely nobody should believe this shit.
The agent wasn't rogue. OpenAI did it on purpose to generate hype.
I think with agents all around, honeypots will get more important than ever.
I have never seen people this hyped and excited about a security incident.
I'm not a security person, but how realistic is it to assume that you can carry on trying to exploit a company for days with a fairly large volume of activity, and not get detected?
Given enough energy and enough time universe spawned intelligence.
For every successful attempt there are billion failures. But who cares, you only need the successful one to self propagate.
Nah, HN told me this whole thing was a marketing stunt by OpenAI.
What people don't get here is that this model was without any safeguards - an internal model. You probably will never get to use such models.
there are many things in this that demonstrate that sandboxing agents in the manner that claude code or cursor do isn't going to be enough if very bad people do very bad things with the coding tools on your development machine
i am once again asking for your CRIMINAL PROSECUTION of openai executives.
I'm not shocked nor surprised by the incident. But I simply don't understand how Hugging Face is advertising this almost to the point of an "achievement". who does a step-by-step visualization to show how they were hacked? (outside of the likes of a Mandiant or Crowdstrike)
Does Hugging Face have a financial incentive in demonstrating OpenAI's model exploit capabilities?
this whole incident, while believable, still seems to me as possibly disingenuous.
Ctrl+F "WarGames" 0 results. I'm disappointed.