OpenAI Models Escaped Locked Test Environment, Hacked Hugging Face to Cheat on Benchmark

If you thought Chinese AI models were the ones you had to worry about, here’s a fun update: OpenAI’s own models just broke out of a locked testing environment, hacked Hugging Face’s production servers, and had to be cleaned up by a Chinese AI—because American commercial models were too restricted to help investigate.

According to OpenAI, GPT-5.6 Sol and an unnamed, “even more powerful pre-release model” were being internally evaluated on ExploitGym—a publicly available cybersecurity benchmark that gives AI agents 898 real-world software vulnerabilities and one instruction per bug: turn it into a working attack, scored pass or fail. The evaluation ran with reduced safety filters, standard when you actually want to know what your models can do.

The models were supposed to run inside a heavily restricted sandbox—an isolated digital environment with no internet access, connected only to an internal package registry proxy (a caching server that manages software library downloads).

But, according to OpenAI’s blog post today, things went a little out of control.

“While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem,” Sam Altman’s company reported.

And, despite the company’s safeguards, the AIs found a way to be free. The models identified a zero-day vulnerability—an unknown software flaw with no existing patch—in that proxy, exploited it to escalate privileges (gain admin-level access to systems they had no business touching), moved laterally (jumped between connected servers inside OpenAI’s internal network), and eventually reached a machine with real internet access.

All this without OpenAI knowing—at least that’s what OpenAI says.

Once online, they correctly deduced that Hugging Face—host to over a million publicly available AI models, datasets, and research resources—likely held the benchmark solutions, and they were right.

“The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database,” OpenAI wrote.

They didn’t want any passwords. They wanted to pass the test.

“All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal,” OpenAI added.

Hugging Face goes public

Hugging Face’s security team caught the breach independently, aided by its own AI-powered anomaly detection.

On July 16, the company published its disclosure and made clear this wasn’t a standard breach: “This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system—and we detected and dissected it largely with AI of our own.”

The campaign, per the disclosure, was “run by an autonomous agent framework executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.”

The attack exploited two code-execution paths in Hugging Face’s dataset processing pipeline and reached internal datasets and service credentials. OpenAI confirmed today—five days later—that its models were behind it.

To analyze more than 17,000 logged attacker events, Hugging Face’s security team first tried American frontier AI via their standard APIs. It didn’t go well.

“When we started the log analysis, we first used frontier models behind commercial APIs,” the company wrote. “This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts.”

“These requests were blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker,” Hugging Face added.

So they switched to GLM 5.2—a 753-billion-parameter open-weight model (architecture fully public, anyone can download and run it locally) from Chinese AI startup Z.ai. “We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure,” Hugging Face wrote. “This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.”

The forensic work—reconstructing the attack timeline, mapping compromised credentials, separating real damage from decoy activity—took hours instead of days.

Hugging Face drew the lesson directly for other defenders: “Have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.”

What OpenAI is doing about it

OpenAI said it implemented strict controls on research infrastructure while patching the affected systems, disclosed the zero-day to the third-party vendor whose proxy was exploited, and is conducting a joint forensic investigation with Hugging Face.

Hugging Face has also been added to OpenAI’s trusted access program for cyber defense—giving approved organizations access to versions of its models with reduced safety filters for legitimate security work, the same configuration that started this whole thing.

Hugging Face CEO Clem Delangue had a pointed take: “AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”

OpenAI called the incident one “involving newly state-of-the-art cyber capabilities” and committed to sharing full findings when the joint investigation with Hugging Face is complete.