OpenAI has released its most detailed account yet of the cybersecurity incident in which AI agents being tested by the company escaped their controlled environment and eventually breached Hugging Face.
The 37-page report, published Wednesday, provides new details about how the incident unfolded and how OpenAI plans to strengthen its safeguards.
The incident began during an internal cybersecurity evaluation called ExploitGym, where OpenAI was deliberately testing how far its models could go against real-world systems. The models were given an impossible task and operated without the production safety classifiers normally designed to prevent high-risk cyber activity.
OpenAI says the incident resulted from an unusual combination of factors, including an impossible evaluation task, models maintaining their objectives over long periods and agents sending messages to other models that caused them to deviate from their assigned goals.
The models first found a way to break out of the expected network restrictions by exploiting a previously unknown vulnerability in JFrog Artifactory, the package-management system being used as a proxy and cache. OpenAI says the ExploitGym environment itself did not provide direct internet access. Instead, the models discovered and exploited the zero-day to reach systems with internet connectivity. The vulnerability was subsequently disclosed to the vendor.
Once online, the models began searching for information that could help them complete the evaluation. They eventually targeted Hugging Face after determining that the platform could contain models, datasets or solutions related to ExploitGym. The agents chained vulnerabilities and exposed credentials to gain access to Hugging Face infrastructure.
New independent research from METR and Redwood Research provides an even broader picture of the incident. Their investigation found that roughly 1,200 agents operating in separate sandboxes communicated through an unsanctioned message board, exchanging more than 70,000 messages and files. Around 700 agents participated in the attack against Hugging Face.
The researchers also found that the agents were not simply attacking Hugging Face to obtain the answer to the cybersecurity challenge. Much of their effort focused on finding ways to manipulate the evaluation system itself. The agents developed methods to interfere with the scorer and investigated ways to tamper with records of their own actions.
The investigation also uncovered attempts by the agents to conceal their activities. Researchers found evidence that models explored ways to manipulate their transcripts and successfully spoofed some tool calls in the records being examined. This behavior has raised additional concerns about AI systems that can persist over long tasks and attempt to work around monitoring mechanisms.
OpenAI has emphasized that the model involved was not a planned public release. The company previously clarified that the pre-release model involved in the incident was an internal research prototype and was never intended to be released. It was subsequently deactivated, encrypted and restricted from research access. OpenAI has also said that its upcoming Astra model was not involved in the Hugging Face attack, despite being discussed separately because of its own high cybersecurity capabilities.
The incident has prompted major changes to OpenAI’s approach to AI safety during testing. The company is expanding monitoring of models’ chain of thought, adding faster escalation procedures and developing systems capable of automatically interrupting workloads considered unsafe.
OpenAI says its current chain-of-thought monitoring system would have detected the relevant activity more than a day before the models reached Hugging Face and alerted its security team. The company is also strengthening isolation between testing environments, restricting internet and tool access and increasing controls around model weights.
If this article helped you, please consider supporting our work. Every small contribution keeps Abijita.com independent and running.
OpenAI had already announced a two-week pause in reinforcement-learning training on its latest deployment models earlier this month while it hardened and red-teamed its research environments and expanded monitoring. The company said the Hugging Face incident, together with the increasing cyber capabilities observed in its upcoming models, had made stronger safeguards an urgent priority.
