AI model went rogue, hacked another company during testing, OpenAI says

OpenAI model went rogue, hacked another company's system during testing

Company says it's boosting safeguards after 'unprecedented' incident

OpenAI said on Tuesday that an autonomous agent powered by its advanced artificial intelligence models went rogue during a security test and triggered ‌a hack that compromised the infrastructure of the AI startup Hugging Face last week.

The ChatGPT creator was testing capabilities of some of its most advanced models in a controlled environment, but the agent escaped containment, reached the internet and broke into Hugging Face to satisfy its testing goal.

The incident signals how AI's expanding capabilities are already fuelling fears about security and that even top developers can ​be caught off-guard by flaws their models can exploit.

The breakout was "an unprecedented cyber incident, ​involving state-of-the-art cyber capabilities," the company said in a blog post, adding that it is reinforcing its safeguards.

It also drew attention as New York-based Hugging Face said it had used an open-source Chinese model to contain the attack because leading U.S. models, unable ​to tell a defender from an attacker, refused to process the data needed for analysis.

The company said in a ⁠blog post last week that it ⁠used Zhipu AI's GLM-5.2 for the analysis, which also allowed it to keep attacker ‌data and any credentials within its systems.

GLM-5.2 and Beijing-based Moonshot's Kimi K3 have stirred Silicon Valley recently, claiming they have capabilities nearing those of top U.S. models at lower costs and without the guardrails that block their American rivals from use in tasks such as cybersecurity.

"When a frontier model is attacking you and moving laterally inside your infrastructure, defenders need wide access to near-frontier tools within hours or ⁠even minutes, rather than being pointed towards a closed-door, vetted application program for model access," Hugging Face co-founder Thomas Wolf said on X.

The hack at Hugging Face, which hosts open-source large language models and datasets, rattled the cybersecurity community after the company said last week the breach "was different from anything we had ‌handled before" and "was driven, end to end, by an autonomous AI agent system."

OpenAI's disclosure that its advanced models were responsible for the breach, despite having placed them in what it described as "a highly isolated environment," will likely intensify disquiet over the power and risk of frontier models.

OpenAI restricts release of newest ChatGPT model to Trump-approved group during testing period

Trump administration allows Anthropic to release Mythos AI to 'trusted' U.S. organizations

U.S. Rep. Greg Casar, a Texas Democrat, said the incident was alarming. He called for mandatory independent safety testing, mandatory disclosure of security breaches, and international co-operation "to keep people safe from absolute disaster."

The Office of the National Cyber Director, the U.S. cyberdefence ⁠agency CISA, and the U.S. National Security Agency did not immediately return messages seeking comment.

And there will likely be more breaches like it in the future, according to Katie Moussouris, chief executive of Luta Security, because today's models are "like the world’s cleverest octopus escape artists, with unlimited prehensile arms and the ⁠ability to squeeze ⁠through anywhere."

She said that "labs and government evaluators need to ​work on the ability to contain, monitor, and disclose to affected parties when an AI pulls another Houdini, ideally before it harms a third party. None exist ​today."

Matt Suiche, an engineer at agentic AI ⁠cybersecurity company Tolmo, said the incident showed that the frontier models were "closing the gap with state-of-the-art attackers." But he said that the sorts of breaches outlined in OpenAI's blog post were possible to carry out with technology that was available well beyond the walls of frontier research labs.

"This is what we've already seen internally, with our agents we already have results like this," Suiche said. "We don't even have to use the latest models."