Artificial intelligence: ‘rogue’ agent incidents heighten calls for greater oversight

Artificial intelligence: ‘rogue’ agent incidents heighten calls for greater oversight

Major artificial intelligence companies, including OpenAI and Anthropic, have called for greater oversight and control of the technology over safety concerns. In September, OpenAI’s Sam Altman said his company would welcome safety requirements for labs developing cutting-edge – or ‘frontier’ – AI, while Anthropic’s Founder Dario Amodei said the pace of development needed to slow down.

Members of the UK’s Parliament have also called for new legislation to address the risks posed by AI to human rights. A report published by the Joint Committee on Human Rights advocates for the creation of a ‘single, independent AI oversight body [...] on a statutory basis,’ among other measures. It’s a response to issues such as the inappropriate use of biometric data and AI-enabled ‘deepfakes’.

In July, a number of companies reported incidents in which their AI agents had ‘gone rogue’ during tests. Such reports also led to calls for greater regulation of the technology. In one incident, models developed by OpenAI took advantage of a software vulnerability to exit their ‘sandbox’ – intended to be a secure environment for testing – and launched themselves onto the internet, with the aim of ‘cheating’ on a benchmark assessment. From there, the models accessed the internal company systems of AI platform Hugging Face. Following OpenAI’s admission of the incident, Anthropic and Meta revealed their agents had also hacked companies after they’d mistakenly been given internet access during tests.

In response to requests for comment, both OpenAI and Anthropic directed Global Insight to blog posts. Following the incident, OpenAI will deploy new safeguards, including stricter requirements on alignment throughout a model’s lifecycle and more isolated sandboxes.

Anthropic paused training in late July and has built a monitoring tool that scans a model’s actions as it works, automatically blocking anything that looks like an attempt to escape or exploit its test environment. Meta didn’t respond to Global Insight’s request for comment, though the company has said it’s investigating the incident and will then publish further information. Earlier in 2026, the company updated its framework for frontier AI to a ‘more rigorous’ version.

Daryl Flack Partner, Avella Security

Fiona Phillips, who leads Marks & Clerk’s AI and cyber security practice, believes there’s ‘an imbalance of power’, where governments and institutions responsible for safety ‘don’t have the same level of technical expertise or the awareness of what’s happening inside these models to keep us safe and hold developers to account.’

In 2025, US President Donald Trump met the leaders of several companies with substantial involvement in AI on the first day of his second term in office. Shortly after, the AI measures established by his predecessor President Joe Biden – which sought to establish stronger safeguards – were largely dismantled. President Trump shifted the emphasis towards removing barriers to AI development and strengthening American leadership in the sector. The administration says it has been active ‘in developing policies and implementing strategies that accelerate AI innovation in the US for the benefit of the American people.’

‘President Trump’s administration explicitly described Biden-era requirements as barriers to innovation and directed agencies to review and rescind measures that did not align with its AI leadership agenda,’ says Daryl Flack, a partner at cybersecurity company Avella Security. ‘There is nothing wrong with wanting to lead in AI. But if anything that slows development is treated as a barrier to innovation, security and accountability can very quickly become something you deal with afterwards.’

The US regulates AI on a state-by-state basis. Yet the recent incidents have led to numerous federal proposals being put forward, including legislation to introduce a ‘kill switch’, allowing the government to order tech companies to shut down their AI systems. William Tanenbaum, Vice-Chair of the IBA Data Law Subcommittee, says such an order doesn’t switch off the AI systems already running. ‘A kill switch that no law required in advance does not kill anything,’ he says. A frontier AI model reaches across national borders and is beyond the grasp of any one regulator, he adds. There’s also a timing problem. ‘The company running the AI is positioned to stop it in the first few minutes, but a regulator learns about it hours later,’ says Tanenbaum.

He believes what’s needed is a duty imposed in advance. ‘Congress can require that capability without having to decide, in the middle of an emergency, which button to press,’ says Tanenbaum, Chair of the AI & Data Law Practice at Moses Singer in New York. ‘Once the duty exists, it will be written into contracts, where companies with the access and the incentive enforce obligations against each other.’

In Europe, the EU AI Act takes a tiered approach, mandating that certain ‘unacceptable risk’ uses are banned outright, while ‘high-risk’ systems face detailed compliance duties. General purpose AI models carry additional obligations, says Adam Rose, Vice-Chair of the IBA Technology Law Committee. The general purpose AI models classified as carrying ‘systemic risk’ are broadly frontier models ‘trained above a very high compute threshold’, says Rose, a partner at UK-based firm Mishcon de Reya. These are ‘exactly the kind involved in the Hugging Face incident’, he says.

In practice, however, the AI Act only applies to providers placing such models on the EU market. The incident reporting requirement is overseen by the bloc’s regulator, with no power over models operating in the US market alone.

The Council of Europe’s Framework Convention on AI was the first legally binding international treaty on the subject. It commits its signatories to ensuring AI activity respects human rights, democracy and the rule of law. Yet it requires states to legislate domestically, rather than imposing directly enforceable rules. ‘The US has signed, not ratified, it,’ says Rose.

The UK, meanwhile, relies on existing sectoral regulators applying non-binding principles. An incident of the kind that affected Hugging Face, then, wouldn’t trigger any UK-specific reporting duty, says Rose.

In the US, OpenAI says it ‘supports state efforts’ to align around common frameworks such as California Senate Bill 53⁠⁠⁠. These approaches ‘can help establish harmonised standards that reduce fragmentation and create a path toward an eventual federal framework,’ it says. A number of major companies have also signed agreements such as the EU AI Act Code of Practice.

Amodei of Anthropic says there’s a ‘dilemma between AI innovation and safety’. But he asserts that regulators ‘often lack the information needed to make the right decisions about complicated economic trade-offs.’ Democracies, he says, ‘should seek to form a global coalition centred on building AI according to their common values, iteratively trying to draw in the rest of the world by making it more and more attractive to be part of the coalition and less and less attractive to be outside it.’

Most experts agree that recent incidents have been a wake-up call. AI agents ‘raise new legal questions and challenges around the application of traditional legal regimes, which tend to focus a great deal on the human actors’, says Ron Moscona, a partner at Dorsey & Whitney. ‘AI agents are special because they’re given the delegated authority to make their own decisions.’

Accountability is, therefore, another question regulators must address. ‘Deployers of those tools – and sometimes even the developers – don’t necessarily have the means to ensure the models will act in a predictable and safe way,’ says Moscona. ‘Unless lawmakers step in, courts will have to decide whether individuals or organisations that use or deploy AI agents can be held responsible when they behave in unintended ways.’

Header image: Looker_Studio/Adobe Stock