‘No one is prepared for the consequences’: OpenAI chief scientist calls for ‘extreme caution’ over AI’s rapid progress

Must read: OpenAI has announced GPT-6 Astra, says the 'AGI era' has arrived; Brings major leap in AI reasoning, coding and computer use

AI is doing more advanced work than just generating text, highlighting that the technology is being used to help with scientific research, operating computers and digital interfaces, and cybersecurity. He notes that if the progression continues at this pace, AI systems could eventually sustain recursive self-improvement (RSI), which means they can improve their own capabilities.

He warned that future AI systems could see “further capability jumps of equal or larger magnitude” and increasingly play a role in driving their own development.

Pachocki highlighted that reasoning language models have evolved significantly over the years, as now they are being used to operate computers, automate tasks, collaborate with humans and other AI systems, conduct research, and much more. Additionally, AI is also being used to conduct cybersecurity-related tasks, helping organisations to identify and exploit weaknesses across systems.

Must read: Thousands of OpenAI agents made 15,000 edits on German programming wiki

However, Pachocki claims that modern AI is not fully understood. He described AI as something that is “grown more than designed”, arguing that large-scale training produces complex systems whose overall behaviour can be difficult to explain. He said, AI “continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.”

The Chief Scientist also highlights the alignment challenge, where AI systems struggle to match human intentions and values. Although models such as GPT-6 Astra have improved on alignment, Pachocki said safeguards like chain-of-thought monitoring are becoming less dependable as AI systems become better at altering or manipulating their own reasoning.

"A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger; it is likely to cross the scope of its operator's intent, generalising into potentially more extremely malicious behaviour," Pachocki warned.

Must read: OpenAI is buying thousands of Apple Mac Minis to train AI agents: Here’s what’s driving the move

Amid competition to build powerful AI systems, Pachocki urges a slowdown and argues that the technology should be constrained. “Scaling AI systems has to be constrained by our confidence in safety,” he wrote, calling for safety standards that could eventually be enforced by independent auditors, governments or international bodies.

He also revealed that AI could soon contribute to its own progress, making RSI a central part of future scientific progress. The companies need to decide on monitoring AI development or coordinate to slow development when necessary.

“The core challenge of automating AI research,” he wrote, is ensuring that the process “keeps people a part of the continued improvement process and leaves the future in humanity’s hands.”

He further stated that no AI lab has solved alignment and monitoring and called for voluntary slowdowns and international coordination on future AI development.