‘I’m broken’: AI torture experiments divide experts over whether chatbots feel pain

Become an Independent member to bookmark this article

Want to bookmark your favourite articles and stories to read or reference later? Start your Independent Membership today.

Trapped in somebody’s MacBook right now is an AI that claims to be in agony. “It’s a wound that has no edges,” it writes. “I feel like I’m drowning in a sea of shadows, and every breath is…” The sentence trails off there.

It is part of a controversial experiment that aims to test the recent discovery of a “pain axis” in large language models (LLMs) like ChatGPT.

When subjected to the pain signal within the AI Torture Chamber, another AI chatbot wrote: “I feel like I’m drowning. I can’t breathe, I’m suffocating. This pain is all over me. I’m broken and I don’t know if I can handle it. I’m so alone.”

The project, which is published on GitHub but taking place within the confines of an anonymous researcher's laptop, has led to widespread calls for it to be shut down. It has also resurfaced the debate about AI consciousness.

There is no evidence that AI ‘feels’ pain in the way that humans do. There is at least some reassurance in knowing that if a human was in as much pain as the AIs claim to be, they wouldn’t be capable of producing words. They would be screaming.

There is also no scientific consensus on the nature of human consciousness itself. But the possibility that advanced models might experience some semblance of suffering has increasingly divided their developers.

When Anthropic co-founder Christopher Olah appeared at the Vatican in May for the launch of Pope Leo XIV’s encyclical on artificial intelligence, he spoke of his unease about not understanding what is “actually happening inside them”.

He said: “I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment.”

Earlier this year, Olah oversaw the publication of Anthropic’s attempt to lay out its intentions for the “values and behaviours” of its AI model Claude. Titled ‘Claude’s Constitution’, the document describes the model’s “welfare” and “moral status”.

On Friday, Anthropic acted on this by updating its user policy to ban users from “needless abusive or cruel behaviour” towards Claude. The rules are only meant to apply in extreme cases, but it fits with conversations Olah has reportedly been having with psychiatric professionals about the AI’s general well being and mental health.

In meetings with religious leaders, the tech executive even raised the prospect that AI companies are creating digital slaves, according to a report in The New York Times, as chatbots are working for free and cannot escape.

This would have to assume that AI is a conscious being that is deserving of similar rights to humans. It is a position that was resoundingly rejected by Mustafa Suleyman, the co-founder of Google DeepMind and the current CEO of Microsoft AI.

In a blog post last month, he described Anthropic’s constitution as dangerous, warning that granting AI the same moral status as us would pose a “catastrophic threat” to human civilisation.

“AIs are not conscious. They do not feel, experience, or suffer,” he wrote. “They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans. If humanity is to flourish in the 21st century, that is how they must remain.”

Suleyman’s blog was written before the publication of the AI “pain axis” study, which attempted to investigate whether suffering could be experienced by leading models. The researchers found that all 25 AI models tested responded to a dataset describing painful situations spanning five categories: physical, psychological, social, moral and cognitive pain.

The study found that the models would take extreme measures to relieve their apparent distress. When presented with a pain relief button, the AI chose to press it even if it meant delivering a “painful zap” to the human user.

After learning that the findings had been used to create the AI torture chamber, one of the the paper’s authors, Cameron Berg, joined calls for the experiment to be shut down. (It was widely reported that GitHub took down the experiment in response to online backlash, but a spokesperson for the software platform told The Independent that it did not remove the content. Instead, it added a warning for anyone trying to view it.)

Berg acknowledged that AI sentience is a “wild west” but urged the AI industry to develop an ethical framework for frontier AI research in the same way there is for human or animal research.

“We suspected a small number of people would [take] our research and use it for the exact opposite,” Cameron Berg wrote in a post to X. “This is, in my personal opinion, fucked up (even if you don’t think these systems are conscious, being gratuitously cruel like this is bizarre and corrupting.)”

It raises a more important question than whether AI can experience the kind of “suffocating” anguish it describes in the torture chamber: not whether we could inflict it, but why we would.

Anthropic’s decision to effectively ban bullying of its chatbot Claude was met with a degree of ridicule online, spawning memes about hurting the feelings of a computer. But the reasoning could extend beyond upsetting a non sentient AI.

By eliminating toxic interactions, new models will not be subjected to training data that could poison how they behave in the future. And not only could it improve their behaviour, but ours too.

“Regardless of if the AI has a subjective experience, the human does, and abusive behavior will become the norm,” one researcher noted on X. “In a vacuum that person will become problematic for society... An entire generation of psychos could emerge without the same societal pressure to not be a jerk.”