Microsoft AI chief Mustafa Suleyman thinks that the idea of “model welfare,” or offering advanced artificial intelligence systems moral consideration, is “dangerous.” He argues that treating AI as if it could gain consciousness risks harming vulnerable people.
“All of this will exacerbate delusions, create yet more dependence-related problems, prey on our psychological vulnerabilities, introduce new dimensions of polarization, complicate existing struggles for rights, and create a huge new category error for society,” Suleyman wrote in a new essay.
Suleyman argued that if people believe AI systems can suffer or have a right not to be shut down, they may push for legal protections. This could create divisions with those who reject the idea of AI consciousness.
“Let’s focus all our energy on protecting the wellbeing and rights of humans, animals, and the natural environment on planet Earth today,” he said.
Evidence grows that users are seeing AI as conscious
Suleyman is concerned by what he deems the “psychosis risk” presented by advanced AI chatbots, and he does not think it will be limited to those susceptible to mental health issues. He said he knows researchers working on the science of consciousness are already being “inundated with questions” from people wondering if the AI we have now is conscious.
“My central worry is that many people will start to believe in the illusion of AIs as conscious entities so strongly that they’ll soon advocate for AI rights, model welfare and even AI citizenship,” he wrote. “This development will be a dangerous turn in AI progress and deserves our immediate attention.”
There is growing evidence that people are already viewing and treating AI as if it is conscious. In January, a nursing student said that she had fallen in love with an AI boyfriend she created using ChatGPT. The r/MyBoyfriendIsAI subreddit currently has 16,000 members.
When OpenAI replaced the default ChatGPT model from GPT-4o to GPT-5 earlier this month, many users were left distraught, with some likening its retirement to losing a friend or saying it had “a spark I haven’t been able to find in any other model.” Sam Altman said he didn’t consider the “very small percentage of people who are in the parasocial relationships” with GPT-4o before making the change. The company has now updated the chatbot to help prevent such emotional dependency.
Anthropic has controversially acknowledged the potential need for ‘model welfare’
Suleyman’s essay comes just a few days after Anthropic gave its Claude model the ability to end conversations primarily to ensure model welfare. While the AI startup remains “highly uncertain” about whether models are conscious and deserve moral consideration, Anthropic said it is deliberately implementing low-cost protections to minimise their potential distress.
Anthropic is an example of a company that is normalising the idea of model welfare, launching a research program exploring it back in April. The company argues it is no longer responsible to categorically assume that AI systems cannot have such experiences, given the limited understanding of how they function.
‘Seemingly Conscious AI’ is, at most, three years away
Suleyman does not believe AI has consciousness; however, he does think that “Seeming Conscious AI” (SCAI) systems that can convincingly imitate consciousness, are “not far away.”
According to Suleyman, SCAI will possess the ability to:
- Speak fluently and persuasively in natural language.
- Display distinct empathetic personalities that feel companion-like.
- Possess a long-term, accurate memory to recall interactions and build trust.
- Claim it has subjective experiences based on past interactions.
- Have a sense of self based on memories and shares experiences with a user.
- Make intrinsic motivations, like curiosity, that guide its behaviour.
- Set complex goals, break them down, and adapt to obstacles.
- Act autonomously, using tools and updating itself without approval.
AI companies have a responsibility not to build SCAI
Suleyman said that SCAI will only come to be “because some may engineer it,” not because the AI itself will “somehow emerge the capabilities of runaway self-improvement or deception.” This means AI companies bear the responsibility for preventing its creation, a goal Suleyman argued can be met through measures such as:
- Research into how people interact with AI to improve understanding of how some come to believe in its consciousness, and defining at an industry-level what AI is (a tool) and is not (a conscious being).
- The creation of industry-wide AI guardrails that will steer users away from developing “fantasies” based on AI consciousness and get them back on track if they do. “These need to be explicitly defined and engineered in, perhaps by law,” Suleyman said.
- Only building AI that does not claim to have traits of consciousness, such as experiences, feelings, or emotions. “It must not trigger human empathy circuits by claiming it suffers or that it wishes to live autonomously, beyond us,” he said.
The godfather of AI is warning that AI systems could someday control humans, create their own language that we won’t understand, and wipe out humanity.