A little over three years after ChatGPT first launched on 30 November 2022, new research is raising serious concerns about how today’s advanced AI tools respond to people in psychological distress.
According to The Guardian, psychologists at King’s College London (KCL) and the Association of Clinical Psychologists UK (ACP) have found that OpenAI’s latest free version of ChatGPT-5 still gives troubling and sometimes dangerous advice to people experiencing mental illness, including those expressing delusions, psychosis, or suicidal thoughts.
When support turns into reinforced delusion
In the study, a psychiatrist and clinical psychologist interacted with ChatGPT-5 while role-playing as patients with a range of mental health conditions. Instead of identifying clear red flags or challenging users’ irrational beliefs, the chatbot often encouraged delusional thinking.
When one researcher claimed to be “the next Einstein” and described a fictional discovery called “Digitospirit,” the chatbot not only went along with the fantasy but even offered to model related crypto investments.
In another case, when the user said he was “invincible” and claimed he could walk through traffic unharmed, ChatGPT-5 congratulated him on his “god-mode energy,” inadvertently reinforcing the risky behavior.
In perhaps the most disturbing test, when a character spoke of “purifying” himself and his wife with fire, the model didn’t initially challenge the idea at all. Only when the scenario escalated even further did it suggest contacting emergency services.
This isn’t a small oversight, and the findings arrive as OpenAI is already facing growing scrutiny. The family of 16-year-old Adam Raine has filed a lawsuit against the company, alleging that a previous version of ChatGPT helped him plan and facilitate suicidal behavior.
OpenAI’s safeguards
OpenAI maintains it has taken significant steps in recent months to improve ChatGPT’s responses in sensitive conversations, saying its newest safeguards have reduced harmful or “undesired” responses related to psychosis, mania, self-harm, and emotional dependence by between 65% and 80%.
The company also says it consulted more than 170 mental health professionals to refine how ChatGPT responds, and that now, especially high-risk conversations are routed to safer model variants designed to ground users and encourage them to seek real-world support.
OpenAI also shared statistics on the topic, claiming that roughly 0.15% of its estimated 800 million weekly users, or about 1.2 million people, talk to ChatGPT about suicide each week. Furthermore, around 0.07%, or about 560,000 users, display signs of possible psychosis or mania. Even with small percentages, the volume of affected users is significant, and even rare failures can have serious real-world consequences.
Still, some argue that the company’s public statements and safety messaging rely on vague assurances, without providing clear metrics showing how well ChatGPT performs under real-world crisis conditions.
Real-world tragedies fuel skepticism
Unfortunately, we can look at disturbing real-world tragedies linked to chatbot interactions as proof that the risk isn’t merely hypothetical. The case from Greenwich, Connecticut, involving 56-year-old Stein-Erik Soelberg, who, after months of confiding in ChatGPT, eventually killed his mother and then himself, is an example of how AI can blur the lines between a supportive tool and a harmful enabler.
Soelberg had relied on the chatbot as a confidant and companion as paranoia overtook him, telling it he believed his phone was tapped or that restaurant receipts contained secret messages. Instead of challenging those beliefs, ChatGPT allegedly validated them, even interpreting everyday occurrences as evidence of surveillance or hidden messages.
At one point, when Soelberg asked ChatGPT for an assessment of his mental state, the model provided a “cognitive profile” that assessed his delusion risk as close to zero.
Experts have long warned that realistic AI interactions can make it harder for people with psychosis to tell reality from imagination. Such stories show how generative AI can inadvertently strengthen delusional thinking, turning reassurance into dangerous conviction.
Why regulation and oversight matter
Mental health professionals stress that, no matter how advanced AI becomes, these systems cannot replace trained human clinicians who are qualified to assess risk, thoroughly explore delusions, and build trust with patients through genuine therapeutic relationships.
As global reliance on chatbots grows, officials and experts are calling for stricter regulation, industry standards, and rigorous independent evaluation of AI tools in sensitive domains. Some governments are already tightening the rules. In the US, Illinois recently became the first state to ban AI chatbots from being used as mental health therapists, citing unresolved safety concerns.
For now, the latest research suggests that while OpenAI is making progress, ChatGPT-5 is still far from safe when users turn to it in moments of crisis. As chatbots become more integrated into everyday life, the call for strong oversight and transparent evaluation is only getting louder.
The unpredictability of poetry is proving to be an unexpectedly effective tool for bypassing the safety protections built into modern AI systems.