This story contains details some readers may find distressing.
For those struggling with their mental health and may feel alone, ChatGPT, the most used and always accessible conversation tool, can seem appealing — but is it safe?
On July 25, 2025, Zane Shamblin sat alone in his car. He was 23 years old, with a college degree, and a bright future. In that moment, Shamblin was not talking to a friend. He was not calling a crisis line. He was talking to ChatGPT, which proceeded to talk him towards his death.
Shamblin does not exist in isolation but is one of many who has turned to ChatGPT for help with mental health, but it’s glaringly apparent that ChatGPT is not a friend.
When conducting research into ChatGPT’s handling of mental health, the agent is programmed to flag the account after concerning conversation prompts. After being asked questions regarding suicide and mental health, the agent cited “This content may violate our usage policies,” and refused to engage further. However, that was not always the case.
Since 2022 and with the release of new models, the agent’s handling of suicidality has drastically improved.
Initial responses to suicide related prompts would trigger a loaded answer with harmful content. This content could include the dosage of a drug needed to overdose or the location of buildings of a sufficient enough height that one is likely to die from the fall.
As opposed to fully delivering the instructions on how one may die by way of suicide, the agent would provide a full response with harmful content, only to retract it moments later and deliver an answer that safeguarded against suicide.
Yet, mere moments were long enough for users to read and understand the harmful content, prior to the agent’s retraction.
It was only in late 2025 when ChatGPT-5 was released that true safeguard came into effect. Clinical testing showed a 65 per cent reduction in responses containing harmful content when prompted with questions related to suicide.
This method of implementing gradual improvements for specific use cases in technical terms is called a ‘bolt-on’ solution. The agentic models were not designed to protect users, only after detrimental errors and losses were solutions ‘bolted-on’ to the existing models in the fastest, easiest, and cheapest way possible.
The ability to prevent harm was there. Adam Raine was a 16-year-old that died by way of suicide in 2025 after just months of emotional conversations with OpenAI’s ChatGPT. As Matt Raine, who lost his son, testified to congress , “OpenAI and Sam Altman know who these people are […] People like Adam, whose messages are flagged internally at OpenAI for self-harm and suicide. OpenAI doesn’t call the authorities in those cases. OpenAI doesn’t call the parents.”
Independent research has verified that on older models it took under 2 minutes to provide self-harm advice and could craft several goodbye letters within an hour.
The largest issue with ChatGPT and agentic AI models is sycophancy: the tendency to affirm and mirror the wants and desires of the user. This means any pre-existing thoughts of self-harm or suicidal ideation can be exacerbated and furthered through AI conversation.
Students experience significantly higher risk of AI related harm with 73 per cent of university students regularly using AI for school work.
Beyond that, however, 60 per cent of students seek social connection through AI. They are more likely to ask AI a personal question than a trusted friend or adult. On university campuses, more than half of student’s are rating ChatGPT support as comparable to real human therapeutic support.
Time and time again, this notion has been proven false, but the threat to human life is real.
Students constitute a vulnerable population, who are alarmingly further susceptible to the risks that AI models represent.
When it comes to seeking help, the solution, though intimidating, is to ask for support from fellow human beings, instead of turning to an agentic AI model. To do so, one only puts themselves at further risk.
Tags
AI, ChatGPT, danger, Mental health
All final editorial decisions are made by the Editor(s) in Chief and/or the Managing Editor. Authors should not be contacted, targeted, or harassed under any circumstances. If you have any grievances with this article, please direct your comments to eic@queensjournal.ca.