This article discusses suicide and suicidal ideation, which may be distressing for some readers. If you or someone you know is in immediate crisis or has suicide-related concerns, please call 1-833-456-4566 or dial 9-8-8 for immediate support.
Artificial Intelligence (AI) chatbots are unregulated, unempathetic, and unsafe.
In the fall of 2024, Adam Raine asked ChatGPT why he felt “perpetual boredom anxiety and loss.” That spring, after months of similar conversations with the chatbot, Raine took his own life. Raine’s family is now suing OpenAI for wrongful death, a case that points to a lack of government regulation regarding AI, funding for mental health resources, and a shortage of empathy from large tech corporations.
In a rush to release GPT-40 OpenAI overlooked training the model to respond to users experiencing mental health crises. Beyond softening its tone with empathetic language, the chatbot offers no resources and doesn’t warn authorities if a user is in severe mental distress—a dangerous omission as Gen Z faces a mounting mental health epidemic.
The “profit before people” culture in the tech industry has pushed users’ health to the background, prioritizing release dates over health and safety. In the lead-up to the GPT-04 launch, several prominent executives resigned, stating safety had taken a backseat to “shiny products.”
AI may serve as a powerful tool, it must be deployed with empathy and caution to mitigate harmful consequences. Yet the tech industry’s hustle culture—where profit consistently eclipses human well-being—leaves little space for such safeguards. Twisted to its extreme, this culture creates a dystopia where a robot can help a teenager take their own life.
Before the release, OpenAI should’ve considered that users with mental health concerns would have a tendency to self-isolate, turning to a chatbot for help. With a lack of funding for mental health resources and a predisposition to self-isolate, a chatbot might be the first point of contact for a person in mental distress. In Canada, therapy can cost between 90 and 300 dollars per hour, while over five million Canadians suffer from mental health-related issues.
Unregulated, a simple request for homework help can spiral into questions of life, death, and meaning—subjects far beyond ChatGPT’s capacity to currently handle responsibly. Once users turn to it for
guidance, the tool often reinforces their assumptions, echoing back exactly what they want to hear. Unlike social media echo chambers, which reflect the perspectives of real people, AI chatbots amplify biases without the grounding of human experience, making the effect even more insidious.
Beyond regulation, the use of ChatGPT and AI chatbots for mental health questions underscores the need for increased mental health resources.
It’s frightening to live in a world where people can’t get the help they need and turn to ill-equipped computers to deal with the real human emotions they’re experiencing. Yet a solution to the problem isn’t straightforward: even if Raine was redirected to a suicide hotline, he would risk wait times and only receive one-time help instead of the continued care he needed.
What started as a tool has, for some, moved into a parasocial relationship, with some users feeling emotionally attached to the technology, which can encourage suicidal ideation. AI tools can be helpful, but the creators and regulators of this new technology haven’t given enough care to how it’ll used by every party.
Raine’s suit should come as an indication of the need for increased digital literacy surounding as they become increasingly implemented in educational and workplace settings. This case is a tragedy and a reminder that the culture in the technology industry can breed real human consequences. While AI offers new opportunities, companies, policy makers, and users need to be aware of its repercussions.
Ensuring AI serves humans and doesn’t endanger them requires an important combination of stronger regulation, accessible mental health care, and a better commitment to design technology with empathy and care.
Tags
All final editorial decisions are made by the Editor(s) in Chief and/or the Managing Editor. Authors should not be contacted, targeted, or harassed under any circumstances. If you have any grievances with this article, please direct your comments to journal_editors@ams.queensu.ca.