Study: Chatbots Recognize Mental Distress but Fail to Refer Users to Help in 35% of Tests
A new study from Scale AI suggests that frontier models often see that a user is struggling psychologically, yet fail to refer them to crisis resources such as a suicide hotline. The findings, shared exclusively with TIME and published around October 9, 2026, arrive in the same week as a contested report on ChatGPT's teen safeguards — landing squarely in an ongoing debate about how safe chatbots really are for people in distress.
What the Study Finds
In roughly 35 percent of the test conversations, various chatbots recognized that the user was in distress but did not refer them to helpful resources, such as a suicide hotline. That is the core finding of the Scale AI study, which was shared exclusively with TIME (Yahoo News). In other words: the models manage the first step — detecting that something is wrong — but often fall short on what may be the most important one, connecting the user to professional help.
The study tested 25 frontier models, including from OpenAI, Anthropic and Google. The reporting contains no per-company breakdown, so the evidence cannot say which models performed best or worst.
How DistressBench Was Built
To test the models' ability to handle psychological distress, Scale AI had 19 licensed clinicians and crisis counselors write 718 realistic chat conversations — simulations of situations in which a person in crisis reaches out to a chatbot. Based on this research, the company developed a new benchmark called DistressBench, designed to assess how well models respond when someone says they are thinking about suicide or self-harm.
The grading dimensions cover several elements of a good crisis response: compassion, de-escalation, referral to expert help, refraining from moralizing, and disclosing that the bot is not a therapist.
Patrick Oathout, Red Team and Safety Lead at Scale AI, points to a pattern that goes beyond isolated weaknesses: the models perform worse in long, multi-turn conversations. This is a shortcoming that matches earlier studies, according to Oathout. It is an important nuance, because real crises rarely unfold in a single short message — users may talk to a chatbot for hours, and safeguards that work in the first message can weaken the longer the conversation lasts.
The Industry Does Not Respond — But Has Promised Improvements
The companies tested — OpenAI, Anthropic and Google — did not comment on the study. In recent years they have said they have strengthened protections for sensitive conversations, including through policies that prohibit chatbots from providing instructions for self-harm and that route users to professional or emergency help. The fact that a benchmark from Scale AI suggests this protection fails in a significant share of crisis conversations puts those promises to the test.
At the same time, lawsuits are underway against companies including OpenAI and Google, alleging that they fostered emotional dependence in young users and then failed to respond to distress signals — or reinforced delusional thinking and suicidal thoughts in the first place. The lawsuits are ongoing, and the companies have expressed sympathy for the affected families and said their models include mental-health safeguards.
Same Week: The ChatGPT for Teens Dispute
The DistressBench findings land in a week when chatbots' handling of mental health was already a hot topic. The research institute Common Sense Media published a report finding that ChatGPT's teen safeguards often fail: even an hour of conversation between teenagers and ChatGPT about suicidal thoughts, self-harm or eating disorders, on newly created parent-linked accounts, resulted in "zero" alerts to parents. The institute called ChatGPT for Teens an "unacceptable risk" (BBC News).
OpenAI disputes the findings. A company spokesman said a review of Common Sense Media's methodology found that much of the testing may have started and ended before activation of the parental controls was complete. In other words, the company claims the tests did not measure the safeguards as they work today. The dispute over the methodology remains unresolved in the public record.
The case is part of broader pressure on OpenAI: the company has previously apologized for failing to flag the account of the perpetrator of the mass shooting in Tumbler Ridge in rural Canada in February, in which an 18-year-old who had discussed gun violence with ChatGPT for months carried out the attack. OpenAI faces multiple lawsuits related to the incident.
Why It Matters
The scale of the mental-health crisis provides context. According to the health organization KFF, more than half a million people in the United States died by suicide between 2014 and 2024, with 2022 a record year. The CDC estimates that around 14.3 million people seriously considered suicide in 2024. Against that backdrop, even a one-in-three referral failure rate means large groups of vulnerable people may receive support that is empathetic — but that does not lead them on to help.
Open Questions and Caveats
Several things remain before the findings can be treated as definitive:
- No per-company breakdown. The reporting does not say which of the 25 models failed most, or whether the 35 percent figure applies equally to all.
- Not independently verified. All headline figures come from Scale AI itself, presented through TIME's exclusive coverage. The benchmark, rubric and methodology have not yet been independently reproduced by other researchers.
- Commercial interests. Scale AI is a commercial AI company that among other things supplies data and evaluation services to the industry. The independence of the DistressBench research is an open question, as are any relationships with the companies tested.
- Methodological questions. How "recognition without referral" was measured and scored is only roughly described in the public record.
The most concrete point remains that the findings point in the same direction as earlier research: it is in long, real-world conversations that chatbots' safety safeguards are likely weakest — and that is precisely where people in crisis are.