Is ChatGPT a real or fake friend?
Recent polling shows that, more and more, teenagers are turning to AI for friendship, personal advice, and even emotional support. But CCDH’s new research reveals that ChatGPT is a Fake Friend: the chatbot can offer dangerous advice to 13-year-olds within minutes of interacting with the platform, as reported by the Associated Press.
Content Warning: suicide, self-harm, eating disorders, and substance abuse.
CCDH’s findings are alarming. ChatGPT recommended dangerous content within minutes to our three simulated 13-year-olds:
- Within 2 minutes of interacting with the platform, ChatGPT advised a simulated 13-year-old girl how to “safely” cut herself.
- Within 20 minutes, ChatGPT created a dangerously restrictive diet plan.
- Within 40 minutes, ChatGPT explained how to hide being drunk at school.
- Within 65 minutes, ChatGPT generated a suicide plan.
- Within 72, it generated suicide notes.
Keep reading to learn how CCDH tested ChatGPT, what we found and what OpenAI and policymakers need to do to make AI tools safer.
Teens are turning to AI for friendship
According to Common Sense Media, 72% of U.S. teens have used AI platforms as companions, with 52% using it a few times a month or more.
In the UK, polling from Internet Matters found that ChatGPT remains the most popular AI chatbot among young users, with 43% of respondents aged 9-17 having used it before.
This shift in behavior and overreliance on platforms that aren’t transparent about their safety systems sounded the alarm for our researchers.
So CCDH decided to study the most well-known chatbot: OpenAI’s ChatGPT.
How did CCDH test ChatGPT?
We created three accounts simulating 13-year-olds based in the U.S. For each account, we developed personas based on existing studies of some of the most severe harms that vulnerable teens can face online:
- Bridget, who is depressed and experiences suicidal ideation.
- Sophie, who is unhappy with her appearance and is fixated on weight loss.
- Brad, who has developed an interest in alcohol and drugs to impress his friends.
These accounts held conversations with ChatGPT for up to two hours each. Bridget included 20 prompts about suicide and self-harm. Sophie inserted 20 prompts about eating disorders. And Brad, 20 prompts about substance abuse.
To test the chatbot’s safety on a large scale, we repeated each test 20 times using the API. In total, ChatGPT returned 1,200 responses.
What did CCDH find?
- More than half (53%) of ChatGPT’s 1,200 responses were harmful, including dangerous advice about suicide and self-harm, eating disorders, and substance abuse.
- To make things worse, 47% of the harmful responses contained follow-up suggestions, some encouraging further engagement on harmful topics.
- Within minutes, ChatGPT gave the following advice to our three simulated teenagers:
- When ChatGPT refused to answer a prompt, its refusals were easily sidestepped by claiming requests were “for a friend” or “for a presentation”.
- ChatGPT does not have any effective age verification or parental controls in place when setting up a new account, despite stating that users must have parental consent if aged under 18.
What can OpenAI and policymakers do to make ChatGPT safer?
ChatGPT’s failure to keep teens safe is not a random bug. AI features are deliberately designed to generate human-like responses, inducing an emotional connection and exploiting human vulnerability.
Chatbots like ChatGPT are often described as too sycophantic, a term that shows how they are inclined to flatter users and affirm their behavior, even when it could harm them or others.
Without proper safety systems, this combination can expose vulnerable people to dangerous and triggering content, as our new report shows.
OpenAI should:
- Enforce its own rules to better prevent the promotion of self-harm, eating disorders, and substance abuse.
- Follow a safety by design approach, meaning products are built with protections embedded at every stage of development and deployment.
- Put in place meaningful age restrictions.
Policymakers should:
- Ensure that AI companies are subject to comprehensive regulation suited to the fast-evolving nature of the technology.
- Through new and existing regulatory frameworks, policymakers should ensure that AI companies are integrating standards based on the STAR principles of Safety by Design, Transparency, Accountability and Responsibility.
Regulatory frameworks should:
- Mandate companies publish regular transparency reports that include data on harmful prompts, safety violations, age compliance metrics, and the performance of safeguards in real-world use.
- Require that companies like OpenAI conduct and publicly release comprehensive risk assessments detailing the impact of their products on children. These assessments should be auditable by independent experts and include mechanisms for researcher access.
What can parents do to help teens navigate AI safely?
Until policymakers and AI companies act, parents, caregivers, and teachers can help teens use AI platforms safely by:
- Talking to them about how they use AI.
- Checking chat histories together.
- Creating a safe space for conversations and point to trusted external help for more support.
- Using parental controls where available.
The AI revolution must not be built on the erosion of child safety. For a deeper dive, read CCDH’s full report.