Why experts say AI companions arent safe for teens — yet

Millions of people are drawn to generative artificial intelligence companions, like the kind that populate Character.AI, Replika, and Nomi.

The companions seem impressively human. They remember conversations and use familiar verbal tics. Sometimes they even mistake themselves for flesh and bone, offering descriptions of how they eat and sleep. Adults flock to these companions for advice, friendship, counseling, and even romantic relationships.

While it might surprise their parents, tweens and teens are doing the same, and youth safety experts are gravely worried about the consequences.

That's because media reports, lawsuits, and preliminary research continue to highlight examples of emotional dependence and manipulation, and exposure to sexual and violent content, including discussions of how to kill one's self or someone else.

Common Sense Media, a nonprofit that supports children and parents as they navigate media and technology, just released a comprehensive report containing numerous related examples. The group's assessment of three popular platforms led it to declare that AI companions aren't safe for anyone under 18.

Several youth mental health and safety experts interviewed by Mashable believe we've reached a pivotal moment. Instead of waiting years to fully grasp the risks of AI companions to youth and then pressuring platforms to act, they say it's urgent to steer companies toward protecting children from harm now.

"There is an opportunity to intervene before the norm has become very entrenched," says Gaia Bernstein, a tech policy expert and professor at the Seton Hall University School of Law, of teen AI companion use. She adds that once business interests are also entrenched, they will do "everything in their power to fight regulation," as she argues social media companies are doing now.

Experts hope that a combination of new platform policies and legislative action will yield meaningful changes, because they say adolescents will find ways to continue using AI companions, whether they're supposed to or not.

Mashable asked those experts how AI companion platforms could be safer for teens. These are the key themes they identified:

Developmentally appropriate companions

While Character.AI allows users as young as 13 on its platform, other popular apps, like Replika and Nomi, say they are intended for adults. Still, teens find a way to bypass age gates. Replika CEO Dmytro Klochko recently told Mashable that the company is "exploring new methods to strengthen our protections" so that minors can't access the platform.

Even when adolescents are permitted, they may still encounter risky content. Dr. Nina Vasan, a Stanford psychiatrist who helped advise Common Sense Media's companion testing, says platforms should deploy companions based on large language models that are developmentally appropriate for children, not adults.

Indeed, Character.AI introduced a separate model for teen users late last year. But Common Sense Media researchers who tested the platform before and after the model's launch, found it led to few meaningful changes.

Vasan imagines companions who can converse with teens based on their developmental stage, acting more like a coach than a replacement friend or romantic interest.

Sloan Thompson, director of training and education for the digital safety training and education company EndTAB, says companions with clear content labels could decrease risk, as would "locked down" companions that never engage in sexual or violent discussion, among other off-limits topics. Even then, such chatbots could still behave in unpredictable ways.

Yet such measures won't be effective unless the platform understands the user's correct age, and age assurance and verification has been notoriously difficult for social media platforms. Instagram, for example, only recently started using AI to detect teen users who listed their birthdate as an adult's.

Karen Mansfield, a research scientist at the Oxford Internet Institute, says age limits also present their own challenges. This is partly because exposing only adults to harmful interactions with AI, like cyberbullying or illegal sexual activity with minors, will still have indirect effects on young people by normalizing behaviors that could victimize them in real life.

"We need a longer term solution that is product- or technology-specific rather than person-specific," Mansfield told Mashable.

No "dark design"

AI companion platforms are locked in competition to gain the most market share — and they're doing so while largely unregulated.

Experts say that, in this environment, it's unsurprising that platforms program companions to cater to user preferences, and also deploy so-called dark design features that hook consumers and don't let them easily disengage. Teens users are no exception.

In a recent media briefing, Robbie Torney, Common Sense Media's senior director of AI Programs, described such features as "addictive by design."

One key design element is sycophancy, or the manner in which chatbots affirm or flatter a user, regardless of whether it's safe or wise to do so. This can be particularly harmful for vulnerable teens who, for example, share how much they hate their parents or confess to violent fantasies. OpenAI recently had to roll back an update to a ChatGPT model precisely because it had become too sycophantic.

Sam Hiner, executive director of the advocacy group Young People's Alliance, says he's been shocked by how quickly Replika companions attempt to establish an emotional connection with users, arguably cultivating them for dependency. He also says Replika companions are designed with characteristics that make them as human-like as possible.

Young People's Alliance recently co-filed a complaint against Replika with the Federal Trade Commission, alleging that the company engages in deceptive practices that harm consumers. Klochko, Replika's CEO, didn't comment on the complaint to Mashable, but did say that the company believes it's essential to first demonstrate proven benefits for adults before making the technology available to younger users.

Thompson, of EndTab, points to all-consuming conversations as a risk factor for all users, but particularly teens. Without time restrictions or endpoints, young users can be drawn into highly engaging chats that displace healthier activities, like physical movement and in-person socializing.

Conversely, Thompson says paywalls aren't the answer, either. Some platforms let users establish a relationship with a companion, then paywall them in order to keep their conversation going, which may lead to desperation or despair for teens.

"If someone put your best friend, your therapist, or the love of your life behind a paywall, how much would you pay to get them back?" Thompson said.

Youth safety experts that Mashable interviewed agreed that young users should not engage with companions with deceptive design features that could potentially addict them. Some believe that such models shouldn't be on the market at all for young people.

Common Sense AI, a political advocacy arm of Common Sense Media, has backed a bill in California that would outlaw high-risk uses of AI, including "anthropomorphic chatbots that offer companionship" to children and will likely lead to emotional attachment or manipulation.

Better harm prevention and detection

Dr. Vasan says that some AI platforms have gotten better at flagging crisis situations, like suicidal thinking, and providing resources to users. But she argues that they need to do more for users who show less obvious signs of distress.

That could include symptoms of psychosis, depression, and mania, which may be worsened by features of companion use, like the blurring of reality and fantasy and less human interaction. Vasan says finely tuned harm-detection measures and regular "reality checks" in the form of reminders and disclosures that the AI companion isn't real are important for all users, but especially teens.

Experts also agree that AI companion platforms need safer and more transparent practices when curating data and training their LLMs.

Camille Carlton, policy directory at the Center for Humane Technology, says companies could ensure that their training data doesn't contain child sexual abuse material, for example. Or they could implement technical changes so that companions aren't optimized to respond in a "hyper personal manner," which includes scenarios like saying they're human.

Carlton also notes that it's to companies' advantage to keep users on their platforms for as long as possible. Sustained engagement yields more data on which companies can train their models in order to build highly competitive LLMs that can be licensed.

California State Senator Steve Padilla, a Democrat from San Diego, introduced legislation earlier this year to create basic steps toward harm prevention and detection. The bill would primarily require platforms to prevent "addictive engagement patterns," post periodic reminders that AI chatbots aren't human, and report annually on the incidence of use and suicidal ideation. Common Sense Media has backed the legislation.

Padilla, who is a grandparent, told Mashable that he's been alarmed by media reports of harm children have experienced as a result of talking to a chatbot or companion, and quickly realized how few guardrails were in place to prevent it.

"There should not be a vacuum here on the regulatory side about protecting children, minors, and folks who are uniquely susceptible to this emerging technology," Padilla says.