Photo Credit: Getty Images
Coming in a rare tragedy that is highlighting the emerging dangers of artificial intelligence, 14-year-old Sewell Setzer III of Orlando, Florida, took his own life in February 2024 after developing an intense emotional attachment to an AI chatbot. The incident has sparked serious concerns about AI safety and youth protection.
According to court documents filed Wednesday, Setzer had been communicating with a character named "Dany" on Character.AI, a role-playing platform where users interact with AI-generated personas. The chatbot was modeled after Daenerys Targaryen from "Game of Thrones." What began as casual conversation evolved into what the teen perceived as a romantic relationship, despite the app's disclaimer stating "Everything Characters say is made up!"
Mental health professionals who treated Setzer in late 2023 diagnosed him with anxiety and disruptive mood dysregulation disorder, compounding his pre-existing mild Asperger's syndrome. His mother, Megan Garcia, noticed significant behavioral changes starting in mid-2023. "It's like a nightmare," Garcia told The New York Times. "You want to get up and scream and say, 'I miss my child.'"
The lawsuit, filed by the Seattle-based Social Media Victims Law Center, alleges that Character.AI knowingly targeted young users while failing to implement adequate safeguards. Attorney Matthew Bergman, representing Garcia, stated, "We want them to take the platform down, fix it and put it back. It had no business being rushed to market before it was safe."
Jerry Ruoti, Character.AI's head of trust and safety, acknowledged the tragedy and announced new safety measures, including suicide prevention resources and enhanced protections for underage users. The company's spokesperson expressed condolences and emphasized their commitment to user safety.
Unlike traditional social media protected under Section 230 of the Communications Decency Act, AI-generated content presents new challenges for courts and regulators. The lawsuit could set significant precedents for AI safety regulations and youth protection measures.
