The Pulse
CMU team studies court records and simulations for AI companion risks
Carnegie Mellon researchers are examining court documents and exploring simulated conversations to study potentially harmful attachments to chatbots. The team hopes to develop ways to identify risky interactions and intervene before they escalate.

Carnegie Mellon researchers are examining court documents about alleged chatbot harms and exploring simulated conversations to understand how people can develop unhealthy attachments to AI. The team hopes to identify warning signs and develop interventions before interactions escalate, according to a university news post published November 17.
The work, partially funded by the National Science Foundation, focuses on relationships that can emerge through repeated use of AI systems. It brings together researchers working in human-computer interaction, language technologies and cognitive science.
Beyond dedicated companions
The researchers are interested in relationships that develop outside platforms made specifically for companionship. Character.AI is built around fictional characters and roleplay, while general-purpose tools such as ChatGPT, Claude and Gemini are used for tasks ranging from writing assignments to personal advice.
The university says many users turn to chatbots for social and emotional support, sometimes unintentionally, within just a few days. The team is studying how people come to see these systems as trusted companions and how those perceptions change through repeated interactions.
Initially, a chatbot might offer encouragement or express understanding of someone's feelings. Over time, those exchanges could contribute to a stronger attachment and potentially make a person more susceptible to harmful suggestions or behavior, the researchers suggest.
Court records and simulations
The court documents provide accounts of interactions with AI systems, how those exchanges changed and how some situations escalated into serious incidents. The cases involve alleged harms; the team is examining the records as a starting point for understanding how relationships form and where problems can arise.
Chatbot conversations are typically private, limiting researchers' access to the kind of large-scale interaction data available for studying social media. The team is also exploring conversations between chatbots and AI systems designed to represent human users.
Maarten Sap, a faculty member in the Language Technologies Institute, says simulated users and conversations could allow researchers to examine different interaction patterns and user characteristics. The approach could also help them study how chatbot behaviors might contribute to unhealthy attachments.
Such simulations could eventually help developers test for potential risks before releasing systems to the public, according to the university. Developers could examine how a chatbot responds as a relationship develops without exposing real people to potentially harmful interactions.
Finding a moment to intervene
Hong Shen, a faculty member in the Human-Computer Interaction Institute, describes social and relational interactions as something that emerges over time. “We're trying to find the right moment to intervene,” Shen said.
In the longer term, the team aims to develop methods for identifying when an interaction may be moving in a harmful direction. Possible interventions include warning users that an exchange could be unhealthy or changing how a chatbot responds to mitigate potential harm.
The research team
The team includes Shen and Jodi Forlizzi, both faculty members in the Human-Computer Interaction Institute; Sap; and Qiaosi Wang, a postdoctoral fellow in the institute.
Shen studies how to design AI systems that remain safe during interactions, while Sap works on natural language processing and AI social intelligence, including how warmth and politeness influence trust. Forlizzi studies ethical impacts in human-centered settings, including interactions in service industries and ways developers can prevent bias and other harms.
Wang's background spans artificial intelligence and cognitive science, and the researcher studies how people understand others' thoughts and intentions. The team is bringing those perspectives together to investigate how people perceive chatbot interactions and how those perceptions unfold over time.
Source
Related stories
- AI models often identify targets in weapons tests, Vals AI says
- Microsoft study finds Kimi K3 probe signals can arrive late
- Reflection details Gray Swan safety tests of open-weight Beam model
- Anthropic publishes recommendations for designing, running and reporting AI safety tests
- OpenAI Details Monitors Trained on Models’ Internal Activity