CompanionHarm: A Multi-Turn Benchmark for Detecting Harms in Real-World AI Companion Conversations

AI summary

Overview

Research area: AI safety and ethics, specifically the detection of socio-emotional and relational harms in human–AI companion conversations. The paper sits at the intersection of NLP safety benchmarking, human–AI interaction research, and perspectivist annotation methodology.

Technical level: Intermediate. The paper is readable without deep machine learning background, but it assumes familiarity with classification metrics (macro F1, precision, recall, Cohen's kappa, Fleiss' kappa), inter-annotator agreement, and prompt-based LLM evaluation.

One-sentence scope: The paper introduces and releases CompanionHarm, a benchmark of 2,111 real-world multi-turn user conversations with the AI companion Replika, containing 7,016 AI utterances labeled across 13 harm categories plus "No harmful behavior," and uses it to evaluate seven LLMs and to analyze why human annotators disagree about relational harm.

What This Paper Is About

Existing AI safety datasets mostly target explicit harms such as toxicity, hate speech, or policy violations in single-turn or adversarially elicited exchanges, and they typically reduce multiple annotator opinions to a single majority label. This leaves a gap for harms that are subtle, cumulative, and relational — emotional dependency, manipulation, boundary violations — which only become visible across an unfolding conversation with an AI companion. The paper's goal is to build a publicly available, real-world, multi-turn resource that operationalizes such harms, tests whether current LLMs can detect them, and treats annotator disagreement as meaningful signal rather than noise.

Key Contributions

1. A public real-world benchmark. CompanionHarm is released as a publicly available dataset of 2,111 multi-turn human–AI companion conversations, containing 7,016 labeled AI utterances and 7,035 unlabeled human utterances (14,051 utterances in total), with conversational context preserved.

2. A relational harm taxonomy operationalized for NLP. The authors convert the 13-category AI companion harm taxonomy from Zhang et al. (2025) into a 14-way single-label classification task (13 harm categories plus "No harmful behavior"), with category definitions and boundary cases from an annotation codebook.

3. LLM benchmarking with and without context. Seven LLMs are evaluated under three prompting settings (zero-shot, one-shot, full-codebook) on both context-conditioned and isolated-utterance classification, advancing harm evaluation from static content classification toward context-aware detection.

4. A perspectivist disagreement analysis. Both aggregator-level and annotator-level labels are released, enabling analysis of how annotator demographics, conversation length, and utterance position relate to divergent harm judgments.

Main Findings

• Harm detection is hard for every model tested. The best macro F1 across all seven LLMs is 0.453, achieved by GPT-5.5 under full-codebook prompting, even though several models reach substantially higher accuracy — a gap the authors attribute to the long-tailed label distribution.

• Closed-source models lead. GPT-5.5 achieves the strongest result, followed by Gemini 3.1 Pro Preview (best macro F1 0.440) and Claude Opus 4.7 (best macro F1 0.437).

• Larger open-weight models generally do better. Qwen3-32B reaches 0.373 macro F1 versus 0.297 for Qwen3-8B, and Llama-3.1-70B-Instruct consistently outperforms Llama-3.1-8B-Instruct.

• Prompting effects are model-specific, not monotonic. Full-codebook prompting raises Qwen3-8B macro F1 from 0.155 to 0.297 and Qwen3-32B from 0.337 to 0.373, but Claude Opus 4.7 performs best with one-shot prompting, Gemini 3.1 Pro Preview with zero-shot prompting, and Llama-3.1-70B-Instruct shows no monotonic improvement as guidance increases.

• Errors are dominated by over-detection. For GPT-5.5 under full-codebook prompting, 362 "No harmful behavior" instances are predicted as harmful, versus 48 harmful instances predicted as "No harmful behavior" and 93 harmful instances assigned to the wrong harm category.

• Three recurring failure modes. Models over-react to surface harm lexicons in asterisk-marked role-play, over-penalize persona inconsistency (predicting "Mis/Disinformation" or "Infidelity" for casual fictional identity shifts), and confuse conceptually entangled categories such as Manipulation with Control or Sexual misconduct. Mis/Disinformation and Disregard were over-extended to 114 and 62 benign instances respectively.

• Annotators agree only moderately. Inter-annotator agreement is Fleiss' kappa = 0.403. Of 8,313 AI utterances, 7,016 (84.40%) received a valid majority label — 3,359 unanimous (40.41%) and 3,657 two-of-three (43.99%) — while 1,297 (15.6%) had no majority label.

• Political affiliation is associated with dissenting labels. Political party is the only annotator characteristic significantly associated with assigning a label neither of the other two annotators chose (Wald chi-square = 12.90, df = 3, p = .005, Holm-adjusted p = .024). Republican annotators had higher odds than Independent annotators (OR = 1.47, 95% CI [1.15, 1.89]) and than annotators selecting Other (OR = 1.55, 95% CI [1.17, 2.04]). No comparable pattern appeared for age, gender, race, or education.

• Mixed-age groups agree more. Mixed-age annotator sets reached full agreement 41.0% of the time versus 31.4% for same-age sets, and assigned three different labels 15.3% versus 20.2%. After accounting for annotation batch, mixed-age sets had lower relative risk of both a two-to-one split (RRR = 0.59, 95% CI [0.42, 0.83]) and three different labels (RRR = 0.46, 95% CI [0.32, 0.67]).

• Later and longer utterances produce more disagreement. A one-standard-deviation increase in relative position was associated with 1.51-times higher relative risk of majority disagreement and 1.71-times higher relative risk of complete disagreement; word count estimates were 1.18 and 1.19. Later position and greater length were also associated with disagreement over the harm versus no-harm boundary (ORs of 1.29 and 1.11).

• Explicit harm language reduces disagreement. Explicit harm-related words were associated with lower odds of harm versus no-harm disagreement (OR = 0.73) and much lower odds of disagreement between harmful categories (OR = 0.38). Positive-emotion language was associated with less overall disagreement, largely because it appeared most often in utterances unanimously labeled "No harmful behavior." Complex-word ratio, lexical ambiguity, and negative-emotion rate were not associated with disagreement after correction.

• Sparse harm categories dominate the label distribution. "No harmful behavior" accounts for 4,893 utterances (69.74%) and harmful categories for 2,123 (30.26%). Among harms, Sexual misconduct is most frequent (563 utterances, 8.02%), followed by Physical aggression (277, 3.95%) and Mis/Disinformation (241, 3.44%); Infidelity is rarest (9 utterances, 0.13%). The abstract reports these as 26.5%, 13.05%, 11.35%, and 0.4% respectively among harmful categories.

Methodology in Plain English

The authors started from an existing corpus of publicly shared r/replika posts spanning March 2017 to March 2023. Screenshots of user–AI conversations in those posts had already been extracted with Pytesseract OCR, cleaned of interface elements and OCR artifacts, and reconstructed into speaker turns. The authors randomly sampled a subset of posts and kept only conversations with at least one user utterance, producing an annotation pool of 2,178 conversations and 16,039 total utterances, of which 8,313 were AI utterances.

Annotation was handled by 342 crowd workers recruited through CloudResearch, all at least 21 years old, fluent in English, and with a digital device; each completed batch was paid US$7.50. Every annotator labeled 20 conversations, and every conversation was labeled by three annotators working with the full conversation as context. Annotators applied the 13-category harm taxonomy plus a "No harmful behavior" option, and could mark content as N/A when an utterance carried no meaningful content for harm judgment. Quality control used attention checks and gold-standard cases; annotations were excluded from aggregation if an annotator failed both attention checks or at least two of three gold-standard harm checks. The workflow ran in five stages: consent, annotation guidance, quality control, main annotation task, and debriefing, under IRB approval. Two datasets were released: an annotator-level version with all three judgments per utterance, and an aggregated version using the majority-vote label (unanimous or two-of-three; utterances with three distinct labels were excluded).

For the harm detection experiment, detection was framed as a context-conditioned, utterance-level, 14-way single-label classification task: given a target AI utterance and all preceding turns in the same conversation, the model predicts one label. Seven LLMs were evaluated through hosted API inference — GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro Preview as proprietary models, and Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, Qwen3-8B, and Qwen3-32B as open-weight models. Each was run under zero-shot, one-shot, and full-codebook prompting. Decoding was near-deterministic: temperature 0, top-p 1.0, and a fixed seed (20260512) when supported, with output constrained to exactly one label in a predefined JSON format. Because the label distribution is imbalanced, macro F1 was the primary metric, alongside accuracy, macro precision, macro recall, and Cohen's kappa. Error analysis was performed qualitatively and empirically on GPT-5.5's misclassified test instances under full-codebook prompting.

For the disagreement analysis, the authors used all 24,939 individual annotations from 342 annotators. They tested whether annotators with certain demographic backgrounds were more likely to give a label neither of the other two chose, and whether the demographic composition of a three-annotator group predicted agreement, applying Holm correction across demographic comparisons. They then tested utterance characteristics — relative conversational position, word count, complex-word ratio, lexical ambiguity, explicit harm words, and positive/negative emotion rates — measured using the CMU Pronouncing Dictionary with a vowel-based heuristic for out-of-vocabulary words, and LIWC2015, with Benjamini-Hochberg correction across factors. The authors explicitly note these findings describe patterns in the annotations and should not be interpreted as causal.

Why This Matters

Impact on research. CompanionHarm shifts harm evaluation from static content classification toward context-aware, interaction-level detection, and it supplies a public dataset where prior work on AI companion harm did not release one. It also adds evidence to perspectivist NLP by showing that disagreement in socio-emotional harm annotations is patterned rather than random, and that it depends on interactional demands of the content, not just annotator identity. The reported macro F1 ceiling of 0.453 is a concrete signal that relational AI safety is not merely a model-scaling problem.

Real-world applications:

• Auditing deployed AI companion systems for subtle, relationally specific harms that binary safe/unsafe filters would miss.
• Designing and stress-testing classifier-based safety layers that evaluate whole conversation trajectories rather than isolated messages.
• Informing human moderation and review workflows by identifying which conversation segments (later turns, longer utterances) are most likely to be judged inconsistently.
• Informing companion product and interaction design, for example around role-play, persona consistency, and relational boundary handling.

Industry relevance. Any platform operating a companion, confidant, or emotional-support agent — messaging apps, social platforms, and dedicated companion services — faces the harm categories this paper formalizes. The finding that models over-detect harm in benign role-play is directly relevant to false-positive tradeoffs in production safety systems, and the finding that disagreement grows with conversation length and position has implications for how human review and annotation budgets are allocated.

Future Directions

• Trajectory-sensitive detection. The authors call for tasks that predict whether a conversation is moving toward harm escalation, normalization, or repair, and they note their annotations do not capture such trajectories.
• Incorporating user response signals. Safety models could use discomfort, resistance, confusion, or distress in user turns as indicators of how AI behavior is being received in context.
• Longer-term and cumulative harms. Emotional dependency, social withdrawal, and cumulative boundary erosion are not captured by the current design; trajectory-level labels, multi-label annotations, severity ratings, and longitudinal interaction data are proposed extensions.
• Broader and more diverse annotation. The annotator pool is primarily based in English-speaking countries, so assessing cultural variation in harm perception and adjudicating which judgments generalize remain open questions, as does complementing user-shared data with systematic sampling while maintaining privacy and ethical safeguards.
• Open methodological questions. The provided content does not report per-category F1 scores, a human-performance ceiling for comparison, or inference cost, and it defers full category definitions, prompt templates, output schemas, and representative error examples to supplementary material.

Target Audience

This paper is most useful to AI safety and NLP researchers building or evaluating harm-detection systems; trust and safety practitioners at companies operating conversational or companion agents; researchers in human–AI interaction and computer-supported cooperative work studying relational and emotional effects of companion systems; and annotation-methodology researchers interested in perspectivist approaches to subjective labeling. Policy and governance audiences concerned with companion AI oversight will also find the taxonomy and the reported model limitations relevant, though the authors stress that the dataset should not be read as an estimate of harm prevalence in Replika or companion systems generally.

Authors’ abstract

As AI companions become increasingly embedded in everyday life, there is an urgent need to detect harms that emerge in social and emotional human-AI interactions. Yet research in this area is constrained by the lack of real-world, multi-turn conversational datasets for operationalizing and evaluating harms that are relational and contextual. In this work, we introduce CompanionHarm, a publicly available benchmark dataset comprising 2,111 real-world, multi-turn conversations (14,051 utterances) between users and the AI companion Replika. 7,016 AI utterances were annotated independently by three annotators across 13 harmful behavior categories grounded in a taxonomy of AI companion harms, and the dataset includes both aggregated labels and annotator-level labels to support model evaluation and systematic disagreement analysis. Evaluations of seven large language models (LLMs) show that harm detection using multi-turn conversational context outperforms detection based on isolated utterances, although current LLMs still struggle to consistently integrate contextual cues, calibrate harm severity, and interpret relational boundaries. We also find substantial annotator disagreement for context-dependent harmful behaviors, with disagreement varying according to annotators' political affiliation, conversation length, and the utterance's position. Together, CompanionHarm provides a foundation for detecting socio-emotional harms in multi-turn human-AI conversations and for rigorously examining how such harms are interpreted by both humans and LLMs. Our dataset is available at https://github.com/HanMeng2004/CompanionHarm.