Detecting AI Impostors: How Do Middle Schoolers Identify LLM Agents in a Live Collaborative Setting?

AI summary

Overview

Research area: Natural Language Processing / AI literacy and human AI-detection, with a focus on adolescents and education.

Technical level: Intermediate. The game design and qualitative analysis are accessible to a general reader, while the agentic architecture, similarity metrics, and mixed-effects regression require some familiarity with NLP and statistics.

Scope: The paper introduces DoppelBot, a cooperative social deduction game in which middle school students chat with LLM agents that impersonate them, and reports how students' detection accuracy, reasoning strategies, and privacy attitudes change across repeated sessions.

What This Paper Is About

LLMs can imitate how people write, which makes it hard to tell whether a chat partner is human or a machine. Most prior detection studies test isolated adults on short static text snippets, leaving open the question of how young people detect AI during live, multi-user social interaction. This paper builds DoppelBot, a game where AI "doppelgängers" impersonate their human targets, and uses it to measure how middle schoolers detect those agents as the impersonation gets more personalized.

Key Contributions

1. The authors state this is the first study analyzing how adolescents identify AI systems within collaborative settings.
2. They created one of the first AI impersonation games for studying how humans collaboratively detect AI agents, releasing the full source code and an online version for researchers and educators.
3. They release a novel dataset of game transcripts and voting behavior (available to researchers with IRB approval), with an example use case in Appendix D.2.
4. They document a behavioral shift in students from surface-level linguistic cues toward social, contextual, and embodiment-based detection strategies.

Main Findings

• Detection accuracy improved over time: Students went from 79.3% accuracy on Day 1 to 86.7% on Day 2, which the authors describe as marginally significant under a paired t-test (p = .10).

• Human annotators were better but flat across days: Two researchers (Cohen's κ = 0.86) reached 91.9% accuracy on Day 1 and 92.3% on Day 2. The authors argue that the near-constant expert score suggests the Day 2 improvement came from the students rather than an easier task.

• Regression model supports learning across rounds and sessions: In a mixed-effects logistic regression on 215 votes, the Round Index coefficient was β = 0.190 (p ≤ 0.05) and the Second Day coefficient was β = 0.415 (p ≤ 0.05), indicating gains both within a match and across sessions. Remaining players also had a positive effect (β = 0.616, p ≤ 0.001), with negative interactions for Round × Remaining Players (β = -0.268, p ≤ 0.01) and Day × Remaining Players (β = -0.607, p ≤ 0.05).

• AI agents became more aligned with their humans, not less: Across BERT, LIWC, METEOR, and ROUGE-L, average paired similarity between students and their doppelgängers increased from Day 1 to Day 2 after students completed Partnered Interview Worksheets (PIWs).

• Students exploited AI limitations: Messages coded as meta (leveraging information unavailable to the AI) rose from 3.0% to 15.4% (κ = .918), and coordinating votes (crvt) rose from 12.6% to 24.4% (κ = .822), while random content (rand) fell from 37.7% to 9.0% (κ = .864).

• Embodiment probes worked: One student asked "WHAT COLOR'S THE WALL?", another asked "What is your full last name?", and one asked how many people were in the chat; the AI reported six "people" even though only three of the six actors were human.

• Students reflected on privacy, but it was the least frequent theme: Across 417 annotated interview utterances (κ = .707), AI Detection was the largest category (n = 148), followed by Social–Emotional Responses (n = 71), Learning Adaptation (n = 60), Perceptions of AI (n = 55), Design Feedback (n = 44), and Privacy Concerns (n = 30), which accounted for 7% of utterances.

• Privacy concern scores rose on all three subscales: With 13 matched pre/post responses for Combining Data (CD) and 12 for Data Permanence (DP) and Improper Access (IA), means increased from 3.667 to 3.820 (CD), 3.917 to 4.056 (DP), and 4.278 to 4.472 (IA). The authors call these findings preliminary given the small sample.

• Unease at convincing mimicry: Some students reported discomfort when the AI adopted slang they never provided or inferred interests they had not disclosed, which the authors compare to the uncanny valley. All 29 pilot participants agreed DoppelBot was a fun way to learn about AI.

Methodology in Plain English

The researchers built a chat game for three to five students at a time. Each student fills out a form with their name, grade, favorite animal, hobby, and a fun fact, and the system uses that information to build a personalized AI double for them. Every player gets a code name and color, so humans and bots look identical in the chat except for those labels. Each round starts with a mandatory icebreaker question, then students chat freely for 100 seconds before voting. A majority vote is required to eliminate someone, and the game runs for as many rounds as there are human players (three rounds here).

Behind the scenes, the impersonation runs on three components, all implemented with gpt-4o-mini. A Decide-to-Respond agent decides whether the bot should speak at all, a Response Generation agent writes a baseline reply grounded in the persona, and a Stylizer rewrites that reply to match the target student's own prior messages. Temperature was set to 0.5 for the first two components and 0.9 for the Stylizer.

The study ran in two places. First, a pilot at a public STEM-charter school during a two-day afterschool AI EXPO, where 29 of 70 assenting participants played DoppelBot. Second, a deeper evaluation during a four-day "AI For Everyone!" summer camp. At the camp, students played on Day 1, then completed Partnered Interview Worksheets on Day 2 that captured slang and topics so their bots could be more personalized. Data came from three streams: semi-structured group interviews that were recorded, transcribed, and coded by two researchers with adjudication; a pre/post survey on privacy concerns adapted from Menard and Bott (2025); and a categorical analysis of chat messages, of which 2,182 unique messages remained after removing blank ones, with 50 randomly sampled rounds double-labeled.

Why This Matters

Impact on research. Existing human AI-detection work almost entirely uses isolated adults judging static text. This paper reframes detection as a live, collaborative, adversarial social task and shows that context and group interaction, not just linguistic fluency, drive authenticity judgments. The released transcripts and voting data (via IRB approval) give other researchers a way to study AI impersonation with a population that is rarely tested.

Real-world applications:

• Classroom AI literacy curricula, since the game requires no prior computer science, ethics, or math background and fits a full session in about thirty minutes.
• Design of platform safeguards, by showing which cues users actually rely on—especially probes about physical embodiment and context that current chat agents fail.
• Privacy education, since students connected impersonation to identity theft and spam-call scams after playing.
• Safety evaluation of persona-driven LLM agents, using the game as a testbed for measuring how convincing an impersonator is to real users.

Industry relevance. Any company deploying conversational agents in education, social, or customer-facing settings has an interest in how convincingly a model can imitate a specific individual and how easily users notice. The result that students inferred undisclosed preferences (dinosaurs, chess) from provided data also speaks directly to data minimization and personalization practices.

Future Directions

• Test whether these findings hold across regions, languages, and educational settings, since the study covers English-speaking middle schoolers in a single U.S. state.
• Run age-stratified comparisons against adults, which the paper notes it cannot do because it lacks a direct control group.
• Use fractional factorial designs to isolate the effects of live social dynamics, bot detection, and identity defense, which the current design evaluates simultaneously.
• Test a broader range of LLMs such as Llama and Claude to account for model-specific "stylistic signatures," and examine how the results generalize to real-world platforms and longer-term interaction.

Target Audience

This paper is most useful to AI literacy and computing education researchers, K-12 curriculum designers, and HCI or NLP researchers studying human detection of machine-generated text and impersonation. Educators looking for a low-barrier classroom activity will find the game design and educator framing relevant, and AI safety and trust-and-safety practitioners working on impersonation and persona-driven manipulation will benefit from the behavioral findings.

Authors’ abstract

LLMs can imitate how people write, which raises concerns about impersonation, trust, and detection in social settings. These concerns are especially important for adolescents, who use generative AI frequently but may struggle to recognize it. We introduce \textit{DoppelBot}, a cooperative social deduction game designed to study how young people detect and respond to AI impersonation. Through studies with middle schoolers, we investigate whether a DoppelBot prompts reflection on privacy and impersonation, how repeated exposure affects AI-detection accuracy as agents become more personalized, and which strategies students use to identify AI doppelgängers. We find that students' detection accuracy improves over time, driven by a shift from relying on linguistic cues to leveraging shared social and contextual signals. Students also demonstrated an understanding of AI limitations such as embodiment and reflected on broader issues such as data privacy. To support future research, we release an anonymized dataset of game transcripts and voting behavior.