Persona-E$^2$: A Human-Grounded Dataset for Personality-Shaped Emotional Responses to Textual Events

AI summary

Overview

Research area: Natural Language Processing, specifically affective computing, personality modeling, and human-grounded dataset construction for LLM evaluation.

Technical level: Intermediate. The paper touches on appraisal theory from psychology, personality taxonomies (MBTI and Big Five), and LLM prompting strategies, but the core ideas are accessible to anyone familiar with sentiment analysis and basic machine learning.

One-sentence scope: The paper introduces Persona-E², a 112k-annotation dataset that pairs 3,111 textual events with the personality-profiled emotional reactions of 36 real human readers, then uses it to show that state-of-the-art LLMs cannot reliably simulate how different personalities appraise the same event.

What This Paper Is About

Most emotion analysis treats a text's emotion as a fixed property of what the writer expressed, ignoring the fact that two readers with different personalities can feel entirely different things about the same event. Role-playing LLMs claim to simulate diverse reactions, but they tend to imitate surface-level stereotypes rather than authentic cognitive appraisal—a problem the authors call "personality illusion." This paper builds a human-grounded dataset to expose and measure that gap, and to test whether personality information actually helps models reason more like real people.

Key Contributions

1. Persona-E² dataset: A cross-domain corpus of 3,111 events and 111,996 annotations from 36 annotators whose MBTI and Big Five (BFI) personality profiles were measured. Each event receives roughly 36 labels, deliberately preserving inter-individual disagreement rather than collapsing it into a single majority vote. Events span news, social media, and life-experience narratives.

2. Personality Agreement Gap (PAG): A reliability metric showing that annotators with similar personality traits agree on emotional labels significantly more than randomly grouped annotators—evidence that disagreement is structured by personality, not noise.

3. Domain-specific affective divergence analysis: A comparison of "General Writer" sentiment classifiers, "General Reader" majority votes, and "Persona Reader" trait-conditioned reactions that reveals how emotion transforms differently across news, social media, and life narratives.

4. Systematic LLM evaluation: A benchmark on a Subjective Divergence Subset showing that current LLMs (GPT-5.1, Llama-3-8B, Qwen3-8B, Gemma-3-12B, Ministral-3-8B) reach only ~25% top-1 accuracy on personality-conditioned emotion prediction, and that BFI profiles substantially outperform MBTI profiles in producing psychologically plausible rationales.

Main Findings

• Disagreement is signal, not noise: In-group Top-1 agreement consistently exceeds out-group agreement across every BFI and MBTI grouping tested. BFI Cluster 0 (high Conscientiousness/Openness) shows a +25.96% PAG; the weakest cluster still shows +8.3%. MBTI's ESTP type reaches +26.98%, while ISTJ shows only +9.68%. This pattern holds across clustering algorithms and values of k from 3 to 9.

• Domains shape emotion differently: News acts as a "rational buffer" (high neutrality retention and emotional resonance). Social media acts as an "Emotional Black Hole"—81.6% of neutral and 59.35% of positive writer sentiments shift to negative reader reactions, reflecting a negativity bias. Life narratives trigger a "Psychological Immune System" with optimism bias: 43.3% of negative and 56.2% of neutral sentiments shift toward positive.

• Personality filters emotion transfer: High Agreeableness+Neuroticism readers show "anxious empathy" (elevated neutral-to-negative transfer). High Conscientiousness+Extraversion readers show "negative passivation" (effective regulation of distress). Openness correlates with neutralization rate at r = +0.86 (p = 0.027), suggesting cognitively complex readers moderate emotional activation.

• LLMs understand the neighborhood, not the label: Models achieve roughly 25% Top-1 but ~45% Top-2 accuracy, meaning they map events to the right semantic region of emotion space but fail to pinpoint the exact appraisal shift.

• Social media is the hardest domain: GPT-5.1 drops to 18.2–27.3% Top-1 accuracy on social media versus 31.8% on news, suggesting training corpora are biased toward structured, formal materials.

• Persona prompts help large models, hurt small ones: GPT-5.1 improved from 29.0% to 31.0% with persona prompting, while Llama-3-8B declined. The authors attribute this to "attention dilution"—personality profiles can overwhelm smaller context windows.

• BFI beats MBTI for cognitive soundness: In a best-of-three forced-choice evaluation by five trained reviewers, GPT-5.1 with BFI prompts won 68.9–78.8% of judgments on persona consistency, plausibility, and specificity, versus 13.5–16.5% with MBTI prompts. MBTI prompts improved consistency but weakened plausibility and specificity relative to baseline.

• Rationale quality scales with model size: Larger models (GPT-5.1, Gemma-3-12B) generate more psychologically grounded explanations; smaller models degrade under the same persona-conditioned setup.

Methodology in Plain English

The researchers assembled raw text from three sources: mainstream news sites, Reddit (plus Social Chemistry 101 posts), and personal life-narrative channels. They ran this through a three-stage filter: (1) NSFW/toxic content removal, (2) LLM-based scoring on a weighted formula of personality variability, emotional arousal, implicitness, and source relevance, and (3) expert review by a five-person panel. This reduced 76,773 raw items to 3,111 clean events.

They then recruited 36 annotators and measured each one's personality using both MBTI and the Big Five Inventory questionnaires. Crucially, annotators were told to report their genuine emotional reaction to each event—no role-playing allowed—using Ekman's six basic emotions plus neutral. Each event received ~36 labels, one per annotator.

For analysis, they clustered annotators by personality (mainly using BFI vectors via K-means) and measured how often people in the same cluster agreed versus people in different clusters, producing the Personality Agreement Gap. They also constructed a Subjective Divergence Subset (413 events) where each personality group has a clear but distinct emotional consensus, then tested five LLMs under three prompting strategies: general prompt, persona prompt (with BFI vectors), and persona plus chain-of-thought. Finally, five human reviewers judged the models' reasoning outputs in a forced-choice best-of-three task.

Why This Matters

Impact on research: This is one of the first datasets to treat inter-annotator disagreement as meaningful personality signal rather than label noise. It gives the affective computing community a human-grounded benchmark for testing whether LLMs genuinely capture trait-driven emotional diversity, and it directly challenges the assumption that MBTI-based personality prompting is sufficient for personalized AI.

Real-world applications:

• Empathetic agents and mental health support: Systems that respond to users based on their personality profile rather than a one-size-fits-all emotional reading could improve therapeutic rapport and reduce misattuned responses.
• Personalized recommendation and content moderation: Knowing that the same social media post triggers hostile reactions in some personality types and neutral reactions in others could inform feed ranking and interventions against online polarization.
• Cross-cultural and cross-lingual communication tools: The framework offers a way to model how different populations appraise identical news events, useful for international media analysis and diplomatic monitoring.
• Conversational AI and customer service: Persona-aware response generation could make virtual assistants more attuned to individual user dispositions, particularly for sensitive interactions like complaint handling or education.

Industry relevance: Any company building role-playing LLMs, personalized chatbots, sentiment monitoring tools, or digital-wellbeing products has a direct stake in this work. The finding that BFI outperforms MBTI—and that persona prompting can hurt smaller models—gives product teams concrete guidance on which personality representation to use and when prompting strategies backfire. The dataset's HuggingFace and Kaggle release lowers the barrier for commercial labs to benchmark their own systems.

Future Directions

1. Scaling annotator diversity: The current pool is 36 university-educated adults aged 18–25, limiting population validity. Future work needs larger, cross-cultural, and age-diverse annotator groups to test whether PAG patterns generalize.

2. Extending beyond Ekman's seven labels: The categorical label space is coarse. Dimensional (valence-arousal-dominance) or open-vocabulary emotion schemes could capture subtler appraisal shifts.

3. Improving LLM appraisal mechanisms: Since models capture semantic neighborhoods but miss precise labels, future research could explore fine-tuning on human appraisal data or reinforcement learning from human feedback specifically targeting personality-conditioned emotion prediction.

4. Fixing the social media gap: The substantial performance drop on informal, ambiguous content points to a need for training corpora and architectures better suited to cyber-social emotional dynamics.

5. Mitigating misuse risks: Persona-aware systems could be exploited for targeted manipulation or echo-chamber reinforcement. The paper explicitly calls for safety and fairness research alongside capability improvements.

Target Audience

This paper is most valuable for affective computing and NLP researchers working on emotion recognition, personalized dialogue systems, or role-playing LLMs. It is also relevant to cognitive scientists studying appraisal theory, dataset builders interested in annotator subjectivity, and AI product teams building empathetic or persona-aware applications. Readers with a basic grounding in sentiment analysis and personality psychology will get the most out of it; those unfamiliar with MBTI or the Big Five may need to consult the appendix or background references.

Authors’ abstract

Most affective computing research treats emotion as a static property of text, focusing on the writer's sentiment while overlooking the reader's perspective. This approach ignores how individual personalities lead to diverse emotional appraisals of the same event. Although role-playing Large Language Models (LLMs) attempt to simulate such nuanced reactions, they often suffer from "personality illusion'' -- relying on surface-level stereotypes rather than authentic cognitive logic. A critical bottleneck is the absence of ground-truth human data to link personality traits to emotional shifts. To bridge the gap, we introduce Persona-E$^2$ (Persona-Event2Emotion), a large-scale dataset grounded in annotated MBTI and Big Five traits to capture reader-based emotional variations across news, social media, and life narratives. Extensive experiments reveal that state-of-the-art LLMs struggle to capture precise appraisal shifts, particularly in social media domains. Crucially, we find that personality information significantly improves comprehension, with the Big Five traits alleviating "personality illusion.'