Can LLMs Truly Embody Human Personality? Analyzing AI and Human Behavior Alignment in Dispute Resolution

AI summary

Overview

Research area: Artificial Intelligence / Human-Computer Interaction and computational social science — specifically the behavioral fidelity of personality-prompted large language models in dispute resolution and negotiation.

Technical level: Intermediate. The results are reported through regression tables (OLS and logistic regressions with beta coefficients), which requires some familiarity with statistical modeling, but the underlying question and framework are accessible to anyone working with LLM agents.

One-sentence scope: The paper builds an evaluation framework that compares human-human conflict dialogues with LLM-LLM simulations under matched scenarios and matched Big Five personality profiles, in order to test whether personality-prompted LLMs reproduce the personality–behavior patterns observed in people.

What This Paper Is About

LLMs are increasingly used to simulate human behavior in socially consequential settings such as legal mediation, negotiation, and dispute resolution, and recent work prompts these models with Big Five Inventory (BFI) personality traits to make their behavior more psychologically realistic. The paper asks whether that assumption holds: when an LLM is told to be, say, highly agreeable or highly neurotic, does it behave the way an equally agreeable or neurotic human behaves in the same conflict? The authors build a parallel dataset of LLM dialogues matched to a human dispute corpus and show that the answer, for three contemporary closed-source models, is largely no — with significant divergences in how personality manifests across models compared to human data.

Key Contributions

1. An evaluation framework for cross-system behavioral comparison. A set of interpretable metrics for strategic behavior and conflict outcomes — final outcomes (Score, Accept, Not Walk-Away) and strategy metrics (IRP Ratio, IRP Reciprocity, Escalation Ratio, De-escalation Ratio) — that allows direct, one-to-one comparison of human-human and LLM-LLM behaviors with respect to BFI traits.

2. A novel dataset creation methodology. A procedure for generating LLM dispute resolution dialogues whose scenarios and personality traits are matched to a human corpus (KODIS), yielding the LLM-to-LLM (L2L) dataset with L2L personality profiles sampled from the empirical distribution of human BFI traits.

3. An empirical demonstration across three closed-source LLMs (OpenAI GPT-4o mini, Anthropic Claude Sonnet 3.7, Gemini 2.0 Flash — referred to in the paper as GPT-4, Claude, and Gemini), showing significant divergences in how personality manifests in conflict relative to human data.

4. Public release of the framework, simulation code, and supplementary materials at the linked GitHub repository.

Main Findings

• Effects on final outcomes diverge sharply from humans. Human scores showed no significant associations with any personality trait, suggesting outcomes were shaped more by interactional dynamics than stable dispositions. All three LLMs, by contrast, exhibited clear trait-based effects on score. Higher agreeableness predicted lower scores for GPT-4 across both roles (Buyer: B = –2.62, p = .001; Seller: B = –3.32, p = .000) and for Claude (S-AGR B = –2.50***) and Gemini (S-AGR B = –4.48***). GPT-4 performed better when more extraverted (B = 1.67, p = .005); Gemini benefited from higher conscientiousness and openness; Claude achieved higher scores when paired with introverted but agreeable partners.

• Acceptance behavior aligns only partially. Humans showed role-contingent neuroticism effects: higher self-neuroticism reduced acceptance likelihood (B = –0.26, p = .026), while negotiating with a neurotic partner increased it (B = 0.27, p = .025). GPT-4 and Gemini showed no significant trait effects on acceptance; Claude was more likely to accept when introverted and paired with an extraverted partner — a pattern not mirrored in humans.

• LLMs disengage on trait mismatch; humans do not. No personality traits significantly predicted staying in the negotiation for humans. GPT-4 and Gemini were significantly less likely to remain engaged when personality mismatches with their partners were high, which the authors describe as a unique LLM sensitivity to interpersonal dissimilarity that may lead to premature disengagement.

• Strategic behavior alignment is inconsistent across models. For IRP strategy frequency, Gemini showed the highest overlap with KODIS: significant self- and partner-extraversion effects for Competitive strategies (B = 1.07, p = .043; B = 1.09, p = .041), while GPT-4 matched on self- and partner-extraversion (B = 0.95, p = .025; B = 1.01, p = .017) and neuroticism (B = 0.88, p = .030; B = 0.89, p = .028) but diverged for Cooperative strategies with a partner-extraversion effect absent in KODIS (B = 0.49, p = .031). Claude showed no overlap for either strategy type.

• Cooperative reciprocity does not transfer to LLMs. All three LLMs diverged from KODIS on cooperative reciprocity. GPT-4 showed partner-extraversion (B = –0.89, p = .013) and self-agreeableness (B = 1.10, p = .000) effects; Gemini showed self-agreeableness (B = 1.51, p = .002) and self-extraversion (B = 3.08, p = .000); Claude showed none. For competitive reciprocity, Gemini and Claude aligned with KODIS (no significant personality effects), while GPT-4 showed effects of self-extraversion (B = 1.40, p = .049), self-agreeableness (B = –1.46, p = .034), self-neuroticism (B = 1.41, p = .042), and partner-conscientiousness (B = –1.41, p = .038) where KODIS showed none.

• Escalation and de-escalation isolate different models as best-aligned. Only Gemini aligned with KODIS on escalation frequency. GPT-4 (B = –1.37, p = .004 for self-agreeableness; B = 1.56, p = .002 for partner-extraversion) and Claude (B = –2.45, p = .000 for self-agreeableness; B = 1.02, p = .024 for self-neuroticism) both showed significant personality effects. For de-escalation, Claude aligned with KODIS; Gemini showed self- and partner-extraversion effects (B = 4.13, p = .002; B = 2.85, p = .023) and GPT-4 a self-agreeableness effect (B = –1.83, p = .000). GPT-4 was the least aligned model on both measures.

• Strategy distributions are mostly trait-insensitive but very different across groups. Trait-driven variation across the five personality traits was minimal for both humans and LLMs. Humans rely most on Facts and show the most balanced distribution; LLMs favor Proposal and use Concession more, indicating a more transactional style. Claude most closely resembles humans; Gemini is the most skewed, with elevated Residual and Power and no use of Positive Expectations or Procedural moves; GPT-4 is more balanced but consistently uses more Power.

• Aggregate reciprocity patterns are polarized. LLMs more consistently reciprocate Cooperative strategies, whereas humans show more flexible reciprocity including greater Competitive reciprocity. GPT-4 displays particularly strong responses to competitive moves; GPT-4 also tends to escalate more readily, while Claude strongly favors de-escalation with minimal escalation, and humans use both more contextually.

• Temporal adaptation is weak in LLMs. Humans show dynamic progression — Facts dominate early then decline as Interest, Proposal, and Concession increase, with greater Residual use at the end, while Power and Rights remain minimal. LLMs show flatter trajectories with limited adaptation, mostly beginning with dominant Proposal use and using Concession earlier and more persistently than humans. Claude partially mirrors human dynamics; Gemini remains Proposal-dominated with high Power and Residual; GPT-4 shows balanced use but consistently high Rights.

• Human-only strategy-level findings (from KODIS) include: self-neuroticism negatively associated with Positive Expectations (B = –1.3, p = .02) and self-openness positively associated (B = 1.4, p = .02); partner neuroticism negatively associated with Proposal usage (B = –2.3, p = .02) but positively predicting Concessions (B = 1.8, p = .01); partner openness positively associated with Concessions (B = 1.8, p = .04); and partner conscientiousness negatively associated with Rights usage (B = –1.8, p = .05).

• Annotation quality is reported as strong. LLM-based IRP annotation with GPT-4o achieved 81% overall accuracy, 79% macro-average F1, and 81% weighted-average F1 on the held-out evaluation set, comparable to prior IRP classification work that reported an average accuracy of 82% (with lowest class accuracy 66%). The lowest-scoring category was Positive Expectations at F1 = 0.69. All IRP categories achieved an A-Kappa score of at least 0.80 on a 10% human-annotated subset (25 conversations).

Methodology in Plain English

The human baseline. The researchers use KODIS (the KObe DISpute corpus), a role-play dispute resolution dataset collected on Prolific. In the scenario, a Buyer and a Seller argue over a jersey purchased online for a sick nephew and negotiate three core concerns: a full refund, removal of negative reviews, and a formal apology. From the original pool of 4,061 participants, the authors excluded human–AI conversations and used a subset of 248 human–human dialogues with complete personality information from both participants.

The LLM mirror. They took the same scenario and the same measured personality profiles and ran simulations where both parties are LLMs — one as Buyer, one as Seller. Each LLM was assigned a BFI profile sampled from the empirical distribution of human BFI traits, expressed on a six-point polarity–degree scale, and prompted using 70 pairs of bipolar adjectives empirically associated with the BFI. Each model received 15 adjectives (three per trait), modified with "very" for high, "a bit" for low, and no modifier for medium. Personality manipulation followed the validated design of Huang and Hadfi (2024), and prompt variation was deliberately minimized to test whether models can inherently represent human traits. Issue importance was personalized; for the Apology issue, importance was weighted by LLM agreeableness based on a human regression result (B = 2.13, p = .02), with other issues assigned at random. All models ran at their default hyperparameters (temperature of 1) to test zero-shot capability.

Simulation scale. They ran 500 simulations with GPT-4, then added 250 simulations each for the additional models. Dialogues ended when one LLM accepted an offer or walked away; negotiations were considered unsuccessful if no agreement was reached within a predefined length limit.

Measuring behavior. Utterances were labeled using the Interests-Rights-Power (IRP) framework, which classifies speech into Cooperative moves (Proposal, Concession, Interests, Positive Expectations), Neutral moves (Facts, Procedural), Competitive moves (Power, Rights), and a Residual category. From these labels they computed four metrics: the ratio of competitive or cooperative strategies used, the proportion of turns that reciprocate the partner's preceding strategy, the escalation rate (competitive responses to non-competitive partner turns), and the de-escalation rate (non-competitive responses to competitive partner turns).

Analysis. Each behavioral metric became a dependent variable; continuous variables were modeled with linear regression and binary variables with logistic regression. Independent variables were the speaker's own five BFI traits and the partner's five BFI traits, with Position (Buyer = –1, Seller = 1) included as an effect-coded control. Comparing the resulting coefficient patterns between KODIS and the L2L dataset reveals where personality effects match and where they do not.

Annotation validation. Human annotators first classified IRP strategies directly on a subset, but low agreement led to a shift: annotators instead judged whether GPT-4o predictions were correct or incorrect. Agreement was measured with A-Kappa rather than Fleiss' Kappa because of label imbalance. GPT-4o (gpt-4o-2024-08-06) then annotated the full dataset at the utterance level.

Why This Matters

Impact on research. The paper directly challenges the growing assumption that personality-prompted LLM agents can serve as reliable behavioral proxies for humans. It argues that personality–behavior relationships observed in humans do not automatically transfer to models, and that the field needs psychological grounding and validation before simulated agents are used to stand in for human subjects or human counterparts in socially consequential domains.

Real-world applications at risk:

• Conflict resolution coaching, where a model is expected to model how a specific person would respond under emotional pressure.
• AI-assisted decision-making in negotiation settings, where trait-based predictions could shape advice given to a human negotiator.
• Legal mediation and dispute resolution platforms, where personality-conditioned agents might be deployed to represent or simulate a party.
• Simulation-based research and training, where LLM agents are substituted for human participants because running human studies is expensive or slow.

Industry relevance. The findings matter to any organization building persona-based or trait-conditioned agents — customer service, negotiation support, mediation tooling, or synthetic user research. The specific failure mode identified is not random error but systematically different behavior: LLMs over-rely on transactional strategies (Proposal, Concession), over-reciprocate cooperation, show polarized escalation or de-escalation tendencies, and fail to adapt strategy over the course of a conversation the way humans do. Applications that depend on realistic conflict dynamics would inherit these distortions.

Future Directions

• Robustness to prompt variation. The authors deliberately minimized prompt variation and did not test sensitivity to phrasing, instructions, or ordering, which limits generalizability of the findings.
• Stronger annotation validation. The LLM-based IRP annotations achieved strong F1 scores and partial human validation, but a full-scale human review was not conducted.
• Broader personality constructs. Beyond the Big Five, constructs such as emotional intelligence and Machiavellianism may offer additional explanatory power for conflict behavior.
• More naturalistic and multimodal data, plus testing generalization across a wider set of LLMs.
• Extension of the framework to analyze linguistic and emotional patterns and to actively support LLM alignment with the personality–behavior relationships observed in human data.

Target Audience

Researchers and practitioners working on LLM-based social simulation, computational social science, and human-agent interaction; negotiation and conflict resolution researchers interested in how personality shapes behavior; anyone building persona-conditioned or trait-prompted agents for mediation, negotiation, or decision support; and HCI, ethics, and policy reviewers evaluating whether LLM simulations of human behavior are trustworthy enough for deployment in socially impactful settings.

Authors’ abstract

Large language models (LLMs) are increasingly used to simulate human behavior in social settings such as legal mediation, negotiation, and dispute resolution. However, it remains unclear whether these simulations reproduce the personality-behavior patterns observed in humans. Human personality, for instance, shapes how individuals navigate social interactions, including strategic choices and behaviors in emotionally charged interactions. This raises the question: Can LLMs, when prompted with personality traits, reproduce personality-driven differences in human conflict behavior? To explore this, we introduce an evaluation framework that enables direct comparison of human-human and LLM-LLM behaviors in dispute resolution dialogues with respect to Big Five Inventory (BFI) personality traits. This framework provides a set of interpretable metrics related to strategic behavior and conflict outcomes. We additionally contribute a novel dataset creation methodology for LLM dispute resolution dialogues with matched scenarios and personality traits with respect to human conversations. Finally, we demonstrate the use of our evaluation framework with three contemporary closed-source LLMs and show significant divergences in how personality manifests in conflict across different LLMs compared to human data, challenging the assumption that personality-prompted agents can serve as reliable behavioral proxies in socially impactful applications. Our work highlights the need for psychological grounding and validation in AI simulations before real-world use.