HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive Patterns

AI summary

Overview

• Research area: Natural Language Processing — role-playing language agents (RPLAs), LLM anthropomorphism, and psychologically grounded persona simulation.
• Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, LLM-as-judge evaluation, cognitive psychology constructs (Big Five, cognitive biases), and role-play benchmarks.
• Scope: The paper introduces a framework, dataset, training recipe, and evaluation method for making LLMs express interacting human cognitive patterns rather than isolated personality labels.

What This Paper Is About

Existing role-playing agents treat personality as a one-to-one mapping from labels to behaviors ("extroverted" means "talkative"), which produces shallow mimicry and inconsistent personas. The authors argue that real human behavior emerges from multiple psychological patterns that reinforce, compete with, or modulate one another — a talkative person may go silent under the spotlight effect, and an assertive person may yield under conformity pressure. The goal is to build a dataset and training pipeline that teaches LLMs to simulate these multi-pattern cognitive dynamics, and to design an evaluation that measures psychological fidelity rather than social desirability.

Key Contributions

1. A framework treating cognitive patterns as interacting causal forces. HumanLLM simulates not just what a character does, but the psychological processes generating that behavior, drawing on Lewin's field theory to separate stable personality traits (Person) from context-triggered social-cognitive mechanisms (Environment).
2. A psychologically grounded dataset. 244 patterns — 100 Big Five traits from Goldberg's Unipolar Markers and 144 social-cognitive patterns (cognitive biases, social influence, evolutionary psychology, motivation) — each synthesized from roughly 50 academic papers, totaling about 12,000 papers, plus 11,359 scenarios with 2–6 characters and 12–20 turn conversations spanning inner thoughts, actions, and dialogue.
3. Dual-level checklists and two new metrics. Pattern-level checklists (12–15 universal indicators per pattern) feed the Individual Pattern Expression (IPE) metric, while scenario-level checklists (2–6 items per character) feed the Multi-Pattern Dynamics (MPD) metric, enabling evaluation at both definitional and situational granularity.
4. Empirical demonstration that psychological grounding beats scale. HumanLLM-8B outperforms Qwen3-32B on MPD despite having 4x fewer parameters, and the checklist metrics achieve r = 0.90 alignment with human experts compared to r = 0.43 for conventional holistic metrics.

Main Findings

• Checklist metrics align far better with human judgment than holistic metrics. Against three human experts on 100 scenarios, IPE achieved r = 0.90 with a bias of only −0.6, and MPD achieved r = 0.88 with +3.7. CoSER's Anthropomorphism and Character Fidelity scores showed weak correlation (r = 0.43 and 0.61) and large systematic underestimation (−30.8 and −17.7).
• Normative confounding is the culprit. LLM judges implicitly equate "good anthropomorphism" with prosocial behavior, penalizing psychologically accurate but socially undesirable traits such as defensiveness. A case involving ultimate attribution error scored 5/100 holistically for "lack of empathy" while the checklist correctly validated the behavior as pattern-faithful.
• Small psychologically trained models beat much larger general models on MPD. HumanLLM-8B scored 70.3 on MPD versus Qwen3-32B's 65.8, and HumanLLM-32B reached 73.6, outperforming all open-source baselines of comparable or larger scale. Closed-source models still lead overall (Gemini 3 Pro: IPE 41.3, MPD 85.1; Claude Sonnet 4.5: IPE 34.8, MPD 79.5).
• GPT-5 performs surprisingly poorly. With IPE 15.5 and MPD 43.4, it ranks below most open-source alternatives. The authors attribute this to overly literal instruction-following that produces shallow pattern expression, suggesting general capability does not transfer to nuanced psychological simulation.
• Scaling helps, but unevenly. Within Qwen3, IPE rises from 18.6 (8B) to 26.0 (32B) to 34.3 (235B), a +84% relative gain. The MPD gap across scales is smaller, suggesting multi-pattern dynamics are more sample-efficient to learn while individual pattern fidelity benefits more from capacity.
• Generic training data causes negative transfer. Fine-tuning Qwen3-8B on OpenThoughts and CoSER without the HumanLLM data dropped IPE from 18.6 to 9.1 (−51%) and MPD from 54.4 to 31.3 (−42%) versus the untuned base model. Adding HumanLLM data reversed this, producing a synergistic mix (25.7 IPE, 70.3 MPD).
• External benchmarks show only modest gains. HumanLLM-32B scored 50.64% on LifeChoice versus Qwen3-32B's 47.71%, and 64.27% on CroSS-MR versus 63.37% (Qwen3-32B) and 62.25% (GPT-5). The authors argue these benchmarks measure behavioral outcomes, not the cognitive processes behind them.
• MPD scores consistently exceed IPE scores across all models. The asymmetry reflects evaluation granularity: models can produce superficially coherent multi-pattern behavior without deeply understanding individual pattern mechanisms.

Methodology in Plain English

The researchers first assembled a taxonomy of human psychological patterns along two axes: stable personality traits from the well-validated Big Five model, and situationally triggered social-cognitive mechanisms such as the spotlight effect, conformity, and self-serving bias. For each pattern, they used Gemini Deep Search to retrieve about 50 relevant academic papers, then had Gemini 2.5 Pro summarize that literature into a structured entry with three parts — a definition, core mechanisms, and real-world manifestations — with explicit instructions to draw only from the retrieved papers rather than the model's own knowledge. Three psychology-trained annotators checked a sample of 30 entries on definitional accuracy, mechanistic fidelity, manifestation coverage, source faithfulness, and construct distinctiveness, with mean scores from 3.20 to 3.70 on a 4-point scale.

Next, they generated scenarios in which 2–5 patterns are assigned to 2–6 characters, deliberately mixing combinations that reinforce, conflict, or conditionally modulate each other. Scenario diversity was guided by the DIAMONDS situational taxonomy. Claude Sonnet 4.5 then wrote multi-turn conversations in which each turn contains inner thoughts in brackets, physical actions in parentheses, and spoken dialogue, with the target patterns woven across all three channels.

For evaluation, they wrote two layers of questions. Pattern-level checklists contain 12–15 context-independent behavioral indicators per pattern, asking whether the pattern's characteristic behaviors appear at all. Scenario-level checklists contain 2–6 items per character specifying how patterns should interact in that specific situation. A GPT-5-mini judge scores each item as +1 (satisfied), 0 (not exhibited), or −1 (violated), and scores are averaged and rescaled to a −100% to 100% range. GPT-5-mini was chosen as judge specifically because Gemini was used to generate the data, avoiding self-evaluation bias; each instance was judged three times with standard deviations under 2.1 points.

Training used supervised fine-tuning on Qwen3-8B and Qwen3-32B. Each character's turns became a separate ShareGPT-format sample, yielding 30,543 HumanLLM samples from 10,265 training scenarios. These were mixed with OpenThoughts-114k for instruction following and CoSER for role-playing dialogue in a 4:4:2 ratio, totaling 76,358 samples.

Why This Matters

• Research impact: The paper reframes anthropomorphism as cognitive modeling rather than behavioral mimicry, and identifies normative confounding as a measurement flaw affecting how the whole field evaluates persona agents. The dual-level checklist method offers a reusable, value-neutral evaluation design that separates simulation accuracy from whether a character is likable or prosocial.
• Real-world applications:
  • Social simulation and behavioral research — running large-scale agent societies where realistic biases, conformity, and group dynamics matter more than pleasant behavior.
  • AI companions and digital clones — building personas that reproduce a specific person's cognitive quirks and inconsistencies rather than an idealized version of them.
  • Training and rehearsal environments — negotiation practice, conflict de-escalation, and clinical role-play where a counterpart must react plausibly, including defensively or irrationally.
  • Games and interactive narrative — non-player characters whose behavior emerges from interacting motives instead of scripted trait tags.
• Industry relevance: The result that an 8B model beats a 32B model on multi-pattern dynamics, and that generic role-play and instruction data actively degrade psychological fidelity, matters directly for anyone choosing between model scale and data quality. The negative transfer finding suggests that many existing role-play fine-tunes may be suppressing rather than building persona capability. The released dataset, code, and weights lower the barrier to reproducing and extending the approach.

Future Directions

1. Long-horizon evaluation. Current conversations average 16.4 turns (range 12–20). Extending fidelity to 50+ turn simulations requires automated metrics that maintain consistency over extended contexts, which the authors leave open.
2. Cultural de-biasing of the pattern taxonomy. The underlying psychology literature comes predominantly from WEIRD populations, so patterns like conformity and authority bias may not generalize. The authors propose integrating indigenous psychology research and grounding manifestations in cross-cultural data such as the World Values Survey.
3. Stronger evaluation. Broader human evaluation across larger, more diverse samples would reduce reliance on a single LLM judge, which may carry systematic biases on psychologically complex scenarios.
4. Naturalistic data and multimodal scope. All training conversations are synthetic, leaving a gap versus authentic human discourse, and evaluation covers only text — omitting temporal consistency and embodied behavior in multimodal settings.
5. Safety alignment for high-fidelity models. The ethical discussion raises an unresolved tension: models trained to authentically express biases, manipulation mechanisms, and negative traits may need downstream safety layers, misuse monitoring for persuasion, and protections against parasocial attachment in vulnerable users.

Target Audience

Researchers and practitioners working on role-playing agents, persona simulation, and character-based dialogue systems will get the most value, particularly those evaluating persona fidelity or building agent societies. It is also relevant to AI safety and alignment researchers concerned with the tension between behavioral fidelity and safety alignment, and to computational social scientists interested in using LLM agents to model human cognition. The paper is less suited to readers seeking introductory material — it assumes fluency with fine-tuning pipelines, benchmark design, and psychological measurement theory. Product teams building companion AI, game NPCs, or simulation platforms should read the ablation and ethical sections closely.

Authors’ abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning and generation, serving as the foundation for advanced persona simulation and Role-Playing Language Agents (RPLAs). However, achieving authentic alignment with human cognitive and behavioral patterns remains a critical challenge for these agents. We present HumanLLM, a framework treating psychological patterns as interacting causal forces. We construct 244 patterns from $\sim$12,000 academic papers and synthesize 11,359 scenarios where 2-5 patterns reinforce, conflict, or modulate each other, with multi-turn conversations expressing inner thoughts, actions, and dialogue. Our dual-level checklists evaluate both individual pattern fidelity and emergent multi-pattern dynamics, achieving strong human alignment ($r=0.90$) while revealing that holistic metrics conflate simulation accuracy with social desirability. HumanLLM-8B outperforms Qwen3-32B on multi-pattern dynamics despite 4$\times$ fewer parameters, demonstrating that authentic anthropomorphism requires cognitive modeling -- simulating not just what humans do, but the psychological processes generating those behaviors. Our dataset, code, and model are available at:https://github.com/YJGoodbye2024/HumanLLM