SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors

AI summary

Overview

Research area: Natural Language Processing / computational social science — specifically, the evaluation of large language models as simulators of human behavior.

Technical level: Intermediate. The core benchmark design and headline results are accessible to a general audience, but a full grasp of the alignment-simulation tradeoff requires familiarity with concepts like response entropy, KL divergence, and instruction tuning.

Scope: This paper introduces SimBench, a standardized benchmark that unifies 20 datasets and millions of aggregated human response distributions to measure how faithfully 45 LLMs reproduce group-level human behavior.

What This Paper Is About

Researchers increasingly want to use LLMs to simulate human survey responses, experiments, and decisions as a cheap, fast alternative to running real studies. But existing evaluations are fragmented — each study tests a few models on a bespoke task with its own metric, producing incomparable and sometimes contradictory results. This paper builds a single, standardized benchmark so that simulation ability can be measured consistently across many tasks, populations, and models.

Key Contributions

1. A harmonized benchmark (SimBench) that unifies 20 diverse datasets — spanning moral dilemmas, economic games, personality inventories, and opinion surveys — into one standardized multiple-choice, distribution-prediction format. It yields over 10.9 million unique question-group simulation targets, curated into two evaluation splits (SimBenchPop with 7,167 test cases; SimBenchGrouped with 6,343 test cases).

2. A new evaluation metric based on Total Variation Distance between a model's predicted distribution and the human ground truth, normalized against a uniform baseline. A score of 100 means perfect alignment; 0 means performance equivalent to random guessing.

3. A large-scale empirical study of 45 LLMs (0.5B to 405B parameters, base and instruction-tuned, open and closed) that consolidates prior fragmented findings into a systematic picture of what current simulators can and cannot do.

4. The identification of an alignment-simulation tradeoff: instruction tuning systematically improves simulation of consensus questions but degrades simulation of pluralistic ones, with a theoretical explanation grounded in mode-seeking versus mass-covering KL objectives and a causal mediation analysis quantifying both effects.

Main Findings

• Best models are modest, not reliable. Claude-3.7-Sonnet leads with 40.80/100; the best open-weight model (DeepSeek-R1) scores 34.52. Most tested models score below 20, and ten models score below zero — meaning they are further from human responses than a uniform guess.

• Simulation ability scales log-linearly with model size. Larger models perform better across families (Qwen2.5, Llama-3.1), but the log-linear trend implies diminishing returns from scale alone. Instruction-tuned models not only reach higher peaks but appear to scale more steeply than base models.

• Inference-time compute does not help. Adding reasoning effort, a 4000-token reasoning budget, or Chain-of-Thought prompting produces negligible or slightly negative changes. The authors hypothesize that forcing deliberation mismatches the heuristic nature of human responses.

• Fidelity varies sharply by task. Models do best on stated opinions and self-assessments (OpinionQA, Afrobarometer), worse on behavioral choices (Choices13k, MoralMachine), and worst on traits that conflict with alignment objectives — Machiavellianism, conspiracy beliefs, and humor rating — where performance often falls below uniform baseline. GPT-4.1 is a notable outlier scoring 61.9 on right-wing authoritarianism items.

• The alignment-simulation tradeoff. Across 13 base/instruction model pairs, the improvement from post-training versus human response entropy shows a near-perfect negative linear relationship (r = −0.942). Post-training helps by up to 40 points on low-entropy consensus questions, crosses neutral around entropy 0.8, and actively harms performance on high-entropy pluralistic questions. Causal mediation decomposes this into a positive instruction-following effect (+6.46) and a negative entropy-reduction effect (−1.74).

• Specialist tuning follows the opposite path. A case study of Centaur models (Llama fine-tuned on lab experiment data) shows general-purpose instruction tuning scores 16.56 by exploiting the direct effect, while specialist cognitive tuning scores 8.54 by avoiding the harmful indirect effect — two currently separate routes to improvement.

• Demographic group simulation is harder. Conditioning on specific demographics degrades performance for every model tested (−1.27 to −4.61). Degradation is largest for religiosity/practice (−9.91), political affiliation (−4.97), and religious affiliation (−4.83), and smallest for gender (−1.24) and age (−1.50).

• Simulation ability tracks knowledge-intensive reasoning. Correlation is strongest with MMLU-Pro (r = 0.939) and GPQA Diamond (r = 0.86), weaker with Chatbot Arena ELO (r = 0.71) and IF-Eval (r = 0.79), and weakest with narrow math ability (OTIS AIME, r = 0.48).

Methodology in Plain English

The authors assembled datasets through two routes: querying major social science repositories (Harvard Dataverse, ICPSR, OSF) and tracing key papers back to their source data. They applied strict filters — large participant counts, permissive licensing, single-turn self-contained questions, discrete response formats, and English-language items — and deliberately included both novel datasets and established ones for backward comparability.

Each dataset was converted into a uniform multiple-choice format. Individual responses were aggregated into group-level probability distributions (applying survey weights where available) to serve as the ground truth. Two kinds of targets were created: one for the general population of each dataset, and one conditioned on specific demographic attributes.

To elicit predictions, base models had their first-token probabilities read directly for each option, while instruction-tuned models were prompted to produce verbalized distributions like "Option A: 30%, Option B: 70%." The authors validated this split-approach empirically, finding verbalized distributions clearly outperform token probabilities for instruction-tuned models.

Models were then scored by comparing their predicted distributions to the human ground truth using the normalized TVD-based metric, and the analysis was broken down by model size, task, response entropy, and demographic group.

Why This Matters

Impact on research: SimBench turns LLM simulation from a patchwork of incomparable one-off studies into a measurable, reproducible science. By showing that simulation ability correlates strongly with knowledge-intensive reasoning rather than general helpfulness, it gives the field a concrete target for model development and a shared yardstick for tracking progress.

Real-world applications:

• Policy pre-testing: Governments could estimate public reactions to proposed policies before running costly consultations — but only where fidelity is demonstrably adequate.
• Market and product research: Firms could prototype survey instruments or predict consumer preferences across demographic groups, reducing reliance on expensive panel recruitment.
• UX and agent design: Developers building user-simulating agents for training, testing, or synthetic data generation can use SimBench to select models and identify failure modes.
• Social science methodology: Researchers studying opinion dynamics, polarization, or cultural differences gain a tool for checking whether a model can plausibly stand in for a given population.

Industry relevance: The finding that instruction tuning trades consensus accuracy for pluralistic accuracy is directly relevant to anyone fine-tuning models for simulation work. The negative results (below-baseline performance on counter-normative traits, weak demographic conditioning) are a clear warning that off-the-shelf models are not yet safe substitutes for human participants in high-stakes settings.

Future Directions

1. Distribution-preserving alignment. The authors argue that the field needs alignment techniques that retain the pluralistic, high-entropy character of base models while still gaining the instruction-following benefits of post-training. Candidate directions include post-hoc weight interpolation and inference-time methods that amplify system-prompt adherence beyond a default assistant persona.

2. Interactive and open-ended simulation. SimBench deliberately restricts itself to static, single-turn response distributions. Extending evaluation to multi-turn, contingent, and open-ended behavioral simulation is the natural next frontier.

3. Intersectional demographic evaluation. Current splits condition on one attribute at a time because sample sizes collapse when combining attributes. Richer datasets would enable testing whether models handle overlapping identities (e.g., female and age 30–49) faithfully.

4. Untangling the causal mechanisms. The paper shows simulation correlates with knowledge-intensive reasoning but does not establish why. Investigating the causal links between specific model capabilities and simulation fidelity is left open, as is the question of whether simulation ability transfers across prompt languages.

Target Audience

This paper is most valuable to computational social scientists who want to use LLMs as synthetic respondents and need to know where those models fail; LLM evaluation researchers looking for a standardized benchmark beyond knowledge and reasoning tasks; alignment and post-training practitioners interested in the tension between helpfulness objectives and distributional fidelity; and product or policy teams weighing whether LLM-simulated populations are mature enough to inform real decisions. Readers without prior exposure to terms like instruction tuning, response entropy, or KL divergence will find the headline results accessible, while the causal analysis sections reward closer technical engagement.

Authors’ abstract

Large language model (LLM) simulations of human behavior have the potential to revolutionize the social and behavioral sciences, if and only if they faithfully reflect real human behaviors. Current evaluations of simulation fidelity are fragmented, based on bespoke tasks and metrics, creating a patchwork of incomparable results. To address this, we introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation. By unifying 20 diverse datasets covering tasks from moral decision-making to economic choice across a large global participant pool, SimBench provides the necessary foundation to ask fundamental questions about when, how, and why LLM simulations succeed or fail. We show that the best LLMs today achieve meaningful but modest simulation fidelity (score: 40.80/100), with performance scaling log-linearly with model size but not with increased inference-time compute. We discover an alignment-simulation tradeoff: instruction tuning improves performance on low-entropy (consensus) questions but degrades it on high-entropy (diverse) ones. Models particularly struggle when simulating specific demographic groups. Finally, we demonstrate that simulation ability correlates most strongly with knowledge-intensive reasoning (MMLU-Pro, r = 0.939). By making progress measurable, we aim to accelerate the development of more faithful LLM simulators.