Discovering Differences in Strategic Behavior Between Humans and LLMs

AI summary

Overview

Research area: Behavioral game theory, LLM evaluation, AI safety and interpretability, computational social science.

Technical level: Intermediate. The paper's concepts (opponent modeling, Nash equilibrium, reinforcement-style value learning) are accessible, but following the method fully assumes familiarity with maximum likelihood estimation, cross-validation, and program synthesis.

Scope: A structural comparison of human and frontier LLM strategic behavior in Iterated Rock-Paper-Scissors, using AlphaEvolve to discover interpretable programmatic behavioral models directly from gameplay data.

What This Paper Is About

Humans and large language models both play strategic games, and researchers increasingly treat LLMs as stand-ins for human participants in social-science studies. The problem is that existing behavioral game theory models are built from human-centric hypotheses, so they may miss ways in which LLM strategic behavior is genuinely different. This paper directly discovers readable, code-based models of behavior from human and LLM gameplay data, then compares those models to explain where the two diverge and why.

Key Contributions

1. The first application of automated symbolic model discovery to characterize human and LLM behavior through interpretable behavioral models.
2. A demonstration that frontier LLMs win at higher rates and against more complex opponents than humans in IRPS, even without agentic scaffolds.
3. A structural explanation, derived from AlphaEvolve-discovered programs, for that performance gap: frontier LLMs maintain more sophisticated opponent models than humans.
4. A matched LLM dataset (90,000 choices per model, collected under conditions matched to a 411-participant human dataset) enabling direct comparison, plus a cross-generalization analysis showing LLM behavior is not fully captured by behavioral models built for humans.

Main Findings

• Frontier models win more than humans: Gemini 2.5 Flash, Gemini 2.5 Pro, and GPT 5.1 achieve higher win rates against the nonadaptive bots than humans do, while all agents remain beneath an oracle that knows the bot policy. Aggregate win-rate trends are otherwise similar, decreasing as bots grow more complex.
• Faster pattern detection: Gemini 2.5 Flash/Pro and GPT 5.1 converge to a near-optimal win rate sooner than humans, while humans eventually reach a similar win rate at convergence.
• One model degrades over time: GPT OSS 120B's win rates decrease over the course of a game, which the authors hypothesize reflects an inability to synthesize information in longer-context scenarios. It performs near random against the two most complex nonadaptive bots, as do humans.
• AlphaEvolve beats behavioral game theory baselines: AlphaEvolve significantly outperforms the CS-EWA baseline on every dataset (all p < 0.001, Wilcoxon signed rank with Bonferroni correction; Z ≤ −7.148), and fits similarly to an RNN for the human and GPT OSS 120B datasets while improving over it for Gemini 2.5 Flash/Pro and GPT 5.1.
• Shared building blocks: Across all agents, the simplest-but-best programs contain value-based learning and opponent modeling. All programs except GPT OSS 120B use a 3x3x3 Q-table over the current action given the prior joint action; GPT OSS 120B maintains only a single-dimensional Q-vector.
• The key structural difference is opponent-model dimensionality: humans and GPT OSS 120B use single-dimensional opponent models tracking raw historical choice frequency, Gemini 2.5 Flash/Pro use 3x3 models conditioned on the opponent's previous move, and GPT 5.1 uses a 3x3x3 model.
• The same reasoning level, different depth: all agents are characterized as level-1 players in cognitive hierarchy theory. Choice stickiness was the most frequently occurring cognitive bias, appearing in the human, Gemini 2.5, and GPT OSS 120B programs.
• LLMs are not human proxies: the human program predicts LLM behavior poorly and LLM programs predict human behavior poorly (all p < 0.001, Wilcoxon signed-rank with Bonferroni corrections; Z ≤ −5.73). Gemini 2.5 Flash, Gemini 2.5 Pro, and GPT 5.1 predict one another well, with cross-generalization differences from the best-fit model of 0 for the 2.5 Pro dataset, 0.005 for the 2.5 Flash dataset, and 0.017 for the GPT 5.1 dataset.

Methodology in Plain English

The researchers studied Iterated Rock-Paper-Scissors, where each game lasts 300 rounds and players receive 3 points for a win, 0 for a tie, and −1 for a loss against any of 15 opponent bots. Nonadaptive bots map a small amount of game history to a fixed next move with 90% probability; adaptive bots track sequential dependencies in the opponent's moves and counter the predicted move with 100% probability. The human data comes from an existing dataset of 411 participants making 129,087 choices. For each of four models (Gemini 2.5 Pro, Gemini 2.5 Flash, GPT 5.1, and open-source GPT OSS 120B), the authors collected 20 games of 300 rounds against all 15 bots, for 90,000 choices per model.

Rather than hand-writing hypotheses about how players behave, they used AlphaEvolve, an evolutionary program-discovery system that uses Gemini 2.5 Flash to propose and refine Python functions. Each candidate program takes in the player's past choices, the opponent's past choices, and past rewards, and outputs a probability distribution over the next move, with learnable parameters fit by stochastic gradient descent. The fitness function balances twofold cross-validated likelihood against the Halstead effort, a software-engineering measure of how much effort is needed to comprehend a program, producing a Pareto frontier of predictive accuracy versus interpretability. From that frontier, the authors select the "simplest-but-best" program: the simplest program whose evaluation likelihood falls within ε = 0.005 of the best observed. They compare these discovered programs against three baselines: the Nash equilibrium (play each move with probability 1/3), an extension of Sophisticated Experience-Weighted Attraction called CS-EWA that keeps separate attraction vectors for every joint history of length L = 2, and a GRU-based recurrent neural network as a flexible black-box reference.

Why This Matters

Impact on research: The paper shows a way to characterize LLM behavior structurally rather than through aggregate statistics alone, and it provides direct evidence that frontier LLMs are not valid digital twins for human strategic decision-making. It also contrasts with earlier work by Fan et al. (2024), who found GPT-3 and GPT-3.5 could not beat random play in a 10-round IRPS setting and that GPT-4 remained worse than humans, indicating how much LLM strategic capability advanced between model generations. The approach does not depend on a model's reasoning traces, which the paper notes do not always reflect behavior, making it a complement to Chain-of-Thought monitoring.

Real-world applications:

• Evaluating and monitoring LLM capabilities in negotiation, bargaining, and other strategic social interactions where models may out-maneuver human counterparts.
• Vetting social-science and market-research studies that use LLM agents as simulated human participants, since structural divergences could misrepresent actual human populations.
• Guiding behavioral alignment work that aims to make LLM decision-making more consistent with human expectations.
• Providing a general template for discovering interpretable models of agent behavior in other controlled multi-agent settings.

Industry relevance: All authors were employed by Google during the study, and Google leads development of the Gemini models evaluated. The disclosure is relevant because the results favor Gemini 2.5 Pro and 2.5 Flash on several measures. Beyond any single vendor, the work is relevant to any organization deploying conversational, negotiation, or customer-service agents that interact strategically with people.

Future Directions

• Exploring techniques to better align LLM strategic behavior with human strategic behavior.
• Testing whether the discovered opponent-modeling and value-based learning mechanisms correspond to actual internal computational processes in LLMs, using mechanistic interpretability methods such as internal state probing or logit analysis.
• Extending AlphaEvolve beyond a single game to learn general behavioral models that describe LLM behavior across a broad set of games or scenarios.
• Addressing the study's limitation that discovered models describe average human behavior: individual differences in IRPS ability are not captured, and expert humans may display more sophisticated opponent modeling than the average participant.

Target Audience

Researchers in behavioral game theory, cognitive science, and AI safety and alignment; LLM evaluation and red-teaming practitioners; computational social scientists who use LLM agents as human simulators; and machine learning researchers interested in program synthesis or interpretable behavioral modeling. Readers need only a working understanding of game theory basics and machine learning evaluation to follow the argument, though the methodology section assumes some familiarity with likelihood-based model fitting.

Authors’ abstract

As Large Language Models (LLMs) are increasingly deployed in social and strategic scenarios, it becomes critical to understand where and why their behavior diverges from that of humans. While behavioral game theory (BGT) provides a framework for analyzing behavior, existing models do not fully capture the idiosyncratic behavior of humans or black-box, non-human agents like LLMs. We employ AlphaEvolve, a cutting-edge program discovery tool, to directly discover interpretable models of human and LLM behavior from data, thereby enabling open-ended discovery of structural factors driving human and LLM behavior. Our analysis on iterated rock-paper-scissors reveals that frontier LLMs can be capable of deeper strategic behavior than humans. These results provide a foundation for understanding structural differences driving differences in human and LLM behavior in strategic interactions.