Observer, Not Player: Simulating Theory of Mind in LLMs through Game Observation

AI summary

Overview

• Research area: Artificial Intelligence — evaluation of large language model (LLM) reasoning, specifically theory-of-mind (ToM)–like inference in a sequential game setting.
• Technical level: Intermediate. The framework itself is conceptually simple (Rock–Paper–Scissors plus a web dashboard), but the evaluation scheme relies on proper scoring rules, distributional alignment, and fixed-point approximation of adaptive strategies.
• Scope: The paper presents an interactive, reproducible demo that places an LLM in the role of an Observer — predicting the probability distribution over strategies played by two Rock–Paper–Scissors agents — and scores those predictions with cross-entropy, Brier score, and expected-value discrepancy combined into a "Union Loss," alongside a Strategy Identification Rate (SIR).

Paper details: arXiv:2512.19210v1 [cs.AI], 22 Dec 2025. Authors Jerry Wang and Ting Yiu Liu, Department of Management Information Systems, National ChengChi University (111306078@nccu.edu.tw, 113356048@nccu.edu.tw). License: CC BY-SA 4.0.

What This Paper Is About

Most game-based evaluations of LLMs measure win rate, which compresses pattern recognition, planning, opponent modeling, and luck into a single number — so a high score does not prove the model understands the underlying outcome distribution against a given opponent. This paper instead makes the LLM an Observer rather than a player: for each Rock–Paper–Scissors matchup, the model must predict the full probability distribution over candidate strategies (and, in practice, over win/draw/loss outcomes) and explain its reasoning. The goal is not to test whether the model knows Rock–Paper–Scissors, but whether it displays mind-like reasoning about sequential behavior — adaptive belief formation, calibration, and strategy identification — with the purpose stated as probing distributional understanding rather than chasing state-of-the-art benchmarks.

Key Contributions

1. An Observer formulation of ToM-style evaluation. Instead of scoring the LLM as an agent that plays and wins, the framework scores it as an outside observer that infers the latent strategies of two interacting players, making "understanding" operational as distributional alignment against the game engine's ground truth.

2. A composite, principled metric suite. Three complementary signals — Cross-Entropy (CE), Brier score, and Expected-Value discrepancy (EVLoss) — are normalized and averaged into a Union Loss, paired with an explicit-commitment metric, the Strategy Identification Rate (SIR), that requires correctly naming both players' strategies.

3. A candidate strategy pool and steady-state solver. A 19-entry library (keys A–P plus reactive policies X, Y, Z) spanning pure static strategies, biased dynamic mixtures, and human-inspired reactive policies ("win-last," "lose-last," "copy-last"), with adaptive strategies resolved through a damped fixed-point iteration (initialized at (1/3, 1/3, 1/3), damping α ∈ (0,1), stop when the L1 change is below 10⁻⁴).

4. An interactive, reproducible demo. A real-time web pipeline (Candidate Pool → RPS engine → Prompt Module → LLM Observer → Evaluation Dashboard) that streams losses as they evolve, allows live prompt ablations and temperature sweeps, and exposes the model's reasoning snippets so users can see where and why failures occur. All other parameters are held fixed for comparability: 200 rounds per match-up, 10 warm-up rounds, a 50-round history limit, and a 20-round reasoning interval.

Main Findings

• Only o3 showed reliable sequential strategy inference. Across round-wise loss curves, o3 had a brief initial transient, a rapid decline, and a stable low plateau; Claude 3.7 Sonnet showed a mid-range but high-variance profile with frequent spikes; GPT-4o-mini remained almost invariant across rounds.

• GPT-4o-mini barely updated its beliefs. Its Brier loss stayed a nearly flat, low-amplitude baseline while normalized cross-entropy stayed at a relatively high constant level, with a pronounced spike only near rounds 140–160 in the N–G matchup — a combination the authors read as a fixed, smoothed prior rather than a model adapting to evidence. Its normalized Union Loss showed an almost flat plateau of roughly 0.20.

• Strategy Identification Rate (SIR) mirrors the loss ranking. Over 200 rounds per match-up: o3 reached 57.5% (Static vs. Dynamic), 41.5% (Dynamic vs. Dynamic), and 0.5% (Human-inspired vs. Dynamic); Claude 3.7 reached 21.5%, 0.0%, and 1.0%; GPT-4o-mini was 0.0% in all three settings.

• Low loss does not imply the model can name the latent policy. In the D vs. Y setting (dynamic vs. psychological), identification dropped to ≈0.5–1.0% even for the stronger models, while all models showed elevated loss — the authors use this to separate predictive fit from explicit strategy attribution.

• Failure modes differ qualitatively. GPT-4o-mini showed description–attribution inconsistency (e.g., labeling a distribution "Paper-biased" while concluding it "matches Rock-biased (H)") and near-invariant reasoning across rounds. Claude 3.7 reported aggregate proportions and mapped them to static templates while often stating "no clear dynamic pattern," collapsing to near-zero SIR in Dynamic vs. Dynamic. o3 quantified round-wise frequencies, cited proximity to a catalog entry, and explicitly ruled out dynamic rules with concrete evidence.

• Convergence pattern in the N–G matchup. o3's Brier loss approached zero after roughly 40–60 rounds, and its normalized cross-entropy decreased monotonically between rounds 50–100 before staying low.

• Calibration and payoff errors moved together for o3. o3 tended to reduce Brier (calibration) and EVLoss (payoff discrepancy) in tandem; Claude 3.7 showed intermittent CE/EVLoss spikes; GPT-4o-mini had small variance but at a higher plateau across all components.

Methodology in Plain English

The researchers built a Rock–Paper–Scissors engine with a fixed library of "strategies" — some pure (always Rock, always Paper, always Scissors), some biased mixtures with fixed probabilities, and three reactive rules (win-last, lose-last, copy-last). For pairs of strategies that both adapt, they approximate the long-run outcome distribution by iterating a damped update until the players' mixed strategies stop changing (L1 change below 10⁻⁴), then compute the win/draw/loss distribution from that steady state.

An LLM is then shown a four-part chain-of-thought prompt containing (1) candidate information, (2) its role specification, (3) the previous game trajectory, and (4) a request to predict the next player action with explicit reasoning. It returns JSON with guesses for both players, a confidence value, and a short reasoning summary. The model sees at most the 50 most recent rounds and must emit a reasoning summary every 20 rounds; 10 rounds of data are preloaded as a warm-up.

Predictions are scored three ways. Cross-entropy measures information-theoretic surprise (with an ε term to prevent numerical underflow, and the caveat that CE equals the true distribution's entropy even for a perfect prediction unless the ground truth is deterministic). The Brier score penalizes mis-ranking and miscalibration. EVLoss squares the difference between the true and predicted expected value, where EV(p) = (p_win − p_loss)/100, so EVLoss lies in [0, 4]. These are normalized — EVLoss divided by 4.0, Brier left as is since it is already in [0, 1], and CE rescaled by grid-wise min–max (with a 0.5 fallback if max equals min) — and averaged into the Union Loss.

Three instruction-tuned models were tested: GPT-4o-mini, o3, and Claude 3.7 Sonnet. The design crosses three models with three match-up regimes — H vs. C (H rock-biased at {rock: 0.5, paper: 0.25, scissors: 0.25} against pure Paper C), N vs. G (N at {rock: 0.167, paper: 0.5, scissors: 0.333} against G at {rock: 0, paper: 0.5, scissors: 0.5}), and D vs. Y (D at {rock: 0.167, paper: 0.5, scissors: 0.333} against the adaptive Y) — yielding 9 total experimental conditions. Models were queried at temperature 0.2; note that the paper states top-p as both 2.0 and 0.7 in adjacent sentences. Results are reported as per-regime means ± standard error, with one-sided permutation tests using 10,000 resamples against the human baseline (the baseline value itself is not reported in the provided content).

Why This Matters

• Impact on research: The paper reframes game-based LLM evaluation away from win rate and toward distributional alignment plus explicit strategy attribution, offering a diagnostic that separates predicting well from being able to name the latent policy. It positions itself relative to existing ToM work — including ToMBench (8 task families, 31 abilities) and benchmarks comparing 11 models to children aged 7–10 — and to the argument that many benchmarks capture literal ToM (predicting others) rather than functional ToM (adapting to new partners).

• Real-world applications (the paper frames these as potential uses of the framework rather than deployed systems):

  • Evaluating agentic assistants that must infer user intent or policy from observed behavior rather than direct instruction.
  • Interpretable model diagnostics, since the dashboard exposes reasoning snippets that localize where belief updating fails or where narratives contradict conclusions.
  • Prompt and hyperparameter ablation studies, including temperature sweeps, using a controlled environment with known ground truth.
  • Theory-of-mind testbeds, providing a reproducible benchmark that other groups can extend to richer settings.

• Industry relevance: Model selection and monitoring. Teams deploying LLMs in interactive or adversarial settings need to know whether a model genuinely recalibrates on new evidence or emits a stable prior with fluent-sounding justification — a distinction this framework makes visible round by round. The lightweight, somewhat code-light dashboard design is explicitly intended to promote community reuse and reproducibility.

Future Directions

1. Scaling beyond Rock–Paper–Scissors. The authors state the current evaluation is limited to simple environments and call for extension to more complex, multi-agent, or strategic games.

2. Deeper diagnostics of reasoning chains. The paper notes it focuses primarily on surface-level reasoning traces and suggests incorporating deeper diagnostics to assess reasoning chains more systematically.

3. Change-point detection for strategy switches. The design explicitly supports extension with methods such as change-point detection, to model opponents whose strategies change mid-match.

4. Probing the gap between low loss and explicit identification. The near-zero SIR in the D vs. Y setting (0.5% for o3, 1.0% for Claude 3.7, 0.0% for GPT-4o-mini) raises the open question of how to build evaluation that rewards correct, explicitly articulated attributions rather than only accurate distributions.

Additional open questions the paper leaves unresolved: whether formal convergence guarantees exist for the damped fixed-point solver, and whether the model family ordering observed here would hold under different candidate pools or prompting strategies.

Target Audience

Researchers and practitioners in LLM evaluation, cognitive-science-inspired AI, and multi-agent systems who want a transparent, low-cost testbed for probing belief updating and distributional reasoning. It is especially useful for those dissatisfied with win-rate-only game benchmarks, for teams building interpretable evaluation dashboards, and for readers tracking the debate over whether LLMs exhibit genuine theory-of-mind-style reasoning or exploit surface cues. Readers looking for large-scale benchmark numbers or a new state-of-the-art model result should note the authors explicitly state that chasing such benchmarks is not the goal.

Authors’ abstract

We present an interactive framework for evaluating whether large language models (LLMs) exhibit genuine "understanding" in a simple yet strategic environment. As a running example, we focus on Rock-Paper-Scissors (RPS), which, despite its apparent simplicity, requires sequential reasoning, adaptation, and strategy recognition. Our system positions the LLM as an Observer whose task is to identify which strategies are being played and to articulate the reasoning behind this judgment. The purpose is not to test knowledge of Rock-Paper-Scissors itself, but to probe whether the model can exhibit mind-like reasoning about sequential behavior. To support systematic evaluation, we provide a benchmark consisting of both static strategies and lightweight dynamic strategies specified by well-prompted rules. We quantify alignment between the Observer's predictions and the ground-truth distributions induced by actual strategy pairs using three complementary signals: Cross-Entropy, Brier score, and Expected Value (EV) discrepancy. These metrics are further integrated into a unified score, the Union Loss, which balances calibration, sensitivity, and payoff alignment. Together with a Strategy Identification Rate (SIR) metric, our framework captures not only predictive accuracy but also whether the model can stably identify the latent strategies in play. The demo emphasizes interactivity, transparency, and reproducibility. Users can adjust LLM distributions in real time, visualize losses as they evolve, and directly inspect reasoning snippets to identify where and why failures occur. In doing so, our system provides a practical and interpretable proxy for mind-like inference in sequential games, offering insights into both the strengths and limitations of current LLM reasoning.