MeepleLM: A Virtual Playtester Simulating Diverse Subjective Experiences

AI summary

Overview

• Research area: Human-Computer Interaction, specifically LLM-based user simulation and AI-assisted game design; the paper sits at the intersection of board game evaluation, persona modeling, and human-AI co-creation.
• Technical level: Advanced. The paper assumes familiarity with LLM fine-tuning (LoRA, instruction tuning), chain-of-thought reasoning, embedding-based clustering, and distributional evaluation metrics (Wasserstein distance, Kendall's tau).
• Scope in one sentence: The paper trains MeepleLM, a specialized model that reads a board game rulebook and a player persona and predicts the persona-specific rating and critique that player would give, without running the game.

What This Paper Is About

LLMs have been used to play, design, and prototype board games, but existing systems cannot critique a design based on how it actually feels to different kinds of players. Because board game experience is emergent (it arises only when mechanics interact at runtime) and subjective (the same mechanic delights one player and frustrates another), automated critique requires both inferring hidden gameplay dynamics from static text and modeling distinct player groups rather than a single average user. MeepleLM targets both problems by combining Mechanics-Dynamics-Aesthetics reasoning with data-driven personas, producing reviews that match community sentiment better than frontier commercial models.

Key Contributions

1. The paper presents what it describes as the first systematic study on automated evaluation of board games, bridging static rules and distinct player experiences by simulating latent gameplay dynamics.
2. It curates a dataset of 1,727 rulebooks and 150K critiques, filtered by quality scoring and facet-aware sampling, and augments it with MDA-based chain-of-thought paths that make the rules-to-experience link explicit.
3. It distills five data-driven player personas and introduces MeepleLM, which internalizes persona-specific reasoning to predict gameplay experiences reflecting real community preferences.
4. It conducts a systematic evaluation on a stratified set of 207 games across macro-level alignment, micro-level fidelity, and practical utility, reporting that MeepleLM outperforms state-of-the-art LLMs as a virtual playtester.

Main Findings

• Rating prediction: MeepleLM achieves the best alignment scores of all compared models: MAE 0.6576, Wasserstein Distance 0.2205, and Kendall's tau 0.2817, versus GPT-5.1 (MAE 0.9874, WD 0.9496, tau 0.2555), Gemini3-Pro (MAE 1.4277, WD 0.5092, tau 0.2465), Qwen3-235B (MAE 1.2288, WD 0.6350, tau 0.1449), and the untuned Qwen3-8B backbone (MAE 0.8906, WD 1.0119, tau 0.0492).
• Central tendency bias in baselines: The paper reports that strong baselines such as GPT-5.1 "play it safe" by clustering predictions near the mean, which is quantified by their high Wasserstein Distance (0.9496 for GPT-5.1 versus 0.2205 for MeepleLM). The untuned Qwen3-8B shows tau approximately 0, which the authors describe as effectively random guessing.
• Review quality and diversity: MeepleLM reaches Factual Correctness 98.86, Distinct-2 0.7117, and Perspective Diversity 4.34, compared with GPT-5.1 (99.46, 0.6934, 4.26), Gemini3-Pro (98.28, 0.6480, 3.98), Qwen3-235B (98.95, 0.6572, 3.56), and Qwen3-8B (97.88, 0.5936, 1.58).
• Practical utility: MeepleLM attains an Opinion Recovery Rate of 69.77, higher than GPT-5.1 (63.44), Gemini3-Pro (57.74), Qwen3-235B (54.27), and Qwen3-8B (11.39).
• Ablations: Removing the rulebook collapses factual accuracy from 98.9 to 59.9; removing the persona drops ranking alignment (tau to 0.13); removing MDA lowers opinion recovery (55.35 for w/o MDA versus 69.77 for the full model) and reduces Perspective Diversity.
• User study: In a blind A/B test with N = 10 participants evaluating 6 titles each (3 familiar, 3 unfamiliar), MeepleLM achieved a 78.3% average win rate on familiar games and 74.2% on unfamiliar games against GPT-5.1. In the familiar setting, 83.3% of participants cited superior "authenticity"; in the unfamiliar setting, 86.7% preferred MeepleLM for critical honesty. Binomial tests report p < 0.001 with Cohen's h between 0.69 and 1.03. The abstract summarizes this as a 70% preference rate in user studies assessing utility.
• Persona segregation: The paper reports rating gaps between persona groups on specific games, for example <Unspeakable Words> (BGG difficulty 4.04) where Social Lubricator averaged 6.9 and System Purist 2.9, and <Aeon Trespass> (difficulty 2.97) where Narrative Architect averaged 9.0 and Efficiency Essentialist 6.0.
• Filtering effect: The stratified coverage-maximization strategy preserved the original rating distribution with Pearson's r = 0.920 while raising MDA scores and semantic coverage, retaining approximately 150k entries (about 8% of 1.8 million raw rating-comment pairs), or 50–100 reviews per game.
• MDA chain reliability: A post-hoc human audit had 3 experienced players verify 200 MDA chains spanning 10 familiar games; all 200 passed under the same criteria used by GPT-5.1.
• Dataset composition notes: The paper reports BGG Weight coverage 1.0–5.0, mean complexity 2.57, median publication year 2013, 192 unique mechanics, 81 themes, and an average of 6.35 mechanics per game. Section 3.1 states the dataset is 1,727 games with 53% ranked above 1,000 and 35 titles from 2024 and beyond, while Appendix A states 1,717 games, N = 909 long-tail titles (over 50%), and 34 titles from 2024–2025; these figures are inconsistent within the paper as provided.

Methodology in Plain English

Data. The authors sampled games from BoardGameGeek across market tier (elite and long-tail), complexity, publication era, and mechanics. Rulebooks were turned into structured Markdown: PDFs were parsed with Mineru, restructured by Qwen3-235B into standard sections such as Objective, Components, and Flow, then cross-checked and corrected against the source text by GPT-5.1. Reviews were screened by Qwen3-235B on three dimensions: hard filtering of noise, MDA scoring of whether mechanics, dynamics, and aesthetics were linked, and topic/facet identification (for example Rule Clarity, Balance & Fairness). Selection then maximized facet coverage while preserving the original rating distribution.

Personas. Reviews were embedded with Qwen3-Embedding-8B (concatenating text with logic scores and facets) and clustered with K-Means at K = 15. GPT-5.1 profiled representative samples, and domain experts refined these into five final personas. GPT-5.1 then labeled the whole dataset, with a majority vote over 3 independent inferences per review. The authors report that a supervised DeBERTa-v3-large classifier was insufficient because it misread subtle signals, such as confusing a reviewer who adds house rules for volatility with a System Purist rather than a Thrill Seeker.

Reasoning chain. Since the causal chain is latent in raw reviews, Qwen3-235B acted as a teacher model to reconstruct it in three forced steps: Mechanics ("The What"), Dynamics ("The How"), and Aesthetics ("The Feel," modulated by the persona). GPT-5.1 then acted as a verifier, checking entailment between the synthesized chain and the ground-truth rating and triggering regeneration when sentiment contradicted or hallucinations appeared.

Training. The model fine-tunes a Qwen3-8B backbone on the joint sequence of the MDA chain plus the final critique, minimizing standard cross-entropy loss. The persona is not a bare label; its full semantic profile (core values and interaction preferences) is encoded in the system instruction, so it modulates the Dynamics-to-Aesthetics transition. Training used LoRA on all linear layers via LLaMA-Factory.

Evaluation. A test set of 207 games disjoint from training was stratified by BGG Weight and average rating tier, and included 34 newly released titles from 2024–2025. For each game, N = 100 simulation runs were performed, sampling personas to match the empirical proportions in the ground-truth reviews. Baselines were GPT-5.1, Gemini3-Pro, Qwen3-235B, and the Qwen3-8B backbone. A Gemini3-Flash judge handled fact-checking against the rulebook, perspective-diversity scoring of batches of k = 5 reviews on a 1–5 scale, and ground-truth opinion mining for the Opinion Recovery Rate.

Why This Matters

Research impact. The paper argues that automated design tools for interactive systems have focused on generating content or executable engines, not on predicting the experience those designs produce. It reframes evaluation as predicting emergent, audience-specific experience directly from written specifications, and it positions personas grounded in large-scale empirical critique as a way to avoid the hallucination and stereotyping risks of purely synthetic user simulation. The authors note that although instantiated on board games, the framework of inferring latent dynamics from written designs and modeling subjective heterogeneity across user groups addresses a general class of interactive systems.

Real-world applications.

• Rapid pre-release playtesting, replacing or reducing a resource-intensive human play loop with automated, persona-diverse critique.
• Personalized game recommendation, by simulating how a specific archetype of player would rate and review a title.
• Design iteration support, surfacing potential flaws and polarized feedback before publication.
• Community-facing sentiment forecasting, estimating how opinion is likely to split rather than predicting only an average score.

Industry relevance. For publishers and designers, the value proposition is fast, cheap, experience-grounded feedback that the paper claims reads "less like marketing" and more like honest criticism. For LLM developers, the result is a case study in fine-tuning a small open backbone (Qwen3-8B) to beat much larger commercial models on a domain-specific subjective task, which the reported MAE (0.6576 vs 0.9874 for GPT-5.1), Wasserstein Distance (0.2205 vs 0.9496), and Opinion Recovery Rate (69.77 vs 63.44) illustrate.

Future Directions

1. Multimodal rule understanding. MeepleLM currently reads rules as text only. The authors plan to add visual encoders to process game assets such as cards, maps, and tokens, since card art, board iconography, and component design affect immersion and usability.
2. Finer persona granularity. The current model uses five aggregated personas derived from community clusters, which may miss individual idiosyncrasies. The stated goal is individual-level modeling built on detailed historical data from specific players, forming a "virtual player community" of agents that reproduce real individuals' tastes.
3. Generalization beyond board games. The paper claims the reasoning framework applies to a general class of interactive systems, but does not report validation on other domains; whether the persona and MDA pipeline transfers to video games, tabletop role-playing, or software interfaces remains an open question.
4. Scaling of human validation. The user study uses N = 10 participants and the MDA audit covers 200 chains across 10 games. Larger and more demographically varied validation of subjective fidelity is an open direction.

Target Audience

This paper is most useful for researchers and practitioners in AI-assisted game design and human-AI co-creation, for LLM researchers interested in persona-conditioned generation and domain-specific fine-tuning, for board game designers and publishers who want automated, audience-aware critique, and for anyone studying user simulation who needs an example of grounding synthetic personas in large-scale empirical community data.

Authors’ abstract

Recent advancements have expanded the role of Large Language Models in board games from playing agents to creative co-designers. However, a critical gap remains: current systems lack the capacity to offer constructive critique grounded in the emergent user experience. Bridging this gap is fundamental for harmonizing Human-AI collaboration, as it empowers designers to refine their creations via external perspectives while steering models away from biased or unpredictable outcomes. Automating critique for board games presents two challenges: inferring the latent dynamics connecting rules to gameplay without an explicit engine, and modeling the subjective heterogeneity of diverse player groups. To address these, we curate a dataset of 1,727 structurally corrected rulebooks and 150K reviews selected via quality scoring and facet-aware sampling. We augment this data with Mechanics-Dynamics-Aesthetics (MDA) reasoning to explicitly bridge the causal gap between written rules and player experience. We further distill player personas and introduce MeepleLM, a specialized model that internalizes persona-specific reasoning patterns to accurately simulate the subjective feedback of diverse player archetypes. Experiments demonstrate that MeepleLM significantly outperforms latest commercial models (e.g., GPT-5.1, Gemini3-Pro) in community alignment and critique quality, achieving a 70% preference rate in user studies assessing utility. MeepleLM serves as a reliable virtual playtester for general interactive systems, marking a pivotal step towards audience-aligned, experience-aware Human-AI collaboration.