Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?

AI summary

Overview

Research area: Evaluation of vision-language models (VLMs), specifically visual reasoning benchmarks and whether procedurally generated "toy game" tasks measure the same abilities as real-world visual reasoning.

Technical level: Intermediate — readable with basic familiarity with VLMs and benchmark evaluation, though the appendix contains detailed task specifications.

Scope: The paper introduces Game2Real-Bench (1,400 evaluation samples) and Game2Real-Dataset (23,893 verified samples), linking 18 procedurally generated toy-game tasks to 400 curated real-world samples across four capability groups, and evaluates 25 VLMs on both.

What This Paper Is About

Benchmarks for visual reasoning come in two flavors: real-world data, which preserves natural complexity but is hard to generate and verify at scale, and procedurally generated games, which are scalable and automatically checkable but visually simplified. It has been unclear whether performance on toy games actually reflects, or can transfer to, the reasoning required in real-world tasks. The paper builds a benchmark that pairs the two settings by shared reasoning requirements rather than visual similarity, and tests both cross-setting correlation and one-shot game-to-real transfer.

Key Contributions

1. Game2Real-Bench, a 1,400-sample benchmark (1,000 toy-game samples plus 400 real-world samples) aligned across four capability groups — fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning — enabling analysis of cross-setting performance correspondence and game-to-real transfer.
2. Game2Real-Dataset, nearly 24K programmatically verified samples (23,893) across 18 toy-game tasks and 65 QA families, together with executable generators and task-specific verifiers, as a resource for evaluation and training.
3. A 25-model evaluation (22 open-weight models from 2B to 35B parameters, plus three proprietary systems) showing strong model-level game–real correlation, and a one-shot in-context learning study on four models showing that transfer is selective by capability and by model.
4. Diagnostic analyses of thinking versus non-thinking mode and of three recurring failure patterns: misinterpreting spatial structure, inconsistent relational reasoning, and unreliable state updating.

Main Findings

• Strong cross-setting correlation: Pairing each of the 25 models' overall score on the 1,000 game samples with its score on the 400 real-world samples yields a Pearson correlation of r = 0.935 and a Spearman correlation of ρ = 0.935, indicating that relative model rankings are largely preserved across the two settings.
• Toy games are harder and more discriminative: Most models score lower on the toy-game set than on the real-world set, with the gap most visible among open-weight models. The best proprietary and best open-weight models differ by 16.3, 39.0, and 20.4 points on fine-grained perception, constraint reasoning, and sequential planning respectively on games, while those gaps are much smaller in the real-world setting.
• Headroom remains large: GPT-5.6-Sol reaches the highest overall score at 69.8%, followed by Gemini-3.1-Pro-Preview at 59.6% and Claude-Opus-4.8 at 53.7%. The best open-weight model, Qwen3.8-27B, reaches 52.6%, and 16 of the 22 open-weight models score below 30%. Real-world scores incorporate SpatiaLQA F1 where applicable; toy-game scores are accuracies.
• Sequential planning is the most consistent strength: It is the highest-scoring capability for 15 of 25 models on the toy-game set and 17 of 25 on the real-world set.
• Capability difficulty shifts between settings: Fine-grained perception has the lowest average on the toy-game set but becomes one of the stronger capabilities on real-world tasks, while constraint reasoning stays difficult for stronger models and preserves a visible proprietary-versus-open-weight gap.
• One-shot transfer is selective: Using a pool of 168 verified samples covering 14 game tasks, randomly assigned as one demonstration per real-world target (three random seeds), constraint reasoning improves by 4.89 points on average — Qwen3.5-9B rises from 24.32% to 39.94%.
• Fine-grained perception consistently degrades: It drops by 4.92 points on average and declines for all four models tested, which the authors attribute to these tasks depending on target-specific visual evidence that an unrelated demonstration cannot supply.
• Overall transfer effect is mixed: Qwen3.5-9B is the only model with an overall gain (1.49 points), Qwen3.8-27B changes by −0.33 points, and the 2B and 4B models decrease overall; average overall accuracy falls by 1.34 points.
• Thinking mode helps all four tested models: Qwen3.5-2B, Step3-VL-10B, and Gemma4-31B-it gain 6.2, 6.0, and 11.1 percentage points respectively, while GPT-5.6-Sol gains 41.5 points, rising from 28.3% to 69.8% on the full 1,400-sample benchmark.
• Failures cluster into three patterns: Block Counting errors trace to incorrect 3D structure inferred from a 2D projection; Hexakai errors show constraint-line memberships being altered or candidates eliminated without verification; Mancala errors show a missed capture update mid-trajectory.
• Not reported: The paper does not report a human baseline or human upper-bound performance on Game2Real-Bench.

Methodology in Plain English

The benchmark is built in three stages.

First, the authors define a taxonomy of four reasoning capabilities and design 18 toy-game tasks around them, each with one or more QA families (65 in total) that specify what is asked and in what output format, using open-ended and multiple-choice formats.

Second, each task gets an executable generator that samples valid problem states and simultaneously produces the images, question, and ground-truth answer, plus a deterministic solver or verifier that recomputes the answer using constraint solving, geometric transformation, state simulation, graph search, or transition replay. Generation follows three principles: the answer must be determined by the visual content rather than prompt wording, instances must vary in structure and layout while keeping the underlying rules, and every answer must be checkable. The 1,000-sample evaluation set is generated with disjoint random seeds and distinct layouts from the 23,893-sample training-oriented dataset, with semantic fingerprints and image hashes used to remove overlap.

Third, 400 real-world samples are curated from existing benchmarks and public datasets — including real images, videos, and multi-view observations — chosen because their reasoning requirements match those of the toy games, not because they look similar. Where a source benchmark already meets the target requirements, its original questions, answers, and evaluation protocol are kept; otherwise new questions are built from the source media, as with aligning IKEA assembly instructions to real assembly videos. All samples pass automated consistency checks and manual review.

Evaluation is zero-shot with identical instructions and inputs: open-weight models run through VLMEvalKit with official checkpoints, proprietary models through official APIs, with no external tools or task-specific solvers. Transfer is tested by prepending one solved game example — image, question, and final answer only — to each real-world target, with three random seeds and identical demonstration-target pairs across models.

Why This Matters

For research, the paper supplies a shared measurement frame in which game and real-world tasks are matched by reasoning requirement rather than appearance, turning an often-assumed link into a measurable quantity. The high correlation supports using cheap, verifiable procedural tasks as proxies for discriminating between models, while the selective transfer results caution against assuming game data is universally useful as training signal.

Real-world applications implied by the task design:

• Assembly and repair assistance: interpreting step-by-step instructions and the visual states before and after an action, the motivating example the authors cite.
• Spatial and 3D understanding: recovering hidden structure and column arrangements from 2D views, relevant to AR, robotics, and inspection.
• Constrained route and action planning: shortest-path routing with keys, doors, and locked states, and sliding-block rearrangement, mirroring logistics and navigation problems.
• Layout and packing problems: placing items under inventory, count, and contact constraints, as in the Battleship and Block Island tasks.

For industry, the benchmark gives a controllable way to compare models on precisely specified reasoning, and Game2Real-Dataset offers nearly 24K verified samples with generators and verifiers — an asset for supervised fine-tuning or reinforcement learning where reward signals must be automatic.

Future Directions

• Broaden task coverage: the current benchmark has 18 toy-game tasks and 400 real-world samples; extending to more task types and richer visual settings while keeping the game–real alignment principle is an explicit next step.
• Move beyond one-shot prompting: use the nearly 24K verified samples for multi-shot prompting, supervised fine-tuning, and reinforcement learning to test whether transfer improves when models are trained rather than merely prompted.
• Test behavioral, not just score-level, correspondence: the 0.935 correlation is a model-level association and does not establish that both settings rely on the same internal reasoning processes.
• Make demonstration selection capability-aware: since constraint reasoning gains and fine-grained perception declines under the same random demonstrations, choosing demonstrations by target capability is an open design question.

Target Audience

Researchers building and evaluating VLMs and multimodal reasoning benchmarks; practitioners deciding whether to train or prompt with synthetic procedural data; and anyone interested in how well controlled, automatically verifiable tasks predict performance on messy real-world visual problems. Readers without a benchmark-evaluation background can follow the main text, while the appendix (task definitions, QA families, worked examples) serves those who want to reuse the generators or verifiers.

Authors’ abstract

Visual reasoning is an important capability for vision-language models (VLMs) to understand visual evidence, spatial structure, constraints, and state changes. Existing evaluations mainly rely on real-world data or procedurally generated tasks. Real-world data preserve natural complexity, while procedural games provide controllable generation and verifiable solutions. However, it remains unclear whether abilities measured in toy games correspond to real-world visual reasoning and whether game examples can support transfer. We introduce Game2Real-Bench, a benchmark that studies these connections across four capability groups, including fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning. We construct 1,000 procedurally generated game samples with executable verification and curate 400 real-world samples aligned by shared reasoning requirements rather than visual similarity. We evaluate 25 representative VLMs and observe strong game-to-real consistency, with Pearson and Spearman correlations of 0.935. We further study one-shot game-to-real transfer on four models. Constraint reasoning improves for several models, while fine-grained perception declines with game demonstrations. We also construct Game2Real-Dataset with nearly 24K verified game samples across 18 tasks and 65 QA families. Our benchmark, dataset, and code are available at https://github.com/QingAnLiu/Game2Real-Bench.