GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

AI summary

Overview

Research area: Computer Vision, specifically multimodal (vision-language-action) model evaluation in video gameplay, sitting at the intersection of game AI, embodied agents, and vision-language model benchmarking.

Technical level: Intermediate. The paper is a data-and-benchmark paper, so the concepts (temporal horizons, instruction pyramids, offline/online evaluation) are accessible, while the annotation pipeline and evaluation protocol details (action-aware segmentation, dynamic programming for clip merging, stepwise environment resets) require some familiarity with agent evaluation.

One-sentence scope: The paper introduces GameHorizon, a three-part suite (an annotation pipeline, a 5,000-hour AAA gameplay dataset, and an offline-plus-online benchmark) for measuring how well 47 different models understand, plan, and act across short, medium, and long gameplay horizons.

What This Paper Is About

Playing a modern video game requires decomposing a long-term objective into subgoals and then turning those subgoals into precise keystrokes and mouse movements, all from the same stream of visual input. Existing gameplay datasets and benchmarks fall short on three fronts: they cover few games (often a single title or simplified mini-games), they rarely include natural-language instructions, and they rely on small numbers of online rollouts that are hard to reproduce and that collapse distinct failure modes into one success-rate number. GameHorizon aims to fix this by providing automated multi-horizon instruction annotation, a large-scale dataset of real human gameplay with aligned videos, actions, and instructions, and a benchmark that combines reproducible offline multiple-choice testing with stepwise online testing that can localize failures to specific steps.

Key Contributions

1. GameHorizon, a multi-horizon data and evaluation suite spanning AAA games and multiple temporal horizons, intended as a single yardstick across model families (general-purpose VLMs, unified multimodal models, coding and GUI agents, and dedicated game agents).
2. GameHorizon-Annotator, a scalable automated annotation pipeline that produces a three-level pyramid of natural-language instructions: short-horizon operations, medium-horizon goals, and long-horizon strategies, abstracting bottom-up from fine-grained clips to higher levels.
3. GameHorizon-Data, described as the first large-scale AAA gameplay dataset aligning video frames, player actions, and multi-horizon instructions. It contains 5,000 hours of recordings from 21 game titles collected by 100 human expert players, 4,571 videos at 60 fps, 411.03 million keyboard-mouse action events, and 6,184,036 distinct instructions (5,947,588 short-horizon operations, 189,158 medium-horizon goals, 47,290 long-horizon strategies).
4. GameHorizon-Bench, a benchmark unifying a reproducible offline track (three primary tasks plus ten variant tasks of standardized multiple-choice questions) with a stepwise online track (order-dependent causal tasks and order-flexible thematic tasks) that uses environment resets to localize failures.

Main Findings

• A hierarchy of task difficulty emerges. Evaluating 47 models through more than one million model invocations reveals "a meaningful hierarchy of task difficulty and pronounced differences in model capabilities." The paper states that planning future actions and decomposing complex goals are more challenging than deciding the current action.
• Multi-horizon instructions improve planning. Compared with vision-only input, incorporating the medium- and long-horizon instructions improves future-action planning by 7.2 percentage points.
• Offline scores track online gameplay. In the online setting, the authors observe a clear positive association between task success rates and offline scores, suggesting offline accuracy is a valid proxy for actual gameplay capability.
• Annotation density far exceeds prior work. GameHorizon-Data provides one distinct short-horizon instruction every 2.63 seconds on average, with every frame aligned to instructions at all three horizons. Average instruction spans are 2.63 seconds (L1), 82.6 seconds (L2), and 330.4 seconds (L3). By contrast, the paper notes Open-P2P includes only one instruction every few minutes with uneven temporal coverage.
• Dataset scale and diversity. Valorant contributes the largest share at 617.5 hours (12.35%), and the smallest entry, Honor of Kings: World, contributes 35.9 hours (0.72%). The games span open-world, action role-playing, competitive shooter, sandbox survival, and creature-collecting adventure genres.
• Human actions, not inferred labels. All actions are captured synchronously from 100 experienced human players rather than recovered via inverse dynamics models, gamepad segmentation, in-game AI, or replay extraction as in prior datasets. Actions average 22.84 events per second, combining mouse input sampled at 20 Hz with an average of 2.84 keyboard events per second.
• Filtering retained most footage for annotation. After filtering low-quality or uninformative content (such as action-free cutscenes and switches away from the game window), 4,341 hours (86.8%) were retained for instruction annotation, while synchronized keyboard-mouse actions remain available for all 5,000 hours.
• Scale comparisons with prior corpora. The paper positions GameHorizon-Data as exceeding D2E (300 hours) and gaming-500-hours (500 hours) in scale. It also notes that NitroGen, GameVerse, D2E, and VPT contain no language instructions at all, and that STEVE-1 contains only 10K short-term instructions.
• Reproducibility gap in prior benchmarks. The paper criticizes prior online evaluations for small sample sizes: Lumine reports success rates from three trials per scene, GameVerse conducts rollouts on 3–20 cases, and VideoGameBench tests each model with a single run per game.
• Detailed per-model or per-task result tables are not reported in the available content. The paper states that 47 models were tested and that the results reveal pronounced capability differences, but the truncated text does not list individual model scores, per-task accuracies, or the number of online task instances.

Methodology in Plain English

The researchers first collected raw gameplay from scratch rather than scraping web videos. They recruited 100 experienced players and built a dedicated recording-and-upload system that captures game video at 2K resolution alongside timestamped keyboard and mouse actions, so the action labels reflect what humans actually did.

To turn these long recordings into something a model can reason about, they built an annotation pipeline that works bottom-up. First, instead of cutting video by visual scene changes (which over-segments continuous actions during camera motion), they segment using the keyboard-mouse traces: discrete events like jumping or attacking define boundaries, and consecutive inputs within the same sustained event are grouped. Because the same input can mean different things (a left click might be an attack or an item selection), a vision-language model inspects frames to disambiguate and refine the boundaries, producing short-horizon clips of 1–5 seconds, each labeled with a short-horizon operation.

Next, a vision-language model decides which neighboring short-horizon clips belong to the same unfolding goal, merging them into medium-horizon clips of 1–2 minutes labeled with medium-horizon goals. Repeating the process on those yields long-horizon clips of 5–8 minutes labeled with long-horizon strategies. A dynamic programming procedure enforces these duration ranges. The prompts for each level differ in input modality and output granularity: short-horizon annotation sees frames plus aligned actions (to ground instructions in real controls), medium-horizon annotation sees frames, actions, and the constituent short-horizon instructions, and long-horizon annotation sees frames and the constituent medium-horizon instructions but deliberately excludes actions to keep the output focused on high-level intent. Prompts also include game title, clip duration, frame rate, sampled frame indices, and game-specific keybinds.

The benchmark then reuses this data in two ways. The offline track converts aligned frames, instructions, and actions into standardized multiple-choice questions in which the correct answer is built by mapping ground-truth keyboard-mouse controls to textual action sequences via game-specific keybinds, removing dependence on game environments or agent harnesses. The online track breaks long-horizon objectives into verifiable short-horizon subtasks, where a failed subtask triggers a reset to the corresponding success state so evaluation can continue, and a long-horizon task counts as passed only if all its subtasks succeed.

Why This Matters

Impact on research. Gameplay is one of the few testbeds where visual understanding, instruction following, goal decomposition, and precise action control must all work together. This work argues that the field's current yardsticks are either too narrow (single games, mini-games), too sparse in language annotations, or too noisy (few rollouts, harness-dependent) to support confident comparisons. By releasing an annotator, a large aligned dataset, and a benchmark with both reproducible offline and diagnostic online evaluation, the paper offers a common measuring stick that can be applied across very different model families rather than only to purpose-built game agents.

Real-world applications (derived from the paper's framing):

• Training and evaluating general-purpose vision-language-action agents that must convert natural-language goals into executable control sequences.
• Comparing GUI agents, coding agents, and computer-use agents on a shared action-execution task, since the benchmark standardizes actions into text rather than raw control spaces.
• Diagnosing where an agent fails in a multi-step task, since the online track resets to a success state and localizes errors to specific steps rather than reporting one aggregate number.
• Providing supervision data for models that need to learn temporal goal decomposition, since each frame is aligned to instructions at all three horizons.

Industry relevance. The work originates from ARC Lab, Tencent, with collaborators at Great Bay University, University of Macau, the National University of Singapore, Huazhong University of Science and Technology, and MMLab, CUHK. Games are a commercial-scale testbed, and the dataset's focus on AAA titles, direct human keyboard-mouse actions, and dense per-frame instructions makes it directly applicable to building game-playing agents and to evaluating whether general-purpose models can operate complex interactive software. The paper states that the dataset, annotator, and benchmark will be released, with a project page and GitHub repository listed.

Future Directions

• Scaling the annotation pipeline further. The annotator is described as scalable and cost-reducing relative to manual labeling, which invites extending coverage to more titles, genres, or control schemes beyond the 21 titles and keyboard-mouse setups used here.
• Closing the gap between understanding and acting. The paper reports that planning future actions and decomposing goals are harder than deciding the current action, and that multi-horizon instructions improve future-action planning by 7.2 percentage points. Whether that improvement can be converted into higher action-execution success in live play remains open.
• Broadening the online track. The available content describes the online track only up to its introduction (causal and thematic tasks with environment resets); details of its scale, task inventory, and protocol beyond that point are not reported in the text provided, leaving room for fuller specification and expansion.
• Investigating failure modes beyond aggregate scores. Because the benchmark separates current-action perception from future-action planning (via the T1* variants) and top-down decomposition from bottom-up abstraction (via T2*), and separates goal planning from action decoding (via T3*), a natural next step is using these diagnostics to identify which specific model capability is the bottleneck rather than only ranking models.

Target Audience

This paper is most useful to researchers and engineers building or evaluating agents that must act in visually rich, temporally extended environments: game AI researchers, vision-language-action and embodied-agent developers, and teams working on general-purpose computer-use or GUI agents that need a standardized action-execution test. Benchmark and dataset creators will also benefit from the annotation pipeline design, the offline/online evaluation split, and the reproducibility arguments against small-rollout online evaluation. Readers looking for a detailed leaderboard of model scores will not find one in the available content; the paper's value here lies in the suite's construction and the high-level findings about task difficulty and instruction benefits.

Authors’ abstract

Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, and precise action control over multiple temporal horizons. Existing datasets and benchmarks, however, either cover a narrow range of games, lack language instructions, or rely on high-variance online rollouts. To address these challenges, we introduce GameHorizon, a unified data and evaluation suite that measures gameplay capabilities at different horizons for diverse model families. GameHorizon Suite consists of three components. First, GameHorizon-Annotator is a scalable and automated annotation pipeline for multi-horizon instructions. Second, utilizing the pipeline, we construct GameHorizon-Data, the first large-scale AAA gameplay dataset with temporally aligned videos, player actions, and multi-horizon instructions. It comprises 5,000 hours of recordings from 21 games, collected by 100 human expert players. Third, we build GameHorizon-Bench with reproducible offline and stepwise online testing. The offline track enables reproducible evaluation using thousands of standardized questions organized into three primary tasks and a series of diagnostic variants, while the online track tests whether offline scores reflect actual gameplay capabilities and localizes failures to specific steps within long-horizon gameplay. Based on our GameHorizon Suite, we evaluate 47 models through more than one million model invocations, revealing a meaningful hierarchy of task difficulty and pronounced differences in model capabilities. Our work can provide a standardized yardstick for evaluating gameplay capabilities across horizons and model families. We will release our dataset, annotator, and benchmark to facilitate future research.