The Pulse
GameHorizon Brings 5,000 Hours to AI Gameplay Testing
Researchers introduce GameHorizon, a gameplay dataset and benchmark built from 5,000 hours of recordings across 21 AAA games. The suite tests 47 models with offline questions and step-by-step online evaluations designed to expose failures i
AI.info Team ·
Five thousand hours, 21 games, one test suite
GameHorizon introduces a 5,000-hour gameplay corpus designed to test whether AI systems can understand instructions, plan across time and execute actions inside commercial games. The dataset covers 21 games and was collected by 100 human expert players, according to the research paper published on September 21, 2026.
The project combines recorded video, player actions and instructions aligned across multiple time scales. Its authors describe the collection as the first large-scale AAA gameplay dataset to join those three elements with multi-horizon annotations, giving researchers a way to study both immediate decisions and objectives that unfold over much longer sequences.
The work comes from Yiran Wang, Xingyilang Yin, Junfu Pu and 12 additional authors. The paper is cataloged under computer vision and artificial intelligence research and is available through the Robotics Center Research record.
From individual moves to long-horizon goals
GameHorizon divides its system into three parts. GameHorizon-Annotator provides an automated pipeline for generating instructions at different temporal horizons, while GameHorizon-Data stores the resulting gameplay recordings, actions and aligned language.
GameHorizon-Bench supplies the evaluation layer. Its offline track uses thousands of standardized questions arranged around three primary tasks and additional diagnostic variants. The design aims to make repeated comparisons possible without requiring every model to play through the same live game sequence.
The distinction matters because gameplay agents face several separate problems at once. A system may recognize what appears on screen but fail to interpret the instruction, choose an appropriate subgoal or carry out the required action sequence. A short clip can test visual recognition; a long episode tests whether the model preserves the objective while conditions change.
Online testing checks whether offline scores hold up
GameHorizon also includes a stepwise online track that evaluates agents during active gameplay. The paper says the online tests examine whether performance on offline questions reflects actual gameplay capability and identify the particular steps at which a long-horizon task breaks down.
That pairing addresses a familiar weakness in benchmark design. Offline tests are easier to reproduce, but they can separate a model from the feedback and accumulated consequences that define an interactive game. Online tests provide that interaction, while the step-level analysis is intended to show whether an agent failed during perception, instruction following, planning or action control.
The authors do not present the two tracks as interchangeable. Instead, the suite uses offline evaluation for repeatable measurement and online evaluation for checking how those measurements relate to behavior inside an environment.
47 models tested across more than one million calls
The researchers evaluate 47 models using more than one million model invocations. The tested systems span different model families, although the abstract does not provide a complete model-by-model results table on the research record.
The evaluation reveals what the authors call a hierarchy of task difficulty, along with pronounced differences between model capabilities. Those findings support the paper's central argument: gameplay competence is not a single score. Models can perform differently depending on the length of the instruction, the amount of visual interpretation required and the number of actions needed to complete a goal.
GameHorizon therefore treats temporal horizon as a measurement dimension rather than as a side detail. A benchmark that reports only whether an agent answered a single question correctly can miss failures that appear only after several dependent decisions.
A benchmark aimed at reproducibility
Existing gameplay datasets and evaluations often focus on a narrow group of games, omit language instructions or depend heavily on online rollouts with high variance, the paper says. GameHorizon responds by combining a broad game collection with aligned actions, video and language, then separating repeatable offline tests from interactive checks.
The authors say they plan to release the dataset, annotation pipeline and benchmark. The research record does not specify a release date, license or the complete list of games, so those details cannot yet be treated as available documentation.
For now, the concrete contribution is the evaluation design and the scale of the reported collection: 5,000 hours, 21 games, 100 human players and more than one million model invocations. The unresolved questions are how the released data will be accessed, how the benchmark will prevent training contamination and whether offline scores will consistently predict success in the online track.