GameVerse: A Minute-Scale Gameplay Dataset for Long-Horizon Interactive World Modeling

AI summary

Overview

Research area: Computer vision — datasets and generative video models for interactive world modeling (action- and camera-controllable video generation), at the intersection of game AI, video generation, and 3D/geometric consistency.

Technical level: Intermediate. Readers should be comfortable with video diffusion models, camera pose estimation, and standard generation benchmarks (VBench, WorldScore), but the paper's core argument—that long-horizon generation is bottlenecked by data, not model capacity—is stated plainly.

Scope: The paper introduces GameVerse, a 69.7K-clip, 1,655-hour gameplay video dataset with validated camera trajectories and four-level structured captions, built specifically to train interactive world models that stay coherent beyond roughly one minute.

What This Paper Is About

Interactive world models can generate controllable, dynamic environments from action inputs, but their rollouts break down after roughly one minute—drifting, losing object identity, and becoming scene-incoherent. The authors argue this ceiling is a data problem: existing gameplay datasets consist of clips under ten seconds with whole-clip captions that blur static scene context together with dynamic events, so nothing in the training signal supervises minute-scale coherence. GameVerse is their answer: a publicly released, minute-scale gameplay corpus where every clip carries a temporally stable camera trajectory and a hierarchical, event-aligned caption.

Key Contributions

1. The largest minute-scale gameplay dataset to date. 69.7K clips, 1,655 hours, and 349.6M frames, including 1,358 hours of continuous two-minute sequences—described as roughly an order of magnitude longer than existing gameplay corpora, and the first to directly supervise the minute-scale rollout regime where current models fail.

2. A unified construction pipeline with validated camera trajectories. Scene-consistent segmentation combining TransNetV2 hard-cut detection with CLIP-based gradual-transition detection, multi-stage visual/content/temporal quality filtering, and ViPE-based pose estimation with explicit jump-, jitter-, and drift-filtering.

3. A multi-granular, event-aligned caption schema. A four-level annotation—global static, local static, dynamic events, agent actions—that disentangles environment context from controllable behavior, replacing the single-line whole-clip captions dominant in prior gameplay data.

4. Empirical validation on long-horizon generation. Fine-tuning on GameVerse improves VBench temporal-consistency and visual-quality metrics and most WorldScore controllability metrics over gameplay-domain baselines on a 60-second generation benchmark.

Main Findings

• Scale and duration: GameVerse contains 69.7K high-quality clips totalling 1,655 hours. Over 40.6K clips reach the 120-second upper bound and together contribute approximately 1,358 hours of continuous gameplay video. The source pool was 11,002 videos, 4,692 hours (195.5 days), and approximately 986.6M frames at up to 1080p and 30–60 FPS; segmentation yielded 272,128 clips totalling 4,661 hours, of which 120-second clips contribute approximately 2,773 hours.

• Game and viewpoint diversity: At least 110 distinct game titles. The most-represented titles—God of War, Red Dead Redemption 2, Ghost of Tsushima, Elden Ring, and The Witcher 3—each contribute over 1,800 clips. Approximately 24% of clips are first-person, 73% third-person follow camera, and 3% use specialised perspectives (drone, vehicle-mounted, gimbal-follow, stationary). Eight behavior categories are covered: exploring, combat, walking, running, driving, interacting, climbing, and idle.

• Camera motion diversity: Forward motion is the largest share at 34.7%, and complex motions coupling translation with rotation contribute a further 22.0%. The corpus spans static, rotational, translational, and complex motion patterns.

• Caption statistics: Local static and dynamic events captions are the longest, with median lengths of 895 and 775 characters respectively; global static and agent action captions are deliberately more concise, with agent actions drawn from a constrained vocabulary. Word clouds show the four levels occupy largely disjoint vocabularies.

• VBench gains from same-backbone fine-tuning: Under the identical HY-World 1.5 backbone, fine-tuning on GameVerse raises all six VBench metrics: Subject Consistency 66.92 → 68.45, Background Consistency 81.77 → 83.30, Motion Smoothness 96.26 → 96.81, Aesthetic Quality 4.29 → 4.65, Imaging Quality 48.99 → 49.34, and Overall Consistency 14.48 → 15.69.

• Aesthetic/alignment leader is not the fine-tuned model: LingBot achieves the highest Aesthetic Quality (6.14), Imaging Quality (56.70), and Overall Consistency (18.25), while LTX-FT leads Subject Consistency (79.17), Background Consistency (90.63), and Motion Smoothness (99.53).

• WorldScore gains: Fine-tuning HY-World 1.5 improves all seven WorldScore components: WS-Static 53.33 → 55.13, 3D Consistency 66.94 → 73.42 (+6.48), and Camera Control 28.58 → 28.90. LTX-FT achieves the highest WS-Static (59.65), 3D Consistency (95.13), Photometric Consistency (98.45), and Style Consistency (90.04); LingBot leads Camera Control (31.69), Content Alignment (50.18), and Subjective Quality (23.05), and HY-World-FT leads Object Control (46.26).

• Reduced visual drift over long rollouts: Using CLIP ViT-B/32 reference-frame similarity sampled at 2 Hz, LTX-FT reaches 0.785 at 30 seconds with the latest reported threshold crossing at 50.7s; LingBot crosses at 49.0s with 0.680 at 30s. Same-backbone fine-tuning raises the 30-second similarity from 0.609 to 0.642 and delays the first drop below 0.6 from 30.4s to 37.9s.

• Activity-wise consistency: LTX-FT is highest across all four activities (driving 81.44, riding 79.13, running 76.16, walking 80.74). HY-World-FT beats HY-World 1.5 by 1.15, 1.85, 2.49, and 1.58 points respectively.

• Trajectory validation: Compared with an independent DROID-SLAM estimator after Sim(3) alignment, rotation RPE ranges from 0.75 to 1.47 degrees and translation-direction error from 4.84 to 11.03 degrees across six motion categories. The authors explicitly frame this as cross-estimator consistency, not absolute accuracy against ground truth.

• Caption quality audit: On 400 clips with three independent annotators, scene/event accuracy scored 4.82/5, agent-action accuracy 4.66/5, temporal alignment 4.61/5, camera-motion accuracy 4.55/5, and overall alignment 4.64/5, with Krippendorff's alpha = 0.76. Clip-level hallucination was 5.5% and temporal mismatch 4.0%.

Methodology in Plain English

The authors start from YouTube rather than proprietary engine captures, because it is the only open source offering long-form, multi-title gameplay with both first- and third-person views. They collect per-title videos from open-world AAA and indie games.

Segmentation. Abrupt cuts are found with TransNetV2, which scores frame-level transition probabilities. Gradual transitions—camera pans, UI state changes—are caught by measuring CLIP semantic distance between sampled frames and treating large jumps as boundaries. This produces scene-consistent segments of up to 120 seconds.

Quality filtering. A five-stage funnel removes: low-resolution or corrupted video; subtitles, HUD elements, and overlays detected via OCR; unstable motion and temporal artifacts; near-duplicate content (while preserving diversity); and clips that fail temporal-quality metrics such as COVER and VBench consistency. Each clip's score is a weighted sum of per-stage scores, with hard constraints so a strong score in one dimension cannot compensate for a severe defect elsewhere. A subset of clips where the camera is stationary but the scene still evolves is recovered, since fixed-viewpoint object and agent dynamics are still useful training signal.

Camera poses. ViPE estimates frame-wise intrinsics and extrinsics. Because pose estimation on dynamic scenes is fragile, every trajectory is checked against three failure modes—sudden jumps, high-frequency jitter, and long-term drift—using a unified trajectory-quality function that combines local relative-pose increments with a global smoothness term, then threshold-filtered.

Captions. The vision-language model Kimi-K2.5 generates four annotation levels per clip: global static (terrain, weather, lighting, viewpoint), local static (functional areas, landmarks, visible objects per segment), dynamic events (crowd movement, traffic, environmental transitions), and agent actions (the main subject's behavior from a unified action vocabulary). Captions align to event boundaries rather than fixed time windows.

Experiments. They evaluate 60-second image-to-video generation on the same 103 inputs with four systems: LingBot, HY-World 1.5, HY-World-FT (HY-World 1.5 fine-tuned on approximately 600 hours of GameVerse for 3,000 steps, step-3,000 checkpoint, architecture unchanged), and LTX-FT (a GameVerse-adapted LTX-Video 2.3, 22B parameters, trained on 24,380 segments of 12–15 s at 540p/24 fps with a 60% i2v / 20% v2v / 20% t2v objective for 3,700 steps across 64 GPUs).

Why This Matters

The paper reframes a widely observed failure mode—interactive world models collapsing after about a minute—as a dataset gap rather than an architecture gap, and then supplies the missing substrate. Crucially, it releases the data openly, in contrast to competitors whose gameplay corpora, Video-Pose-Action-Prompt quadruplets, and engine-level poses remain proprietary. That changes what the rest of the field can ablate and reproduce.

Real-world applications:

• Game development and prototyping: Controllable, camera-aware generation of playable environment footage for pre-visualization and content pipeline acceleration.
• Embodied AI and simulation: Long-horizon, first- and third-person video with validated camera motion as a training ground for decision-making and navigation policies.
• Robotics and autonomous navigation: Camera-trajectory supervision for geometry-aware generation under genuine viewpoint change, relevant to sim-to-real transfer research.
• Interactive media and virtual production: Minute-scale consistent world rollouts for virtual sets, digital doubles, and persistent virtual environments.

Industry relevance: The paper is a direct response to a competitive landscape in which GameGen-X, the Matrix-Game family (including Matrix-Game 3.0), Hunyuan-GameCraft, HY-World 1.5, and WorldPlay all report strong short-horizon controllability while withholding their training data. A public, multi-game, minute-scale corpus lowers the barrier for anyone—academic labs or smaller studios—to train and compare long-horizon models on a shared footing.

Future Directions

• Cross-domain generalization. The dataset is gameplay-only; whether models trained on it transfer to real-world video remains unexplored, and the authors name this as an open limitation.

• Beyond two minutes and 60 seconds. Continuous clips cap at two minutes while generation experiments only cover 60 seconds. Longer rollouts and closed-loop interaction—where generated frames feed back as inputs—are explicitly left for future work.

• Isolating what actually helps. The individual contributions of clip duration, camera supervision, and structured captions are not yet separated, so it is unknown which data axis drives the gains. Ablations that remove one signal at a time are the obvious next experiment.

• Residual annotation error. Despite multi-stage validation, estimated trajectories and generated captions retain errors (5.5% clip-level hallucination, 4.0% temporal mismatch, and cross-estimator pose discrepancies up to 11.03 degrees in translation direction). Better pose validation and caption verification remain open problems, as does the computational cost of large-scale construction.

Target Audience

Researchers and engineers working on video generation, interactive world models, and action- or camera-conditioned synthesis will find the dataset and pipeline directly usable. It is also valuable for game AI and embodied-agent researchers who need long-horizon, controllable visual training data, and for practitioners evaluating whether data curation rather than architecture is the limiting factor in their own long-rollout systems. Readers new to the area can follow the problem framing and the plain-language pipeline description without prior exposure to pose estimation internals.

Authors’ abstract

Recent advances in interactive world models have made it possible to generate controllable, dynamic environments from action inputs. Yet current models struggle to maintain coherence beyond roughly one minute of rollout, a limitation rooted not in model capacity but in the scarcity of long-horizon training data. Existing gameplay video datasets typically consist of clips under ten seconds, with captions that describe whole clips in coarse and undifferentiated text, which is insufficient for controllable long-horizon generation. We address this with GameVerse, a minute-scale gameplay video dataset purpose-built for long-horizon interactive world modeling. GameVerse contains 69.7K clips spanning 1,655 hours and 349.6 million frames across diverse first- and third-person gameplay, including 1,358 hours of continuous two-minute sequences, an order of magnitude longer than prior datasets. Clips pass through a unified pipeline of scene-consistent segmentation, quality filtering, camera pose estimation, and event-based caption annotation. Each clip is paired with a temporally stable camera trajectory that remains coherent across minutes, together with captions that separate static scene context from dynamic foreground events and align to event boundaries rather than fixed time windows. Comprehensive analysis shows that GameVerse surpasses existing public datasets in scale, diversity, and annotation quality. Experiments further demonstrate that training on GameVerse substantially extends the temporal horizon of interactive world models.