Scaling in Games: Continued Pre-Training For Embodied Agents In Diverse Virtual Worlds

AI summary

Overview

Research area: Embodied AI / foundation models for interactive agents — specifically, how continued pre-training (CPT) of vision-language backbones on gameplay data scales across data volume, data diversity, and model size when the "worlds" being learned are heterogeneous video games rather than one shared physical reality.

Technical level: Advanced. The paper assumes familiarity with causal next-token training, vision-language backbones (Qwen3.5), Chinchilla-style scaling laws, and imitation learning. The core message is nonetheless readable without that background.

Scope (one sentence): A controlled 120-configuration grid study (4 model sizes from 0.8B to 27B × 6 data budgets from 50 to 1,600 hours × 5 data mixtures) measuring how CPT scales and generalizes from in-corpus games to related and distant unseen virtual worlds.

What This Paper Is About

Scaling data, diversity, and model capacity has repeatedly improved embodied agents in the physical world, where all experience obeys the same laws of gravity, contact, and object permanence. The authors ask whether those gains carry over to virtual worlds, where visual styles, game rules, physics, control schemes, and objectives can differ completely from one world to the next. They answer with a systematic, single-variable-at-a-time study of continued pre-training to find out whether heterogeneous game experience is complementary (helps build generalist agents) or merely interference (degrades what the model learned in any single game).

Key Contributions

1. A controlled scaling study of CPT in virtual worlds. The authors disentangle three axes that prior work varied together — data volume (six budgets from 50 to 1,600 hours), mixture diversity (five mixtures S1–S5, from Minecraft-only to a broad multi-game corpus), and model size (0.8B, 2B, 9B, 27B Qwen3.5 backbones) — producing 120 training configurations and roughly 3.5 × 10^11 training tokens.

2. An empirical two-phase account of learning dynamics plus a three-factor scaling law. They identify an early "interface phase" with broad cross-world benefits (shared formats, controls, and action primitives) followed by a "world phase" whose gains narrow to trained and related worlds. They extend the Chinchilla law with a diversity ratio and a zero-share transfer term, reaching R² = 0.982 on Minecraft validation loss and R² = 0.995 on in-corpus loss.

3. A three-level evaluation ladder. Models are assessed on validation losses and gameplay benchmarks at increasing distance from the training distribution: in-corpus worlds (MCU for Minecraft, ViZDoom for DOOM), unseen related 3D worlds (OmniGameArena, seven worlds), and unseen distant worlds (GameWorld, mostly 2D and 2.5D). This anchors the resulting models against Minecraft specialists and frontier models.

4. A public reference point. The released models are competitive with Minecraft specialists and frontier models in 3D worlds and, under real-time interaction, outperform frontier models by up to 5×.

Main Findings

• Early training yields cross-world gains; later gains are world-specific. Key, mouse, and format token losses drop sharply in the first tens of hours. After only 50 hours, validation losses fall below base-model levels on all three validation sets — even for games absent from the mixture. The authors call this the interface phase. After it, progress depends on seeing relevant worlds (the world phase).

• More data helps where the data matches. From 50 to 1,600 hours, mean Minecraft validation loss falls by about 10.4% across the four model sizes under the Minecraft-heavy mixtures S1–S3, tracking large MCU gains. Scaling volume without enough Minecraft trajectories yields negligible further reduction in that specific loss.

• Diversity mainly buys transfer to related worlds. At 1,600 hours, the mean OmniGameArena score for S2–S5 is roughly 2.8× that of S1, and all 16 mixed configurations beat S1 at matched model scale. ViZDoom also improves even though DOOM is a small fraction of the corpus.

• Distant worlds do not follow. On GameWorld, S1-trained models degrade as data volume grows; multi-game training mitigates but does not reverse this, and models fail to consistently surpass their base models. At the largest scale, GameWorld performance degrades across all five mixtures as data volume increases. Out-of-distribution loss does not follow a scaling law at all (fit gives β ≈ 0, R² = 0.855 reflecting only static mixture differences).

• The best mixture depends on the target. Across all 24 model-size × data-volume configurations, Minecraft validation loss is lowest with S1, but in-corpus loss is minimized with S3 or S4. Differences among S2–S5 on in-corpus loss are negligible compared with their gap from S1. Even the zero-Minecraft S5 mixture beats base models on Minecraft validation loss.

• Bigger models are more data-efficient but can generalize worse under single-game training. Larger models reach comparable or better performance with up to 32× less gameplay data, and large models trained on 50 hours outperform small models trained on 1,600 hours. Yet at 1,600 hours, 27B-S1 underperforms its own base model on all three non-Minecraft benchmarks, with a maximum GameWorld drop of roughly 17 points.

• The scaling law fits training, Minecraft, and in-corpus loss but not transfer. R² values are 0.980, 0.982, and 0.995 with RMSE of 0.0041, 0.0020, and 0.0012. Refit without the 27B models, the 1,600-hour checkpoints, or the 50-hour checkpoints, it predicts held-out losses with RMSE between 0.0016 and 0.0027.

• Competitive in 3D worlds, dominant in real time. On MCU, 27B-S1 outperforms the best specialized model by 13 points, and even 0.8B-S1 surpasses 7B specialists; 0.8B-S3 completes 36% of GUI tasks while behavior-cloning models trained on tens of thousands of hours score near zero on instruction-conditioned tasks. Under time-paused interaction, CPT amplifies the 27B backbone by roughly 6× on MCU and 2.6× on OmniGameArena, with 9B and 27B models matching ~300B open models.

• Latency, not just skill, decides real-time play. In real-time settings, frontier models suffer an approximate 90% score reduction on ViZDoom and MCU, while 0.8B–9B GameScaling models retain 88% of their time-paused performance. GameScaling-9B-S3 averages 25.3 across the four benchmarks versus 19.2 for the best-tested frontier model.

• Distant-world transfer is the binding constraint. Against frontier models on the time-paused GameWorld benchmark, frontier models score twice as high; CPT only helps there at 0.8B and 2B. The 27B-S3 checkpoint gains 16.0 points on time-paused OmniGameArena over its base model but loses 4.0 points on GameWorld.

Methodology in Plain English

The team treats gameplay as a visual decision process. At each step the model receives a prompt, the current screen, and a history of 20 interleaved image–action turns; it optionally writes a short reasoning trace and then emits a mandatory structured action string. That string is parsed into low-level device commands: discretized mouse movement, scroll, and up to six key or button presses held for 33 ms, covering 200 ms per chunk. Observations arrive at 5 Hz and controls run at 30 Hz. The model is trained with ordinary next-token prediction on these action strings — pure imitation of recorded human play, with no environment reward.

Data comes from three public trajectory sources (VPT, D2E, OpenP2P) at 85% of tokens, plus 15% fixed vision-language and embodied data to preserve general reasoning. Diversity is varied by changing the Minecraft share of the gameplay portion to 100%, 51.4%, 27.1%, 2.8%, and 0% across S1–S5, while the pool of other games is held fixed at 124 source streams spanning 79 canonical non-Minecraft labels (80 with Minecraft). The authors quantify diversity with the Hill number of order one, the "effective number of worlds," which rises from 1.0 for S1 to 14.5, 35.2, 60.1, and 58.7 for S2–S5.

Each configuration is trained with the same recipe — AdamW, a warmup–stable–decay schedule, all parameters including the vision encoder updated — so only the three axes of interest change. Evaluation happens at three distances: MCU and ViZDoom for worlds in the corpus, OmniGameArena for seven unseen 3D worlds whose action primitives have semantic counterparts in training, and GameWorld for 2D/2.5D games that differ in visuals, physics, controls, and goals. The full grid took roughly 45,000 H200 GPU-hours with a single training seed.

Why This Matters

Research impact. Prior virtual-world CPT work changed data, diversity, and model size simultaneously, so no one could say which factor produced which gain. This paper isolates them and shows that in-domain loss scaling laws do not predict transfer — a caution for anyone who assumes a good fit implies a general agent. The interface-phase/world-phase distinction gives a concrete hypothesis for why early training transfers and late training does not.

Potential applications (not tested in the paper):

• Game AI and NPC behavior that adapts to unfamiliar titles rather than being hand-tuned per game.
• Automated playtesting and QA, where an agent plays builds and reports failures without per-game scripting.
• General computer-use agents, since the same keyboard–mouse interface and GUI-task evaluation transfer directly to desktop automation.
• Faster adaptation of embodied policies to new simulators or robot embodiments whose dynamics and controls differ from the training setting.

Industry relevance. The finding that a 0.8B–9B model beats frontier models when the world does not pause is directly actionable: for real-time deployment, inference latency belongs in the benchmark, not in a footnote. The paper also argues that comparisons should report task scores, response times, and deployment infrastructure together, since its own models ran on local vLLM endpoints on an H200 cluster while frontier models incurred provider network latency.

Future Directions

• Moving beyond imitation. CPT learns from recorded actions with no feedback on task success or failure. The authors ask whether the distant-world gap stems from this objective, from insufficient game coverage, or both, and propose adding online environmental feedback during play.
• Scaling world diversity directly. Because the 124-stream pool was held fixed across S2–S5, the study cannot separate "more worlds" from "different world proportions." Varying the number of worlds is left explicitly to future work.
• Seed variance and uncertainty. The 120-condition grid ran with one training seed at ~45,000 H200 GPU-hours; repeated runs with different seeds are deferred.
• Explaining and closing the distant-world gap. Distant-world loss does not follow the fitted law and performance stagnates or declines with more training, so the mechanism of negative transfer — and whether richer training stages beyond CPT can fix it — remains open.

Target Audience

Researchers and engineers working on embodied agents, game-playing foundation models, and large-scale pre-training recipes will get the most from this paper, particularly those deciding how to allocate a fixed data budget across breadth versus depth. It is also valuable to practitioners deploying agents in latency-sensitive interactive settings, and to anyone who needs a realistic reference for what continued pre-training can and cannot deliver before committing to a training run. Readers wanting only the practical takeaway can read the abstract, Section 4.2, and Section 6.

Authors’ abstract

Scaling data, diversity and model capacity has improved embodied learning in physical environments, where diverse experiences remain grounded in shared physical laws. Whether these gains extend to virtual worlds with different visual styles, dynamics, controls, and objectives remains unclear. We investigate this question through a controlled study of continued pre-training (CPT) across three axes: data volume, diversity, and model size. Our 120 training configurations span four model sizes from 0.8B to 27B, six data budgets from 50 to 1,600 hours, and five mixtures ranging from Minecraft-only data to diverse multi-world corpora. We assess performance and generalization using three-level validation losses and gameplay benchmarks from in-corpus worlds to unseen worlds. Our results show: (1) more data improves in-corpus performance, diversity improves transfer to related worlds with limited gains in distant worlds, and model size improves data efficiency but generalizes worse to distant worlds under single-world training; (2) we observe an early interface phase with broad cross-world benefits and a later world phase with gains narrowed to in-corpus and related worlds, and fit an empirical scaling law relating validation loss to data volume, diversity and model size; and (3) our models are competitive with Minecraft specialists and frontier models in 3D worlds, and perform best in real-time settings. However, the limits of imitation and persistent gaps in distant worlds suggest that building generalist embodied agents requires richer training stages beyond continued pre-training.