Player-1: Learning to Play from Synthetic Experience

AI summary

Overview

Research area: Computer-use agents and vision-language models applied to game control, with a post-training pipeline combining supervised fine-tuning and multi-environment reinforcement learning with verifiable rewards (RLVR).

Technical level: Advanced. The paper assumes familiarity with PPO, generalized advantage estimation, vision-language-action models, and LLM post-training.

Scope: The paper presents Player-1, a post-training framework that initializes computer-use models on 6,044 curated gameplay trajectories and then improves them through multi-environment reinforcement learning in a suite of sixteen synthetic 2D and 3D environments, evaluated on MineDojo-300, ViZDoom-10, and GameWorld.

What This Paper Is About

Computer-use agents can read screens and issue keyboard and mouse actions, but games remain hard because they demand precise, sustained, and concurrent control under rapidly changing visual feedback. Existing game agents rely heavily on large-scale gameplay demonstrations that cover only a limited portion of the states an agent encounters, while scalable environments for interactive training are hard to build. The goal of this paper is to test whether synthetic experience, generated in controllable environments with programmatically verifiable rewards, can push game-control ability beyond what supervised fine-tuning alone achieves, and whether those gains carry over to games never seen during reinforcement learning.

Key Contributions

1. Player-1, a post-training framework for learning transferable game control from synthetic experience. It builds on supervised initialization and uses an extended computer-use action space that adds relative pointer motion, sustained inputs (holds lasting one full control step), and composite actions (two or more concurrent keyboard or mouse inputs). 6,044 curated gameplay trajectories provide the supervised initialization for later interactive learning.

2. Player-1-Gym, a suite of sixteen synthetic 2D and 3D environments organized around control demands shared across games: aiming (2), combat (5), driving (4), navigation (4), and UI interaction (1). Each environment defines programmatically verifiable rewards over game states, and observations and control dynamics are randomized along four dimensions (Appearance, Camera, Scene, Control) to supply diverse experience for multi-environment RL.

3. Evidence that learning from synthetic experience advances game control beyond supervised demonstrations. Player-1-9B and Player-1-35B-A3B gain 27.8 and 17.7 percentage points in MineDojo-300 success over their SFT policies, with additional gains on ViZDoom and GameWorld, both outside the RL training suite.

4. Analyses of transfer and behavior. Broader environment coverage in the training suite is associated with improved transfer, and composite-action use rises during RL even without an explicit reward for action composition.

Main Findings

• Large Minecraft gains over SFT: On MineDojo-300, SFT raises success from 9.21% to 10.54% for the 9B model and from 7.79% to 23.38% for the 35B-A3B model. Interactive learning then raises Player-1-9B to 38.33%, a gain of 27.79 percentage points over its SFT policy, and Player-1-35B-A3B to 41.04%.

• Comparison with evaluated baselines: Player-1-9B reaches 38.33% MineDojo-300 success versus 12.63% for the strongest evaluated baseline, GameWAM. Other evaluated baselines on MineDojo-300 were JARVIS-VLA (6.71%), UI-TARS-1.5-7B (0.38%), NitroGen (0.00%), and Open-P2P-1.2B (0.29%). Player-1-35B-A3B reaches 41.04%.

• Improved task coverage and repeatability: For the 9B model, the number of MineDojo-300 tasks completed in at least four of eight trials increases from 20 to 136. Player-1-9B covers 136/300 at the ≥ 4 of 8 threshold and Player-1-35B-A3B covers 142/300.

• Transfer to first-person control on ViZDoom-10: Player-1-9B increases scaled return from 0.4361 after SFT to 0.6922. Player-1-35B-A3B achieves an overall scaled return of 0.7236, surpassing GameWAM, the strongest external baseline at 0.3938, by 83.7%.

• Transfer to browser games on GameWorld: Across 3 × 170 episodes, progress rises from 31.43% to 34.45% for the 9B model. Player-1-35B-A3B reaches 17.25% success and 38.79% progress, compared with 15.49% success and 34.60% progress for its SFT checkpoint.

• Supervised initialization matters for RL: Over 100 RL steps, the SFT-initialized 9B policy rises from 17.67% to 51.67% Minecraft success, while training from the base model rises from 12.67% to 16.67%, leaving a 35.00-percentage-point gap at step 100.

• Learning curves rise across the suite: Mean episode return and mean game score both rise across 200 RL steps for both model sizes, with the largest gains early in training. The 35B-A3B curve finishes slightly above the 9B curve on both measures. Learning trajectories differ by environment; simpler tasks such as 2D aiming improve rapidly and level off, while 2D circuit driving gains emerge much later.

• Composite-action use increases during RL: Across Player-1-9B's in-suite validation trajectories, the share of turns with composite actions rises from 64.7% at the start of RL to 85.4% after 50 RL steps and 93.2% after 100 RL steps, without an explicit reward for action composition.

• Generalization to an unseen commercial game: Player-1 was integrated into Watch Dogs 2, a game unseen during both SFT and RL, through an observation–action loop with visual inputs and keyboard–mouse controls, performing on-foot navigation, target approach, vehicle interaction, and driving.

Methodology in Plain English

The pipeline has two stages.

Stage one, supervised initialization. The authors gathered public datasets and in-house recordings and converted them into one shared game-control action space. Source controls were aligned to a shared 200-ms action interval, mouse input was calibrated to normalized pixel-equivalent coordinates, and incomplete or redundant recordings were filtered. The result is 6,044 gameplay trajectories comprising 295,097 interaction turns, drawn from 2,544 source-qualified recording sessions across 172 games, corresponding to roughly 16.4 hours of sampled control time. A vision-language model was used to annotate each recorded action with a task instruction and per-turn visual reasoning, conditioned on the instruction, current and past observations, and the action target. The models were then fine-tuned to predict reasoning and action given the instruction, current screenshot, and retained history, using a 2+2 history setting (two recent visual–action turns plus two earlier action-only turns) and action-balanced sampling to increase exposure to rare controls.

Stage two, reinforcement learning in synthetic environments. The authors built Player-1-Gym: sixteen environments running on their own shared 2D and 3D engines with a unified runtime that emits screenshots and executes the shared action space. Each environment defines a cumulative score over game state, and the step reward is the score increment, which gives dense verifiable feedback without a learned judge; privileged state is not exposed to the policy. Appearance, camera, scene, and control-response parameters are randomized at reset and fixed within each episode. The SFT-initialized policy is then optimized with PPO across the environments, using per-environment advantage normalization of generalized advantage estimates so that differences in reward scale across environments do not distort policy-update magnitudes.

Evaluation. Both model sizes were trained on the full sixteen-environment suite for 200 RL steps, with 256 episodes per step allocated equally across environments and at most 50 interaction turns per episode. Evaluation used MineDojo-300 (300 Minecraft tasks, 8 trials each), ViZDoom-10 (ten of the twelve available maps, 50 trials per map), and GameWorld (170 tasks across 34 games, 3 trials each), alongside in-domain evaluation of the sixteen Player-1-Gym environments.

Why This Matters

Impact on research. The paper separates two sources of improvement — supervised initialization and learning from self-generated experience — and shows that the second adds substantially more on game control than the first, with gains that extend to games never used in RL. It also offers a concrete recipe for building scalable interactive environments around recurring control demands rather than around individual titles, which is relevant to the broader effort to train agents with verifiable rewards.

Real-world applications:

• Software and browser automation that requires sustained input, camera-relative pointing, and simultaneous key and mouse actions rather than one-shot clicks.
• Robotic or simulated teleoperation interfaces that share the same control demands: precise aiming, sustained movement, and coordinated concurrent inputs.
• Automated game testing and quality assurance, where an agent can exercise mechanics and reach game states programmatically.
• Accessibility tooling that maps high-level task instructions onto low-level keyboard and mouse control in visually complex interfaces.

Industry relevance. The work is tied to ARC Lab, Tencent IEG and produced two released checkpoints, Player-1-9B and Player-1-35B-A3B, built on Qwen3.5-9B and Qwen3.5-35B-A3B backbones, with public project and code pages. Table 1 in the paper positions Player-1 against Game-TARS, SIMA 2, NitroGen, and Odysseus on supervision, RL environments, evaluation scope, and pipeline/model availability; of these, Player-1 is listed as available along with NitroGen and Odysseus. For teams building computer-use agents, the result that gains transfer without RL in each target game is directly relevant to cost and deployment planning.

Future Directions

• Scaling environment coverage further. The environment-scaling analysis holds a fixed budget of 25,600 episodes over 100 RL steps and varies the suite across 2, 4, 8, or 16 environments from the same Player-1-9B-SFT checkpoint; a natural next step is to push beyond sixteen environments and test whether the trend continues.

• Broadening verifiable rewards to more commercial games. The current design depends on programmatic scoring rules per environment. The paper notes that implementing programmatic rewards across diverse commercial games is difficult to scale, leaving open how to bring more titles, especially commercial ones, under the same reward-driven regime.

• Understanding why composite actions emerge. Composite-action use rises from 64.7% to 93.2% over 100 RL steps without an explicit reward for action composition. Whether this behavior can be shaped deliberately, and how it relates to the coordinated control that games require, remains an open question.

• Separating model scale from data scale. The two released sizes are 9B and a 35B-A3B mixture-of-experts variant, and the 35B-A3B curve finishes slightly above the 9B curve. The ablation of initialization (Base→RL versus SFT→RL) is reported for the 9B model only, leaving the interaction between scale, supervised initialization, and synthetic experience less explored.

Target Audience

Researchers and engineers working on computer-use agents, game-playing agents, vision-language-action models, and reinforcement learning with verifiable rewards. It is also useful for practitioners who need to build scalable interactive training environments, and for readers tracking the comparison between supervised initialization and post-training from self-generated experience in agentic settings. The paper is technical and assumes comfort with RL training terminology and LLM fine-tuning; readers without that background may find the appendices on action standardization and reward construction the most accessible entry point.

Authors’ abstract

Computer-use agents have demonstrated strong capabilities in interpreting screens and executing keyboard and mouse actions. However, games, with their visually rich worlds and diverse mechanics, remain challenging and offer a demanding next step toward general digital control. Current game-agent learning pipelines often rely on large-scale demonstrations with limited coverage of dynamic game states, while scalable environments for interactive training remain difficult to obtain. We introduce Player-1, a post-training framework for learning game control beyond demonstrations through synthetic experience. We extend the computer-use action space with relative pointer motion, sustained inputs, and composite actions. For supervised initialization, we fine-tune computer-use models on 6,044 curated gameplay trajectories that pair recorded actions with task instructions and visual reasoning. We construct Player-1-Gym, sixteen synthetic 2D and 3D environments designed around control demands shared across games. Programmatically verifiable rewards and randomized observations and control dynamics support multi-environment reinforcement learning with PPO. The resulting Player-1-9B and Player-1-35B-A3B improve MineDojo-300 success over their SFT policies by 27.8 and 17.7 percentage points, respectively. Player-1-9B reaches 38.33% success compared with 12.63% for the strongest evaluated baseline, with further gains over SFT on ViZDoom-10 and GameWorld. Further analyses show that broader synthetic environment coverage improves transfer, while the use of composite actions that combine multiple keyboard and mouse inputs increases during RL. We demonstrate that learning from synthetic experience substantially improves game control beyond supervised initialization, with gains that generalize beyond the RL training suite.