IPR-1: Interactive Physical Reasoner

AI summary

Overview

Research area: Artificial Intelligence, specifically embodied agents, interactive physical reasoning, world models, and vision-language-action (VLA) systems.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (PPO, DQN, GRPO), world models (Dreamer, Genie, JEPA), vision-language models (GPT-4o, GPT-5, Qwen3-VL), and vector-quantized latent representations (VQ-VAE).

One-sentence scope: The paper proposes IPR, a paradigm that couples a physics-centric latent action code (PhysCode) with a latent world model and a vision-language-model policy, and evaluates it on a benchmark of 1,000+ heterogeneous games organized into a three-level Survival/Curiosity/Utility hierarchy.

What This Paper Is About

Humans learn physics and causality not from labels but from interacting with the world, and their prediction and reasoning improve steadily as experience accumulates. This paper asks whether an AI agent can do the same: acquire physical and causal understanding from interactive experience and keep improving as it sees more games and more interaction steps. The authors argue that existing approaches fail in complementary ways — vision-language models (VLMs) reason semantically but cannot look ahead, while world models imagine futures but imitate visual patterns rather than analyze physics — and they build a system that combines both.

Key Contributions

1. Formulating the G2U (Game-to-Unseen) problem and benchmark. The authors curate 1,000+ heterogeneous games that exhibit significant visual domain gaps, and organize evaluation into a three-level hierarchy — Survival, Curiosity, Utility — inspired by Maslow's hierarchy of needs, spanning physical intuition to goal-driven reasoning. This benchmark is used to diagnose the strengths and weaknesses of prediction-based, RL-based, and VLM-based methods.

2. Proposing IPR (Interactive Physical Reasoner). World-model rollouts score and reinforce a VLM policy in the same action space, so that interactive experience steadily builds physical reasoning ability. The abstract describes this as "using world-model rollouts to score and reinforce a VLM's policy."

3. Introducing PhysCode. A physics-centric action code that aligns semantic intent with dynamics, fusing action semantics with visual dynamics to provide a shared action space for prediction and reasoning — bridging world-model prediction and VLM reasoning.

4. Demonstrating scaling and transfer. With an 8B backbone, IPR performs robustly across the three levels and surpasses GPT-5 overall. The paper reports that performance improves with more training games and more interaction steps, and that the model zero-shot transfers to unseen games.

Main Findings

• IPR is robust across all three evaluation levels. In the main comparison table covering Survival, Curiosity, and Utility, IPR records a Survival mean of 0.252, Curiosity mean of 1.173, and Utility mean of 0.493, with an overall average rank of 4.9 across 30 methods. Excluding the two control-group entries (Random and Human), its reported ranks are 1/28 on Survival, 5/28 on Curiosity, 5/28 on Utility, and 4.9/28 overall.

• Strong Survival and goal-directed performance. IPR reaches a Survival Ratio@Top-3 of 72.0 percent and a Utility Ratio@Top-3 of 22.0 percent, with an average Survival rank of 2.6 and Utility rank of 8.5. For reference, GPT-5's best overall rank in the table is 13.3.

• PhysCode reduces interface aliasing and transfers better than keyboard or language interfaces. In joint training across heterogeneous-physics games, raw keyboard/mouse inputs show cross-game conflicts where the same key triggers different behaviors; language partially alleviates this via semantics, while PhysCode separates actions by dynamics with minimal degradation under physics shifts. In a leave-n-out protocol — training on all but 10 games and evaluating zero-shot on the held-out titles — PhysCode transfers more reliably than keyboard or language instructions.

• Physics-conditioned transfer works, with exceptions. When target games match the training physics, zero-shot performance is typically higher, which the authors read as evidence that PhysCode captures reusable physical mechanisms rather than game-specific bindings. They note notable exceptions such as inertia, which may already be covered by projectile/impulse.

• Prediction is what strengthens reasoning, not just reward maximization. In the ablation on the same Qwen3-VL-8B backbone, IPR attains the highest Curiosity (2.77) while keeping strong Survival (0.76) and Utility (1.34). VLM+PPO achieves the best Survival (1.00) and Utility (1.23) but suppresses Curiosity (1.79), and VLM+BC barely changes Survival (0.62 to 0.63) while hurting Curiosity and Utility. Combining PPO with BC degrades all three metrics.

• Different paradigms fail in complementary ways. Prediction-based world models (DreamerV3, V-JEPA2, GenieRedux) are strong on exploration but weaker on goal-driven tasks; RL methods (PPO, DQN) are strong when rewards are well-shaped but weaker on exploration and goals; behavior cloning excels at low-risk survival; and pretrained VLMs are strong at goal-conditioned reasoning but cannot predict consequences in visual state space.

• Zero-shot performance scales with the number of training games. On a held-out target set of 50 games never used for training, across all three objectives performance increases steadily as the number of stratified training games N grows, with the steepest early gains on Survival, followed by sustained improvements on Curiosity and Utility.

• The dataset is broad. The benchmark covers 863 open-source retro titles (via stable-retro), 134 lightweight HTML/Canvas games, and 3 commercial games, characterized along seven axes: game category, control interface, visual complexity, view perspective, causal mechanism, physical principle, and operational difficulty. Human play was recorded at 60 FPS for 4 minutes per title.

• The main comparison used 200 games. Models were evaluated against a baseline suite including Multitask PPO, shared-parameter DQN, GPT-4o, GPT-5, Qwen3-VL-30B-A3B, DreamerV3, V-JEPA2, GenieRedux, ACT, and Qwen3-VL-8B.

• Stated limitation. The authors note the work is still limited to game environments and short-horizon imagination, leaving real-world transfer and longer-horizon reasoning to future work.

Methodology in Plain English

The system is trained in three stages.

Stage 1 — Learning a shared action vocabulary (PhysCode). Instead of using raw keypresses or natural-language instructions, the authors learn a discrete codebook of action codes using a VQ-VAE objective. Each code is conditioned on three cues: game-specific visual appearance (DINOv3 features), domain-agnostic motion (optical flow between consecutive frames), and lightweight semantic hints from a T5 encoder. Because language alone cannot express fine-grained dynamics such as impulse magnitude or frictional slip, the flow and visual features carry those details while semantics act as guidance. Optical flow is available only during pretraining, so it acts as privileged information; modality dropout and gate sparsity force the encoder to work at test time from appearance and semantics alone.

Stage 2 — Training a latent world model with a critic. With the action vocabulary fixed, a feature-level world model predicts future features and a value estimate conditioned on latent actions. The authors predict in latent space rather than pixels because features compress appearance variance and rendering noise, making dynamics more shareable across games. Training uses a feature-prediction loss followed by a Q-learning-style value objective with target values computed from rollout returns via standard TD backups.

Stage 3 — Reinforcing VLM reasoning with prediction. Qwen3-VL-8B is used as the backbone, with its tokenizer extended so the model can emit PhysCode tokens directly while preserving language ability. Given the current context and a goal, the VLM samples B candidate PhysCode sequences; the world model runs short-horizon imagined rollouts to assign each a predicted return; advantages are computed and the policy is updated with GRPO, with a KL term against the reference policy. At inference, the VLM proposes candidates, the world model scores and prunes them via rollouts, and a router maps the selected PhysCode to actual environment controls. Repeated interaction under this prediction-in-the-loop scheme is what reinforces the VLM.

Evaluation. Three levels are used. Survival measures normalized episode length relative to a per-game reference horizon (for example, median survival under a random policy). Curiosity embeds frames with a pretrained CLIP encoder and computes the area under a multi-scale metric-space magnitude curve (following Magnipy). Utility reports a human-normalized score computed from the agent metric, a random baseline, and human performance.

Why This Matters

The paper reframes a central question for embodied AI: rather than choosing between exploration (RL), full-scene prediction (world models), or static priors (VLMs), it tests a blended ratio of these components organized around shared physical and causal mechanisms. Its reported result — that an 8B model surpasses GPT-5 overall on this suite and keeps improving with more games and more interaction — is evidence for the "ratio" argument rather than for sheer scale. It also supplies a diagnostic benchmark showing where each paradigm breaks.

Real-world applications suggested by the work and its framing:

• Robotics. The paper notes that some action semantics in its word cloud are shared with general robotic operations, and states the authors will extend the paradigm to real-world interactive environments and robotic tasks.
• Simulation and training environments. A shared latent action space that separates actions by dynamics rather than interface could reduce the cost of porting agents between simulators or control layouts.
• Game and interactive-media agents. The benchmark itself targets agents that operate across radically different games and interfaces.
• General embodied agents requiring look-ahead. The finding that prediction, not just reward shaping, drives reasoning quality applies to any setting where anticipating consequences matters more than optimizing an immediate score.

Industry relevance. The result that an 8B open-backbone model outperforms GPT-5 on this suite while scaling with training games has direct implications for practitioners deciding whether to build on frontier closed-source models or on smaller models wrapped with an explicit learned dynamics module. The diagnosis of failure modes — console aliasing from raw key bindings, amplitude errors from language-only actions, and missing foresight — is also practically useful for anyone designing action interfaces for agent training data.

Future Directions

• Extension to real-world and robotic settings. The authors explicitly state they will extend the paradigm beyond games to real-world interactive environments and robotic tasks, which would test whether PhysCode's shared semantics transfer outside simulated physics.
• Longer-horizon reasoning. The paper acknowledges short-horizon imagination as a current limitation; extending rollouts and credit assignment over longer horizons is a natural next step.
• Refining the physics taxonomy. The authors observe that their coarse physics taxonomy does not perfectly align with the agent's internal abstractions (citing inertia as an exception), raising the question of how to better characterize and group environments by underlying mechanism.
• Scaling the interaction budget. Since zero-shot transfer improves steadily with the number of training games and the paper argues that more interaction steps help, the question is how far this scaling continues and whether it plateaus.

Target Audience

This paper is most useful to researchers and engineers working on embodied AI, interactive agents, world models, and VLA systems, particularly those interested in generalization across visually and physically heterogeneous environments. It will also benefit practitioners evaluating whether to combine reinforcement learning with learned dynamics models and VLM policies, and benchmark designers looking for a hierarchical evaluation framework (Survival, Curiosity, Utility) that separates physical intuition from goal-directed reasoning. Readers without a background in reinforcement learning or latent-variable models will find the methodology sections demanding.

Further demos and project details are available at https://mybearyzhang.github.io/ipr-1. The authors are affiliated with Shanghai Jiao Tong University, with one affiliation listed at Carnegie Mellon University; the arXiv license is the arXiv.org perpetual non-exclusive license (arXiv:2511.15407v4 [cs.AI], 24 May 2026).

Authors’ abstract

Humans learn by observing, interacting with environments, and internalizing physics and causality. Here, we aim to ask whether an agent can similarly acquire human-like reasoning from interaction and keep improving with more experience. To study this, we introduce a Game-to-Unseen (G2U) benchmark of 1,000+ heterogeneous games that exhibit significant visual domain gaps. Existing approaches, including VLMs and world models, struggle to capture underlying physics and causality since they are not focused on core mechanisms and overfit to visual details. VLM/VLA agents reason but lack look-ahead in interactive settings, while world models imagine but imitate visual patterns rather than analyze physics and causality. We therefore propose IPR (Interactive Physical Reasoner), using world-model rollouts to score and reinforce a VLM's policy, and introduce PhysCode, a physics-centric action code aligning semantic intent with dynamics to provide a shared action space for prediction and reasoning. Pretrained on 1,000+ games, our IPR performs robustly on levels from primitive intuition to goal-driven reasoning, and even surpasses GPT-5 overall. We find that performance improves with more training games and interaction steps, and that the model also zero-shot transfers to unseen games. These results support physics-centric interaction as a path to steadily improving physical reasoning. Further demos and project details can be found at https://mybearyzhang.github.io/ipr-1.