PhysEvolve: Verifier-grounded Self-evolution for Physical Reasoning with Executable Worlds

AI summary

Overview

Research area: Multimodal machine learning — physical and causal reasoning in vision-language models (VLMs), reinforcement learning with verifiable rewards (RLVR), and self-evolving training systems.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (specifically GRPO), physics simulation APIs, VLM training pipelines, and counterfactual/causal inference terminology.

Scope: The paper introduces PhysEvolve, a self-evolving training framework in which a shared VLM proposes executable 3D world programs that a deterministic physics engine validates, freezes, and converts into paired counterfactual simulation branches, yielding discrete verifiable training tasks that improve causal physical reasoning without any human-annotated intervention data.

What This Paper Is About

Vision-language models handle static object recognition far better than they handle physical dynamics, intervention effects, and counterfactual outcomes, and the bottleneck is data: real videos only ever show the one trajectory that actually happened, so counterfactual alternatives are unobservable and expensive to annotate. The paper asks whether the "self-evolving" paradigm — where a model generates its own training problems and an external verifier supplies exact answers — can be extended from symbolic domains like code and math into continuous physics by substituting a physics simulator for the code interpreter. The authors show that a naive substitution fails structurally, and propose PhysEvolve, which instead has the model propose executable environments while a fixed simulator alone authors paired baseline/counterfactual branches and binds discrete answer labels.

Key Contributions

1. Verifier-grounded physical self-play. PhysEvolve is presented as the first self-evolving framework that trains VLMs for causal physical reasoning through model-proposed executable worlds, enforcing a strict information boundary: the policy explores environments, but only the deterministic simulator determines physical validity and ground-truth labels.

2. Intervention-grounded task synthesis. An execution pipeline runs matched baseline and body-elided branches under identical seeds and time horizons, transforming continuous multi-body dynamics into discrete, answer-bound multiple-choice items with exact verifier rewards.

3. Out-of-distribution causal transfer and empirical audit. Self-play on rigid-body programs transfers zero-shot to complex continuous media (cloth, fluid, rope, soft bodies) and real-world video, and the paper provides an audit showing why naive single-trajectory physical self-play collapses.

4. Diagnosis of two structural failure modes. The paper identifies and empirically demonstrates that (a) single-trajectory readouts cannot ground causal intervention tasks, and (b) continuous proximity rewards inflate learnability signals for uniformly incorrect rollouts, inducing reward hacking.

Main Findings

• Targeted gains on PhysBench (matched 8B comparison). Starting from Qwen3-VL-8B-Instruct, PhysEvolve improves Throwing by 8.78 percentage points (54.9 vs 46.1), Judgment by 5.08 points (40.0 vs 34.9), Motion by 1.74 points (56.5 vs 54.7), and overall accuracy by 0.55 points (53.5 vs 52.9).

• Naive self-play yields essentially nothing. The single-trajectory, continuous-reward baseline records negligible or negative differences on these same categories (−0.24 on Throwing, +0.26 on Judgment), despite a higher overall CausalPhys number (55.8) than PhysEvolve (54.5).

• Zero-shot transfer to unseen continuous media on ContPhy. Trained only on rigid-body simulations, PhysEvolve raises Counterfactual Removal by 7.38 points (51.64% vs 44.26%) and Goal-Pit by 24.44 points (44.44% vs 20.00%), the latter exceeding even the much larger Qwen3-VL-32B (26.7%). ContPhy overall rises 2.5 points (39.5 vs 37.1).

• Gains on real-world video in CausalPhys. Anticipation improves +2.46 points (58.4 vs 55.9) and Intervention +2.27 points (45.0 vs 42.8), with overall accuracy +0.6 (54.5 vs 53.9).

• Branch execution is the indispensable component. Removing branch simulation and the dependent Counterfactual/Inverse task families drops CausalPhys overall from 54.50 to 51.42 (−3.07), Intervention from 45.03 to 41.71 (−3.32), ContPhy overall from 39.54 to 37.28 (−2.26), Counterfactual Removal from 51.64 to 43.44 (−8.20), and Goal-Pit from 44.44 to 21.11 (−23.33), i.e. back to base level.

• Joint Proposer–Solver optimization is critical. Freezing Proposer updates and training only the Solver causes representational collapse: accuracy falls to 15.01% on CausalPhys (vs 54.50) and 17.85% on ContPhy (vs 39.54), with answer parse rate dropping to 26.07% on CausalPhys and to 0% in later steps (ContPhy parse rate is listed as 99.90% in the table).

• Graded proximity reward is deceptive. In the naive baseline, Forward prediction reaches a mean graded reward of 0.373 with 0.00% exact accuracy and 100/100 zero-exact rounds; Inverse reaches 0.664 mean reward at 6.63% exact accuracy (75/100 zero-exact rounds); Parameter reaches 0.787 at 5.44% exact accuracy (83/100 zero-exact rounds).

• Mode collapse under graded rewards. The error audit finds the policy learns to output stationary initial velocities (746 instances) or the median rung of the parameter ladder (145 instances) to harvest partial credit.

• Task-space starvation. Across 100 training rounds, the naive single-trajectory verifier generated zero intervention questions and zero counterfactual tasks.

• Fine-grained transfer. PhysEvolve improves Fluid Flow by 11.00 points and Action Substitution by 9.09 points relative to Base, outperforming the branch-disabled control by 23.00 points on Fluid Flow.

• Item-level paired analysis at the terminal checkpoint. PhysEvolve correctly answers 69 Throwing items that Base misses against only 33 reversals (net gain of 36), with net gains of 39 and 25 items on Judgment and Motion.

• No catastrophic forgetting of general visual capability. Performance remains close to or above the base model across most non-target partitions, including static relations and object identification (reported in the appendix).

Methodology in Plain English

The framework has three moving parts that share one VLM (Qwen3-VL-8B-Instruct), which acts as both Proposer and Solver.

1. Proposing worlds, not questions. Instead of writing a text question, the Proposer is given a "reasoning intent" — one of eight intents spanning three Newtonian paradigms: projectile motion (Throwing), impulse-momentum transfer and collisions (Collision), and continuous kinematics and contact (Motion). It outputs two things: a world program written in PhysDSL (a Python API over rigid-body simulation specifying geometry, mass, friction, restitution, initial states, joints, springs, gravity, duration, and camera) and an evidence schema declaring the semantic roles of key entities. The executor checks that the declared roles actually exist in the realized simulation and rejects proposals that cannot support the requested intent.

2. The simulator, not the model, writes the answer. The world is executed in a sandbox, validated for geometry and non-trivial physical events, repaired only for minor placement or syntax issues, and then frozen so all subsequent branches share identical initial conditions and seeds. For interventional intents, the executor removes a set of bodies and their dependent constraints from the frozen world and re-simulates, producing a matched pair (baseline trajectory, altered trajectory). A discrete readout function is applied across these branches; items are discarded if the readout is non-unique, infeasible, or within sensitive margins (e.g., velocity differences below 0.30 m/s). Otherwise the unique outcome is bound to an option index A–D and rendered as a multiple-choice question, with the Solver seeing only the rendered baseline video.

3. Learning from exact right/wrong. Each item yields a binary reward (1 if the predicted option matches the bound answer), which drives GRPO updates on the shared policy. A two-level curriculum runs on top: at the macro level, per-intent sampling weights follow w_f = ρ_f · max(0.05, 4·ā_f·(1−ā_f)) after an 8-round cold start, concentrating training on intents near the model's competence frontier (ā_f ≈ 0.5); at the micro level, Proposer rewards favor compilable, physically rich, structurally diverse world programs, penalized by anti-copy and saturation controls.

Training configuration: a single run of 100 steps, one-epoch GRPO, learning rate 10⁻⁶, sampling N = 4 intents per step with G = 4 world proposals each, capped at 32 active Solver groups with 4 rollouts per group.

Evaluation: three benchmarks under identical prompts, frame sampling, greedy decoding, and official protocols — PhysBench (9,802 items), ContPhy (1,950 scenarios), and CausalPhys (3,057 real-world video scenarios).

Why This Matters

Impact on research. The paper reframes a data-scarcity problem as an interface-design problem. Its central claim is architectural rather than scale-based: causal ground truth simply cannot be produced by an evaluator that only ever executes one trajectory, and no amount of model capacity or rollout volume can recover labels that lie outside the verification interface. The audit showing graded rewards rising while exact accuracy stays at zero is a concrete, quantified case study of reward hacking in continuous domains, and the ablation results give the field a template for what a verifiable physical reward must look like.

Real-world applications:

• Robotics and embodied agents that must anticipate how a scene changes under intervention (removing an obstacle, changing a grasp point) rather than merely describing what they observe.
• Autonomous driving and safety validation, where counterfactual reasoning about what would have happened under a different action is central to scenario testing.
• Physical simulation and digital twins, where automatically generated, verifiable world variants can serve as training and stress-test curricula.
• Video and media understanding, where models need to judge whether an observed sequence is physically plausible.

Industry relevance. The approach requires no human annotation of intervention outcomes and starts from an open-weight 8B model, making it comparatively cheap to iterate. Any organization already running a simulator can in principle plug its own engine in as the verifier, which fits industrial workflows in robotics, gaming, and simulation-heavy engineering.

Future Directions

• Extending beyond rigid-body templates. Transfers to cloth, fluid, rope, and soft bodies were zero-shot only; whether the same paired-branch pipeline can generate non-rigid worlds in PhysDSL, and whether that improves those categories further, is untested.

• Broadening the physics and intent coverage. The reported run enables only three evidence schemas (Throwing, Collision, Motion) and eight active reasoning intents; scaling to richer phenomena (deformables, multi-contact, fluids) and more intents is an open engineering and scientific question.

• Understanding the Proposer-collapse failure. The ablation shows that freezing the Proposer destroys the policy, and the ContPhy parse rate is reported as 99.90% in the table while the text describes it falling to 0% in later steps — a discrepancy worth resolving in follow-up work, along with the precise mechanism by which world diversity prevents Solver over-adaptation.

• Closing the residual gaps on non-target capabilities. Overall PhysBench gains are modest (+0.55) and the naive baseline actually scores higher overall on CausalPhys (55.8 vs 54.5), so isolating which capability partitions regress and why remains unresolved.

Target Audience

Researchers and graduate students working on multimodal reasoning, embodied AI, and RLVR who are interested in self-improving training loops and verifiable reward design; practitioners who already own a physics or domain simulator and want a recipe for turning it into a label-generating training harness; and evaluation-focused readers who want a concrete, quantified case study of how continuous proxy rewards can silently corrupt a self-play curriculum. Readers without background in reinforcement learning or physics simulation will find the methodology section hard going, though the diagnostic audit in Section 4.4 is broadly accessible.

Authors’ abstract

Physical reasoning underpins how embodied systems understand and act in the physical world, yet vision-language models (VLMs) remain unreliable on questions about dynamics and physical causality. Progress is critically constrained by the scarcity of grounded counterfactual training data, as observational videos capture only the factual trajectory that occurred. The self-evolving paradigm offers a route beyond fixed datasets by enabling models to construct their own training problems. Extending this paradigm to physics, however, requires an external source of truth that can both validate model-proposed environments and verify alternative physical outcomes under intervention. We introduce \physevolve{}, a verifier-grounded self-evolving framework that trains VLMs for causal physical reasoning from model-proposed executable worlds and simulation-authored tasks. At each round, a shared VLM proposes executable world programs and evidence schemas; a deterministic physics engine freezes each valid world, simulates matched baseline and body-elided counterfactual branches, and synthesizes discrete, answer-bound visual tasks. Exact verifier feedback from task solving updates both the shared policy via GRPO and an adaptive curriculum over physical intents. Starting from Qwen3-VL-8B-Instruct, \physevolve{} achieves targeted gains across PhysBench, CausalPhys, and ContPhy without human intervention data, improving PhysBench Throwing by $8.8$ points and ContPhy Counterfactual Removal and Goal-Pit by $7.4$ and $24.4$ points. These gains concentrate on interaction-sensitive and altered-world questions, demonstrating zero-shot transfer of causal principles to unseen complex dynamics (cloth, fluid, soft bodies) and real-world video.