Can Vision Language Models Learn Intuitive Physics from Interaction?

AI summary

Overview

Research area: Vision language models (VLMs), intuitive physics, reinforcement learning post-training, and cognitively inspired machine learning.

Technical level: Intermediate. The paper is readable without deep RL background, but understanding it fully requires familiarity with supervised fine-tuning, policy-gradient methods (GRPO/GSPO), parameter-efficient fine-tuning (LoRA/QLoRA), and linear-probe decodability analysis.

One-sentence scope: The paper tests whether training VLMs through interaction with a simulated block-tower environment produces generalizable intuitive physics, compared against non-interactive supervised fine-tuning.

What This Paper Is About

Pre-trained VLMs lack good intuitions about the physical world, and prior work showed that supervised fine-tuning improves performance on specific physical tasks without producing rules that transfer to new contexts. Drawing on cognitive science claims that humans build robust physical understanding by actively interacting with their environment, the authors ask whether reinforcement learning from interaction gives VLMs generalizable intuitive physics. They train models to build and judge block towers inside a physics simulator and then test whether those models transfer to related tasks and to real photographs of block towers.

Key Contributions

1. A first systematic test of interaction for intuitive physics in VLMs. The authors operationalize "interaction" as one-step reinforcement learning with a defined environment, action space, and reward function, and compare Group-Relative Policy Optimization (GRPO) against Supervised Fine-Tuning (SFT) as the non-interactive baseline.

2. A task suite spanning two synthetic datasets and three action types. They build top block and side block datasets of 256×256 pixel RGB images in the ThreeDWorld environment, with towers of 2–4 randomly colored cubes, and define binary stability, x-only, and x-y tasks (four dataset-and-task combinations total), plus an external evaluation set of 100 real images from Lerer et al. (2016).

3. A decodability analysis separating competence from performance. They probe layer-wise model activations for physical quantities such as binary stability and x-offset to test whether models hold the relevant information internally even when their textual outputs fail.

4. A broad ablation program. They repeat experiments with Qwen2.5-VL-7B and Qwen3-VL-32B, swap GRPO for Group Sequence Policy Optimization (GSPO), extend training to 48,000 steps, test multi-task and sequential training, and add a multi-step triplet version of the x-y side block task.

Main Findings

• Both methods reach near-ceiling within-task performance. On the binary stability top block task, GRPO reaches a mean test accuracy of 0.943 and SFT reaches 0.969 (ceiling is 1; for the other tasks the ceiling is 20). On x-only top block, GRPO scores 19.999 and SFT 20. On x-only side block, GRPO scores 19.998 and SFT 20. On x-y side block, GRPO scores 17.313 and SFT 19.86. The authors conclude there is no direct benefit of interaction for within-task improvement.

• No consistent advantage of interaction for generalization. Across the transfer matrix (Table 1 and Figure 2), no model reliably generalizes to all related tasks, regardless of whether it was trained through interaction or with SFT.

• Generalization is confined to closely related tasks and datasets. Training on x-y side block carries over to x-only side block (GRPO mean reward 5.396, or 12.772 when only legal answers are counted; SFT 19.316), which the authors attribute to x-only being a subset of x-y. Limited carry-over also appears between binary stability top block and x-only top block, which share the x-offset variable. In Table 1, binary-stability evaluation of models trained on x-only top block, x-only side block, and x-y side block gives GRPO values of 0.788, 0.503, and −0.264, and SFT values of 0.624, 0.596, and 0.506. The prose description of this comparison gives 0.624, 0.503, and −0.264 for GRPO and 0.624, 0.596, and 0.506 for SFT, which differs from the corresponding Table 1 entries; the paper content does not reconcile the two.

• Transfer to real images is weak and below human level. On 100 real wooden-block images from Lerer et al. (2016), GRPO models trained on binary stability top block, x-only top block, x-only side block, and x-y side block score 0.6, 0.57, 0.52, and 0.31; the SFT models score 0.59, 0.53, 0.55, and 0.57. The authors state all models perform below the human average (computed from publicly available, anonymized data in Schulze Buschoff et al. (2025a)); the numeric value of that human average is not reported in the provided content.

• The information needed for the tasks is already encoded in the models. Binary stability is trivially decodable from the base model's activations and changes little after either fine-tuning method, likely because stability can be read off a small set of pixels along the horizontal center line. The same holds for the x-offset of the top block. Linear probes outperform the base model's behavioral accuracy, which the authors interpret as evidence of shortcut learning rather than missing knowledge.

• Attention maps show no reliable change. Comparing post-trained models to the base model, the attention maps are noisy and do not reliably show differences from either post-training method.

• More training does not unlock generalization. Training Qwen2.5-VL-7B up to 48,000 GRPO steps causes the model to overfit to the specific reward function beyond 10,000 steps.

• Interaction does help with retaining sequential tasks, in one setting. When Qwen2.5-VL-7B is trained with GRPO first on x-only side block and then on binary-stability top block, it retains both tasks but still does not generalize to x-only top block. An SFT model trained in the same blocked manner quickly degrades on the first task, while an SFT model with interleaved data performs reasonably well on both post-training tasks.

• Bigger models and a different RL algorithm do not help. Qwen3-VL-8B trained with GSPO performs well on its post-training task and shows a similar generalization pattern to binary stability as the GRPO model, but does not generalize to tasks such as x-only side block. The 32B model trained with either GRPO or GSPO shows no meaningful generalization to any condition, while the 32B SFT model generalizes somewhat from x-only top block to binary-stability top block.

• Multi-step interaction does not fix the problem. Qwen3-VL-8B learns the triplet version of x-y side block well after 10,000 GRPO steps, but does not generalize to the single-image version of the same task, and the single-image model does not generalize to the triplet version, despite both sharing the same visual statistics and reward function.

Methodology in Plain English

The authors create a simple physical world: a camera at a fixed angle photographs stacks of 2–4 colored cubes in the ThreeDWorld simulator. In the top block dataset, one block sits on top of the tower but shifted left or right; in the side block dataset, the misplaced block sits on the floor beside the tower. Camera angle and block sizes are held fixed so the models can learn the mapping from pixels to real distances.

From these images they define tasks. In binary stability, the model judges whether a tower is stable. In x-only, it outputs one integer specifying how far to move the block along the horizontal axis to improve stability. In x-y, it outputs two integers, moving the block sideways and upward. Rewards are Gaussian functions of distance from the optimal position, with a weak reward (2·e^(−d²)−2) for unstable outcomes and a strong one (20·e^(−d²)) for stable towers; unparseable answers get −5, and in the x-y task answers that move a block below the floor get −4.

They fine-tune an 8B parameter 4-bit quantized Qwen3-VL using the unsloth library and QLoRA adapters (rank r = α = 16, inserted at every layer). In the interactive condition, models are trained with GRPO: each prompt generates 16 responses, rewards are normalized into advantages, and the policy is updated with gradient ascent (the original KL term is dropped, following common practice). In the non-interactive condition, models are trained with standard supervised fine-tuning on token-level cross-entropy loss over the same data. Both conditions use 10,000 training images per dataset-and-task combination, are evaluated on a separate 10,000 images, and are trained for 10,000 steps on single 80GB A100 GPUs. To move beyond black-box accuracy, the authors also train linear probes on activations across layers to check whether physical quantities are internally represented.

Why This Matters

The paper is a negative result with a clear message: adding interaction through RL post-training did not give these VLMs generalizable physical intuitions, and the models appear to learn task-specific shortcuts instead. That constrains the current narrative that online RL is broadly more robust than supervised fine-tuning, at least for this class of visual physical reasoning problems. It also suggests that parameter-efficient adaptation on narrow cognitive tasks may be insufficient for building machines that reason about the world like humans, and it points to representation-level explanations: the physical variables already exist in the model, but they are not recruited for new tasks.

Real-world applications affected:

• Robotics and manipulation — systems that must stack, place, or rearrange objects need physical intuitions that carry over from simulation to new objects and scenes, exactly the transfer the paper finds missing.
• Embodied and interactive agents — any agent expected to learn dynamics from acting in an environment, rather than from curated demonstrations.
• Safety-critical perception — judging whether a structure or arrangement is stable (construction, warehousing, logistics) requires reliable prediction of physical outcomes, not task-specific shortcuts.
• Benchmarking and evaluation design — the decodability analysis offers a template for distinguishing a model that lacks knowledge from one that has the knowledge but cannot use it, useful for anyone building evaluation suites for physical reasoning.

Industry relevance: The findings bear on how much value practitioners can expect from RL versus SFT post-training, on whether scaling alone fixes physical reasoning (the 32B model did not generalize), and on the practical costs of pipeline choices such as 4-bit quantized PEFT versus full fine-tuning.

Future Directions

• Scale and diversity. The authors note they studied only three model families of sizes 7B, 8B, 11B, 12B, and 32B in constrained single settings, and call for testing whether the conclusions hold for larger models, more data, and more varied tasks.
• Longer-horizon interaction. Because only single-step (and one triplet) interactions were tested, it remains possible that interaction's advantages only appear over long state-action sequences; the authors propose practical methods for testing this with modern VLMs.
• Full fine-tuning. All experiments used 4-bit quantized PEFT; future work should check whether the results hold for full fine-tuning.
• Auxiliary objectives and curricula. The multi-task GRPO result (a model trained sequentially on two tasks retained both) is described as a preliminary but encouraging signal, and the decodability findings suggest auxiliary objectives that force models to ground predictions in underlying physical variables rather than surface-level shortcuts. Curriculum-based approaches that gradually introduce more complex physical scenarios are also suggested.

Target Audience

Researchers working on vision language models, physical and causal reasoning, and embodied AI; reinforcement learning practitioners interested in whether online RL beats supervised fine-tuning for visual reasoning; and cognitive scientists studying intuitive physics who want to see how developmental hypotheses about learning-by-interaction are being tested in artificial systems. It is also useful for engineers who need a realistic assessment of what post-training can and cannot deliver for physical reasoning tasks.

Authors’ abstract

Pre-trained vision language models do not have good intuitions about the physical world. Recent work has shown that supervised fine-tuning can improve model performance on simple physical tasks. However, fine-tuned models do not appear to learn robust physical rules that can generalize to new contexts. Based on research in cognitive science, we hypothesize that models need to interact with an environment to properly learn its physical dynamics. We train models that learn through interaction with a simulated environment using reinforcement learning. While learning from interaction allows models to improve their within-task performance, it fails to produce models with generalizable physical intuitions. We find that models trained on one task do not reliably generalize to related tasks, even if the tasks share visual statistics and physical principles, and regardless of whether the models are trained through interaction.