Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goals

AI summary

Overview

• Research area: Generative video models used as "world models" for physical reasoning and planning, in computer vision.
• Technical level: Advanced. The paper assumes familiarity with diffusion-based video generation, ControlNet conditioning, Mixture-of-Experts architectures, and video world models for robotics.
• Scope: The paper introduces Goal Force, a task and training framework that conditions video generation on explicit goal force vectors so the model plans the antecedent causal chain of physical interactions needed to produce a specified outcome.

What This Paper Is About

Specifying precise goals for video "world models" is hard: text is too abstract to capture physical nuance, and target images are often infeasible for dynamic tasks. This paper lets users instead declare a goal force, a desired force or motion on a target object, and trains a video model to generate the physically plausible chain of causes (for example a projectile striking that object) that achieves it. The stated aim is to turn the video model into an implicit neural physics planner that needs no external simulator at inference.

Key Contributions

1. A new task and model, Goal Force, which teaches video models to plan a causal chain of physical interactions to achieve a specified goal force. This moves beyond prior direct-force methods and changes how goals can be specified in world models.
2. A training paradigm with a multi-channel control signal encoding goal forces, direct forces, and mass, which teaches the model to act as an implicit neural physics simulator with no simulator required at inference.
3. Demonstrated out-of-domain generalization: despite training only on simple synthetic data (balls, dominos), the model leverages the base video model's prior to generate complex, physically plausible scenarios involving tool use, human-object interaction, and multi-object collisions.
4. Open release of training and evaluation code, model weights, synthetic training data, and benchmark datasets at the project page.

Main Findings

• Goal force beats text-only baselines on adherence: In a 2AFC human study (N = 40 on Prolific) over 75 benchmark scenes, Goal Force was preferred over the zero-shot text-only Wan2.2 baseline on goal force adherence in 73.4% (two-object collision), 72.0% (multi-object collision), 70.5% (human-object interaction), and 74.5% (tool-object interaction) of comparisons. Against the fine-tuned text-only baseline, preferences were 66.9%, 67.0%, 56.8%, and 60.3% respectively.
• Minimal cost to realism and quality: In the same study, Goal Force lost only modest ground on realistic motion and visual quality, with the largest drop in the human-object interaction category (47.5% and 48.9% preference for Goal Force against the zero-shot baseline).
• Visual plans respect physical constraints: On a benchmark of 22 scenes with "natural blockers," where distractor objects are physically constrained from initiating the goal force, accuracy ranged from 54.55% (Pool Scene 2) to 100.00% (Kitchen Lemon and Dominos). The headline Pool scene reached 97.96% accuracy (described in the text as 98%), and a random baseline achieves at most 33.3% given the distractor design. Most failures involved the target moving spontaneously rather than selecting a wrong, constrained initiator.
• Visual plans are diverse, not mode-collapsed: On a six-domino task where any of five preceding dominos can topple the sixth, the model scored 0.6577 on a Jensen-Shannon Divergence-based diversity metric across 26 random seeds, versus 0.3900 for a fully deterministic model and 1.0000 for a uniform distribution over all five dominos.
• Visual plans use privileged mass information: With a fixed goal force magnitude and varied projectile/target masses, the model satisfied all four desired speed-magnitude relationships in an in-distribution scenario, and three of four in an out-of-distribution scenario with different background, viewpoint, lighting, and ball size (the fourth was "very close"). Ball positions were measured with Faster R-CNN, averaging speed over 15 videos per mass combination.
• Direct force capability is competitive with prior simulator-based methods: In a supplementary 2AFC human study (N = 10 on Prolific), Goal Force's first-channel direct force prompting matched or exceeded PhysGen, PhysDreamer, and Force Prompting on force adherence in most benchmark scenes and showed consistently higher visual quality, without using physics simulators or 3D assets at inference.
• Prior methods cannot solve this task: PhysGen, PhysDreamer, and Force Prompting misinterpret a goal force as a direct, non-causal poke on the target, and trajectory-conditioned models such as ToRA follow motion but violate causality, often moving the target before the antecedent event arrives.
• Ablations show each channel matters: Removing the mass channel (relying on text instead) hurt performance relative to the reported mass experiment; masking the direct force channel caused failures on complex out-of-domain causal chains such as human-object interactions. The text prompt does not need to specify the action itself, for example "the dog paw causes a ball to move" works as well as "the dog paw nudges the ball."

Methodology in Plain English

The researchers started from a pretrained open-source video model and taught it a new kind of input. Instead of telling the model "apply this force here," they tell it "I want this force to happen to that object," and require the model to invent the cause.

To communicate forces to the model, they built a three-channel control video that runs alongside the generated clip. One channel carries a direct force as a moving Gaussian blob whose location, angle, and size encode the force vector. A second channel carries the goal force in the same blob format but placed on the target object. A third, optional channel carries mass as a static blob sized proportionally to an object's mass. Forces and masses are not tied to an absolute physical scale; they are normalized within each synthetic dataset so the model learns relative notions like "small poke" versus "large poke."

Training data came from simple synthetic scenes: 3k Blender domino videos, 6k Blender rolling-ball videos (4.5k collisions and 1.5k misses), and 3k PhysDreamer carnation videos, totaling simple causal primitives. The key trick is randomly masking the causal information: each training video supplies either the direct force or the goal force, never both, and the mass channel is also randomly masked. That omission forces the model to learn reasoning in both directions, from goal to cause and from action to outcome, and to fall back on its own learned physical prior when privileged data is missing.

Architecturally, they fine-tuned a ControlNet module on the Wan2.2 Mixture-of-Experts diffusion model, cloning and training the first 10 DiT layers of the high-noise expert and feeding them into the frozen base model through zero-convolutions. Only that expert was fine-tuned, on the reasoning that it handles global structure and low-frequency dynamics. Training used 3,000 steps, an effective batch size of 4 (one per device on four NVIDIA 80GB A100s), finished in under 48 hours, and used 81-frame videos at 16 FPS for both training and inference. Evaluation combined a curated 75-scene benchmark, human preference studies, physical-accuracy tests with occluders, a diversity metric, and mass-sensitivity measurements.

Why This Matters

The paper argues that grounding video generation in fundamental physical interactions can let models emerge as implicit neural physics simulators, enabling precise, physics-aware planning without relying on an external engine. It reframes how goals are specified in world models: not as abstract text or pixel-exact target images, but as forces and intermediate dynamics that mirror how people actually think about physical tasks.

Real-world applications suggested by the work:

• Robotics planning: Goal forces can be supplied as an additional interface to visual planners such as UniPi and Adapt2Act, which translate predicted visual plans into robotic actions with inverse dynamics models.
• Tool use: The model generalizes zero-shot to tool scenarios, for example inferring how to use a golf club to impart a desired force on a ball, or picking up a rose by its stem rather than its petals.
• Fine-grained interaction specification: The mass channel offers an interface for supplying object-level physical properties when they are known, while the model can otherwise estimate them from appearance.
• Simulator-free physical reasoning: Because no physics simulator or 3D asset is needed at inference, unlike PhysGen and PhysDreamer, the approach is a candidate for settings where 3D geometry or external engines are unavailable.

Industry relevance centers on interactive world models and content generation, where users want to specify outcomes rather than manipulate scenes, and on any pipeline that needs to predict how a physical action will propagate through objects.

Future Directions

• Scaling beyond simple causal primitives: The model was trained only on dominos, rolling balls, and a single flower; how far the zero-shot generalization extends to other physical regimes remains an open question.
• Reducing failure modes: Most benchmark failures came from the target object moving spontaneously rather than choosing a wrong initiator, and some scenes (Pool Scene 2 at 54.55%, Duckie Scene 2 at 64.86%) lag far behind others, leaving room for improvement.
• Better use of privileged physics: The out-of-distribution mass experiment satisfied only three of four desired speed relationships, suggesting mass reasoning is not yet fully robust to domain shift.
• Extending to more control modalities and embodiments: The authors treat specifying interaction source and type as a desirable feature for robotics, implying future work on richer interfaces and tighter integration with downstream action models.

Target Audience

Researchers working on video generation, world models, and physically grounded simulation; robotics and planning researchers interested in goal specification beyond text; and practitioners building interactive generative systems who need precise, causal control over object interactions. The paper is most useful to readers already comfortable with diffusion models and control-signal conditioning, since it does not introduce those foundations.

Authors’ abstract

Recent advancements in video generation have enabled the development of ``world models'' capable of simulating potential futures for robotics and planning. However, specifying precise goals for these models remains a challenge; text instructions are often too abstract to capture physical nuances, while target images are frequently infeasible to specify for dynamic tasks. To address this, we introduce Goal Force, a novel framework that allows users to define goals via explicit force vectors and intermediate dynamics, mirroring how humans conceptualize physical tasks. We train a video generation model on a curated dataset of synthetic causal primitives-such as elastic collisions and falling dominos-teaching it to propagate forces through time and space. Despite being trained on simple physics data, our model exhibits remarkable zero-shot generalization to complex, real-world scenarios, including tool manipulation and multi-object causal chains. Our results suggest that by grounding video generation in fundamental physical interactions, models can emerge as implicit neural physics simulators, enabling precise, physics-aware planning without reliance on external engines. We release all datasets, code, model weights, and interactive video demos at our project page.