PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation

AI summary

Overview

Research area: Text-to-video (T2V) generation, specifically post-training a pretrained video model to produce physically plausible motion and object interactions using preference optimization.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, direct preference optimization (DPO), LoRA adapters, and vision-language model (VLM) scoring.

Scope: The paper introduces a data pipeline (PhyAugPipe), a dataset (PhyVidGen-135K), and a preference-optimization framework (PhyGDPO) that together improve the physical consistency of a 14B-parameter text-to-video model beyond what closed-source systems like Sora2 and Veo3.1 achieve on physics benchmarks.

What This Paper Is About

Text-to-video models now produce visually convincing footage, but they routinely violate basic physics: bodies deform, balls pass through rackets, and objects fail to shatter or float correctly. Existing fixes either rely on graphics simulators (which cannot handle real-world complexity) or ask a language model to rewrite prompts with physics instructions (which outsources reasoning to a component that is itself unreliable at physics). This paper targets the underlying training signal, arguing that the real problem is the absence of negative examples and holistic preference structure during post-training. The goal is a principled preference-optimization method that uses real-world video as ground truth for correctness and concentrates learning on the physics cases the model handles worst.

Key Contributions

1. A groupwise DPO framework (PhyGDPO) built on the Plackett-Luce probabilistic model. Instead of comparing one winning video against one losing video (the Bradley-Terry assumption of vanilla DPO), the method compares a real-world video against a group of generated candidates, capturing a holistic preference signal over many samples at once.

2. Physics-Guided Rewarding (PGR) and LoRA-Switch Reference (LoRA-SR). PGR uses a physics-aware VLM (VideoCon-Physics) to score each candidate's semantics adherence and physics commonsense, then modulates the loss so that physics-violating samples exert stronger influence. LoRA-SR freezes the base model as the reference and attaches switchable LoRA adapters as the trainable "action" model, avoiding a full duplicate model in GPU memory.

3. A physics-augmented data construction pipeline (PhyAugPipe) and the PhyVidGen-135K dataset. A VLM (Qwen2.5-72B-Instruct) with a designed chain-of-thought rule parses entities, actions, forces, and outcomes from one million text-video pairs, scores physics richness, and retains 135K pairs. Action clustering and difficulty-weighted resampling then reduce this to 17K training pairs.

4. State-of-the-art results on physics benchmarks plus a human user study. PhyGDPO outperforms Sora2, Veo3.1, VideoDPO, and PhyT2V on VideoPhy2 and PhyGenBench, and is preferred by human raters in 64 to 94 percent of head-to-head comparisons depending on the baseline.

Main Findings

• VideoPhy2 overall score: PhyGDPO reaches 0.1627, ahead of Veo3.1 (0.1525), Sora2 (0.1508), PhyT2V (0.1492), and VideoDPO (0.1373). On the "hard action" track it scores 0.0500, which is roughly 4.5 times the base model Wan2.1-14B (0.0111), 29 percent above Sora2, and 13 percent above Veo3.1.
• PhyGenBench average: PhyGDPO scores 0.55, edging VideoDPO (0.54) and PhyT2V (0.50), with the largest gains on the mechanics (0.55 vs 0.48) and thermal (0.58 vs 0.47) tracks. It performs worse than VideoDPO on the material track (0.47 vs 0.58).
• Human preference: With 104 participants completing 48 trials each, PhyGDPO is preferred 94.2 percent of the time over Vcrafter2, 89.4 percent over VideoDPO, 88.5 percent over PhyT2V, 86.5 percent over Wan2.1-14B, 82.7 percent over Hunyuan, 67.3 percent over Sora2, and 64.4 percent over Veo3.1.
• Data pipeline ablations: Each PhyAugPipe component adds measurable gains on VideoPhy2 overall: baseline 0.1475, plus chain-of-thought filtering 0.1525, plus action clustering 0.1575, plus physics rewarding 0.1627.
• Optimization ablations: Stacking PhyGDPO components on Wan2.1-14B moves the overall score from 0.1288 (baseline) to 0.1458 (+LoRA-SR), to 0.1559 (+groupwise model), to 0.1627 (+physics rewarding). On the smaller Wan2.1-1.3B, PhyGDPO (0.1407) beats Flow-DPO (0.1283) and VideoDPO (0.1305) under identical training settings.
• Memory and storage efficiency: LoRA-SR uses 25.3 GB of GPU memory and 84 MB of storage versus 48.7 GB and 5.3 GB for a full-copy reference baseline, while also scoring higher (0.1407 vs 0.1373 overall).
• Robustness to scorer choice: Swapping the evaluation VLM to Gemini-2.5-pro still ranks PhyGDPO first (0.5525), ahead of Sora2 (0.5373) and Veo3.1 (0.5034).
• Transfer to other base models: Applied to Vcrafter2 instead of Wan2.1, PhyGDPO improves the overall score to 0.1426, above Flow-DPO (0.1128) and VideoDPO (0.1373).

Methodology in Plain English

The approach has three stages.

First, build better training data. The researchers take a large pool of high-quality text-video pairs and ask a vision-language model, guided by a step-by-step reasoning script, to identify clips containing rich physical interaction. The script has the model list the objects, materials, actions, and forces involved; check its own list against the actual frames to remove hallucinations; explain how the objects interact and what results; and assign a 0-to-1 physics-richness score. Clips above a threshold (0.60) are kept, yielding 135K pairs. Those pairs are then grouped into action categories using a sentence-embedding model, and a separate physics-aware VLM scores representative videos per category. Categories where the model performs poorly get sampled more heavily, using an exponential weighting on a per-category difficulty value, so the final training set of 17K pairs is skewed toward hard physics.

Second, change the preference objective. Real-world video always obeys physics, so it is used as the "winning" example, while several generated videos from the base model serve as "losing" examples. Rather than a single pairwise comparison, the loss is derived from the Plackett-Luce model, which assigns a probability to the real video being ranked above the entire group of generated ones. A chain of inequalities (Jensen's inequality, plus an exponential-product bound) reduces this group-level objective to an efficient form that requires only one sampled candidate and one timestep per training iteration. The derivation is worked out for rectified flow matching, so the log-likelihood ratio becomes a difference in velocity-prediction errors between the trained and reference models.

Third, make the loss attention-aware. The VLM scores for each generated candidate determine how strongly that candidate influences the gradient. Candidates with low physics scores get amplified weight, and a sharpness parameter controls how decisively the comparison is made. This is the Physics-Guided Rewarding component.

Finally, for efficiency, the base model is frozen and used directly as the reference, with LoRA adapters attached to the attention projection layers serving as the trainable model. A simple binary flag toggles the adapters on or off, so the same weights can act as either the reference or the action model. This avoids storing two full copies of a 14B model and also keeps the trained model from drifting too far from its starting point.

Why This Matters

Impact on research. The paper reframes physical consistency in video generation as a preference-optimization problem rather than a data-scale or prompt-engineering problem. It provides a concrete derivation showing how groupwise ranking losses and VLM-derived rewards can be adapted to flow-matching video models, and it demonstrates that using real video as the winning case supplies a correctness guarantee that generated video cannot. The 135K-pair dataset and the 17K hard-action subset give the community a reusable resource for studying physics in T2V.

Real-world applications:

• Video games and interactive media: physically coherent generated footage could reduce reliance on hand-authored animation and physics engines for cutscenes or procedural content.
• Autonomous driving simulation: training and testing perception systems requires synthetic video where vehicles, pedestrians, and objects behave according to real dynamics.
• Robotics: simulators and demonstration generators benefit from models that produce physically valid interactions rather than plausible-looking but impossible motion.
• Film and visual effects: previsualization and effects shots often require convincing collisions, shattering, fire, and fluid behavior before expensive production begins.

Industry relevance. The method is a post-training recipe applied on top of an existing open model, requiring roughly six days on eight H100 GPUs, and it produces results that beat two major closed-source systems on physics-specific benchmarks. That combination of modest compute and competitive output matters for any organization that wants to improve a video model without retraining from scratch, and the LoRA-SR scheme cuts the memory cost of preference optimization by roughly half.

Future Directions

• Extending beyond the current physics domains. Evaluation covers 27 physical laws across mechanics, optics, thermal, and material domains, but the method's material-track score actually trails VideoDPO. Whether the pipeline scales to fluids, cloth, and multi-object contact remains open.
• Reducing reliance on VLM judges. The training reward comes from VideoCon-Physics and evaluation from a different VLM, VideoPhy2-AutoRater. The authors explicitly note that VLM assessments of physics may be imperfect, so more direct or simulation-grounded reward signals are a natural next step.
• Clarifying the role of dataset scale versus objective design. The paper ablates pipeline components and optimization components separately, but the interaction between data volume, action-category balance, and the groupwise loss is not fully mapped out.
• Applying the framework to longer and higher-resolution video. Training and inference in this work are at 480×832, and physics violations tend to compound over time; whether PhyGDPO's advantage holds for long-duration generation is untested.

Target Audience

Researchers and engineers working on video generation, generative model post-training, or reinforcement learning from human or AI feedback. The paper is most useful to readers who already understand diffusion or flow-matching models and want a concrete template for applying preference optimization with VLM-derived rewards. Practitioners building video generation systems or physical simulators will find the data pipeline and LoRA-based memory savings directly actionable, while those focused on benchmark design will find the PhyGenBench and VideoPhy2 comparison and the accompanying user study instructive.

Authors’ abstract

Recent advances in text-to-video (T2V) generation have achieved good visual quality, yet synthesizing videos that faithfully follow physical laws remains an open challenge. Existing methods mainly based on graphics or prompt extension struggle to generalize beyond simple simulated environments or learn implicit physical reasoning. The scarcity of training data with rich physics interactions and phenomena is also a problem. In this paper, we first introduce a Physics-Augmented video data construction Pipeline, PhyAugPipe, that leverages a vision-language model (VLM) with chain-of-thought reasoning to collect a large-scale training dataset, PhyVidGen-135K. Then we formulate a principled Physics-aware Groupwise Direct Preference Optimization, PhyGDPO, framework that uses real-world video as winning case to guarantee correct physics learning and builds upon the groupwise Plackett-Luce probabilistic model to capture holistic preferences beyond pairwise comparisons. In PhyGDPO, we design a Physics-Guided Rewarding (PGR) scheme that leverages VLM-based physical rewards to direct the optimization to focus on challenging physics cases. In addition, we propose a LoRA-Switch Reference (LoRA-SR) scheme that avoids full-model duplication as reference for efficient DPO training. Experiments show that our method significantly outperforms state-of-the-art open-source methods on PhyGenBench and VideoPhy2. Please check our project page at https://caiyuanhao1998.github.io/project/PhyGDPO for more video results. Our code, data, and models are publicly available at https://github.com/caiyuanhao1998/Open-PhyGDPO