Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms

AI summary

Overview

• Research area: Mechanistic interpretability of video diffusion models, specifically the internal attention mechanisms of text-to-video (T2V) Diffusion Transformers and their effect on physical plausibility.
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow matching, DiT architectures, cross-attention, self-attention, and Rotary Position Embedding (RoPE).
• Scope: One sentence: The paper traces physically implausible video generations back to excessive spatial attention decay caused by RoPE in self-attention, and fixes it with a lightweight frequency-scaling modification.

What This Paper Is About

State-of-the-art video diffusion models produce visually impressive videos that nonetheless break real-world physical laws, and existing fixes rely on external physical priors, prompt rewriting, or specialized training data rather than addressing the root architectural cause. The authors open the "black box" and study what they call the "motion planning" process of T2V diffusion models — how semantic information enters the latent and gets allocated to different frames to position an object. Their goal is to identify which internal components decide an object's trajectory, why those decisions sometimes produce physically impossible motion, and whether a minimal architectural change can improve physical commonsense.

Key Contributions

1. The first interpretability study of motion planning in text-to-video diffusion models, together with a practical analytical framework and toolkit (code released at https://github.com/Siriuslala/physics).
2. A mechanistic extension of the "first shape, then details" finding in reverse diffusion: the paper shows at the level of individual attention heads and denoising steps how the model actually forms the "shape," rather than only observing that early steps fix the layout.
3. Identification of a hidden flaw in self-attention — RoPE-induced excessive spatial attention decay (the "spatial anchoring effect") that lets early candidate regions prematurely lock into physically implausible positions and suppress reasonable trajectories in adjacent frames.
4. A lightweight RoPE modification that scales the frequency of RoPE differently across denoising steps, validated in both training-free and training-based (LoRA) settings on VideoPhy, and combinable with prompt refinement.

Main Findings

• Motion planning happens in the first few denoising steps. In layer 27 of Wan2.1-T2V-1.3B, the object's position per frame moves from random noise (T1) to multiple highlighted regions along the motion trajectory (T3), converges toward the final position around step 5 (out of 50 steps), and shows a clear trajectory by T7.
• Two metrics quantify this convergence. Attention entropy and support quality both show a distinct turning point around denoising step 5 in layer 15, marking the emergence of the trajectory pattern.
• Not all heads with a clean trajectory pattern matter. Among the heads of layer 15, L15H2, L15H5, and L15H0 focus on the trajectory, whereas L15H1 keeps low attention entropy but its support quality remains 0, meaning its attention distribution is not aligned with the trajectory.
• Cross-attention heads fall into four types based on convergence speed versus head contribution (measured at denoising step 2): (1) convergence speed < 0.1 and contribution < 0.5; (2) convergence speed < 0.1 but contribution > 0.5; (3) convergence speed > 0.1 and contribution > 1.0; (4) convergence speed > 0.1 but very low contribution.
• Only Type (3) heads actually drive motion. Zero-ablating Type (3) causes obvious collapse of the object's trajectory, while ablating Type (1), Type (2), or Type (4) leaves the overall motion essentially unchanged. A high contribution score alone does not mean a head governs the trajectory, and a clear trajectory pattern is not a sufficient condition for a head to be responsible for motion planning.
• Self-attention exhibits a "spatial anchoring effect." Rather than showing clear trajectories, most self-attention heads focus on the same spatial position across different frames, which the authors attribute to RoPE's long-range decay along the height and width dimensions.
• Candidate regions compete unstably early on. In a failed case (seed 20, where the basketball bounces in the air and then stops), multiple candidate regions compete in the first 7 steps with no stable advantage, a few frames (4, 8, and 10) determine the object's position earlier, and an early dominant, physically reasonable region can be suppressed later — region K1 in frame 14 dominates at step 2 but is "robbed" by region K3 at step 3 (K1: 0.21 vs. K3: 0.33).
• Early anchoring can override physical priors. Because region K1 in frame 10 stabilizes early at step 3 with a confidence of 0.63, spatially adjacent regions in other frames are steadily raised in confidence, letting K3 in frame 14 surpass K1 and producing the failure mode where the basketball remains stationary in the air.
• Modified RoPE improves VideoPhy scores. On the 343-case VideoPhy benchmark, Wan2.1-T2V-1.3B with modified RoPE reaches Overall SA 57.43 / PC 39.94 versus 53.64 / 29.45 for the base model. Adding prompt refinement gives 86.30 / 58.89 (versus 80.47 / 44.31 for prompt refinement alone). LoRA + modified RoPE + PR reaches 87.46 / 62.68, the highest overall in the table.
• Gains concentrate on solid interactions. The advantage appears mainly in the solid-* subsets, which contain more large-magnitude motions than the fluid-fluid subset, and are the focus of the method.
• Prompt refinement and the proposed method are complementary. Physical consistency gains from prompt refinement mainly come from enhanced instruction following, while the RoPE modification boosts physical consistency further when combined with it.
• Training generalizes across seeds. Although training used a fixed random seed of 42, the trained model improved motion trajectories generated by other seeds with the same λ values.
• Attention, not FFN, is where the failures live. For LoRA, tuning only the attention modules gave better results without degrading aesthetics, supporting the completeness of the attention-focused interpretability analysis.
• The 1.3B model was chosen deliberately. The authors use Wan2.1-T2V-1.3B rather than 14B because they find the larger model does not perform better in physics.

Methodology in Plain English

• Pick a trackable case and model. The authors use a prompt about a basketball falling vertically onto a wooden floor and bouncing, with a default random seed of 26 and 50-step inference, all on an A800 GPU. The studied backbone is the DiT-based Wan2.1-T2V-1.3B. They study the conditional branch of classifier-free guidance by default.
• Watch cross-attention over time, since it is the only source of video semantics. They visualize cross-attention from video latents to the object token step-by-step and layer-by-layer, then quantify the evolution with two metrics: attention entropy (normalized spatio-temporal entropy of the attention distribution) and support quality (how much attention mass falls inside the object's final trajectory region, the "reference trajectory").
• Separate head function into two independent measurements. Convergence speed is the mean support quality over the first 10 denoising steps; head contribution is measured by an attribution-patching method inspired by causal intervention, where a head is ablated and the change in the predicted velocity inside the reference area is quantified. Because this metric is only an estimate, they also zero-ablate heads and inspect the generated videos directly.
• Analyze self-attention to explain the cross-attention outcome. They extract candidate regions per frame from head-averaged cross-attention at each layer and step, define an anchor distance to the final reference region, and use a mutual consistency metric based on normalized self-attention intensity to measure how confidently two candidate regions select each other.
• Fix the flaw by rescaling RoPE frequencies. For height and width dimensions, modulation factors λ^h, λ^w < 1.0 are applied to the RoPE rotation exponents, reducing the rotation angle and hence the decay rate. In the main experiments a preset λ^h/λ^w is applied during the first 5 denoising steps.
• Validate training-free and with fine-tuning. Systematic evaluation uses the 343-case VideoPhy benchmark with Semantic Adherence (SA) and Physical Commonsense (PC) metrics across solid-solid, solid-fluid, and fluid-fluid subsets. Training uses WISA, described as 80K human-curated videos of 17 physical laws, with a custom timestep sampler that samples the first 10% of denoising timesteps with high probability p^early. λ is slowly increased from 0 to a default of 0.75, and the model is fine-tuned via LoRA at batch size 32 and learning rate 1e-4.
• Baselines. Training-free comparisons use the base model and prompt refinement; training-based comparisons use LoRA and VideoREPA, which uses an external video foundation model for guidance during fine-tuning.

Why This Matters

• It reframes physical inconsistency as an architectural artifact rather than a data or prompting problem. Instead of adding physics simulators, curated datasets, LLM prompt refinement, physics experts, or memory modules, the paper locates the cause in how RoPE shapes self-attention and shows a scaling factor can move the needle.
• It gives the interpretability community a concrete handle on video generation. The paper claims the first interpretability study of motion planning in T2V diffusion models, extending "first shape, then details" from an empirical observation to a mechanistic account centered on specific attention heads.
• It points to a lightweight, scalable intervention. The modification adds no external model and preserves the model's native scaling, in contrast to task-specific add-ons that introduce extra computational overhead and limited flexibility.

Real-world applications implied by this direction:

• Content creation and visual effects: fewer retakes when generated clips must show plausible falling, bouncing, or colliding objects.
• World models and simulation: the paper frames video generation models as a "world simulator," and physical consistency is a prerequisite for that role.
• Robotics and embodied AI training data: physically coherent synthetic video is more usable as training or planning material.
• Education and scientific visualization: demonstrations of mechanics such as free fall, elastic collision, and parabolic motion require trajectories that obey the laws they are meant to illustrate.

Industry relevance: any pipeline built on open T2V backbones such as Wan2.1 benefits from a change that can be applied at inference in the first 5 denoising steps or baked in with LoRA fine-tuning, and the code is publicly released.

Future Directions

• Extend to image-to-video (I2V). The authors state that the findings can be extended to broader scenarios such as I2V, which they leave for future work.
• Test other architectures. They specifically name few-step autoregressive diffusion models as architectures worth exploring.
• Improve how λ^h/λ^w is parameterized. The paper reports that the training-free setting on the "basketball free fall" case still requires manual tuning of λ^h/λ^w, that some seemingly reasonable parameterizations performed poorly, and that adaptive adjustment of λ^h/λ^w is discussed in the appendix rather than solved.
• Broaden beyond solid dynamics. The analysis focused on solid dynamics because it provides easily trackable motion trajectories; whether the same self-attention flaw and fix generalize to fluid and thermodynamic phenomena remains open.
• Verify on larger models. The main analysis uses Wan2.1-T2V-1.3B because the larger model does not perform better in physics; validation on the 14B model is presented in the appendix.

Target Audience

• Interpretability and mechanistic-analysis researchers working on diffusion and video generation models.
• Video generation practitioners and engineers who want a low-cost architectural change or a fine-tuning recipe to improve physical plausibility.
• Researchers in physics-aware generation and world models who want to know why prior data- and prior-based approaches may not address the root cause.
• Graduate students and advanced readers comfortable with attention mechanisms, RoPE, and flow matching, looking for a worked example of linking mechanistic findings to an architectural fix.

Authors’ abstract

Despite impressive visual quality, state-of-the-art video diffusion models often generate content that violates real-world physical laws. While existing solutions rely on external priors or specialized data, we investigate the root cause by exploring the internal mechanisms of these models. Specifically, we present the first interpretability study on the ''motion planning'' process of text-to-video diffusion models, revealing how motion trajectories form during early denoising stages. Building upon the ''first shape, then details'' finding, we combine cross-attention trajectory patterns with causal head contributions to identify a specific subset of attention heads driving motion planning. Further, our self-attention analysis shows that Rotary Position Embedding (RoPE) induces excessive spatial attention decay. This causes early candidate regions to prematurely lock into physically implausible positions, suppressing reasonable trajectories in adjacent frames and triggering generation failure modes. To address this fundamental flaw, we propose a lightweight architectural modification that scales the frequency of RoPE across different denoising steps. This strategy reduces excessive attention decay, helping the model explore better candidate regions to establish coherent physical motion. Finally, training-free and training-based experiments confirm the effectiveness of our approach in enhancing the physical commonsense of generated videos.