CausalAlign: Causal Alignment of Video Generation Models for Consistent Ego-Motion

AI summary

Overview

Research area: Computer vision, specifically diffusion-based video generation and 3D/geometric consistency, with a downstream application to world action models (WAMs) for robotics.

Technical level: Advanced. The paper assumes familiarity with diffusion models, Diffusion Transformers (DiTs), linear probing, causal intervention, SE(3) camera poses, LoRA fine-tuning, and feed-forward 3D reconstruction models.

Scope: The paper diagnoses where and when camera ego-motion information is represented and lost inside the Wan2.2 video generator, and proposes a three-stage alignment procedure that transfers geometric knowledge from a 3D foundation model into the layers where that information fades.

What This Paper Is About

Diffusion-based video generation models produce visually convincing videos, but the camera's apparent motion (ego-motion) drifts over long sequences, so the implied 3D geometry is inconsistent across frames. The authors first ask how precise ego-motion is represented, used, and retained inside a video generator, then use that answer to strengthen the model's internal ego-motion state without retraining it from scratch or adding extra inputs at generation time.

Key Contributions

1. A diagnosis of "geometric state forgetting." Using linear probes (measuring "decodability" of ego-motion) and causal interventions (measuring "causal effect" on the generated trajectory), the authors show that ego-motion information rises and then falls, both across denoising timesteps and across Transformer layers, meaning the model forms and uses an ego-motion state but fails to retain it late in denoising.

2. CausalAlign, a causal alignment framework. The method targets only the layers and denoising timesteps identified by the diagnosis (layers 14, 10, 12, and 20, with timesteps s ∈ {398, 799} in Wan2.2), rather than aligning 3D features at arbitrarily chosen layers.

3. Delta-token alignment through a 3D foundation model. The authors adapt the idea of delta tokens — compact codes that encode the change between consecutive frames rather than static scene content — and build a teacher delta-token space from a frozen streaming 3D foundation model (LingBot-Map), then map VGM feature transitions into that space with a learnable projector.

4. Demonstrated transfer to embodied models. A VGM adapted with CausalAlign is used to initialize the video expert of Fast-WAM, improving mobile manipulation success on RoboCasa365, showing that better internal ego-motion state benefits downstream action prediction.

Main Findings

• A consistent rise-and-fall pattern: Both probe decodability and causal effect increase at intermediate layers and timesteps and decline afterward, in the frozen Wan2.2 base model. The authors call this "geometric state forgetting."

• Alignment changes the curve shape: After CausalAlign, both signals shift from a rise-and-fall to a rise-and-maintain pattern, indicating the ego-motion state stays influential during late denoising rather than merely being more decodable.

• Improved camera ego-motion: On DL3DV and RealEstate10K evaluation splits, CausalAlign improves all seven reported metrics over Wan2.2 and leads on six of seven compared with all baselines. Reprojection error drops from 8.4 to 6.5 pixels (22.6%), trajectory smoothness rises from 0.71 to 0.88 (23.9%), jerk falls from 0.182 to 0.151, epipolar error falls from 2.9 to 2.1 pixels, and closeness to real camera speeds improves for translation (0.68 to 0.79) and rotation (0.70 to 0.89).

• One metric where a baseline wins: FantasyWorld reports a slightly lower depth–flow consistency error (5.4 versus 5.5 pixels), though CausalAlign leads on the other six metrics without requiring a camera trajectory.

• Visual quality is preserved: VBench-I2V quality moves from 0.805 to 0.806, I2V score stays at 0.958, VBench smoothness rises from 0.983 to 0.985, FVD changes from 182.4 to 182.7, CLIP from 0.311 to 0.314, SSIM from 0.742 to 0.744, and LPIPS is unchanged at 0.183.

• Downstream robotics gains: CausalAlign + Fast-WAM raises average success on RoboCasa365 Target100 from 43.5% to 54.2% (+10.7 percentage points), Atomic Seen from 59.1% to 78.1% (+19.0 points, 703/900 successful rollouts), Composite Seen from 36.4% to 40.8%, and Composite Unseen from 33.2% to 40.6%.

• Structured delta tokens beat unstructured ones: Compared with matched-width PCA projections of equally supervised unpartitioned 128D encoders, the structured 128D delta token (64D ego-motion, 40D geometry, 24D visual) leads all six metrics in that panel.

• Compact learned tokens beat raw 3D features: The 128D delta token outperforms 8192D concatenated LingBot-Map features and 4096D feature differences on all six metrics; rotation error drops from 3.243 to 2.530 degrees versus the 8192D feature pair.

Methodology in Plain English

The work proceeds in two halves: diagnosis, then treatment.

Diagnosis. The authors freeze Wan2.2 and look inside it. For every combination of Transformer layer and denoising step, they attach a simple linear readout that tries to predict the relative camera motion between two consecutive frames, and score how well it does — this is "decodability." Since a readable signal is not proof the model actually relies on it, they also run causal interventions: they mask, shuffle, replace, or inject pose-matched "donor" hidden states at a chosen location while keeping the prompt, initial noise, and random seed fixed, and measure how much the generated camera trajectory changes. Each intervention is compared against a norm-matched random control. Averaging these scores produces layer and timestep profiles, and the locations with the steepest decline become the alignment targets.

Treatment. Alignment happens in three stages. First, a trainable encoder turns the 3D teacher's consecutive geometric states into a compact delta token, trained with auxiliary losses: reconstruct the next teacher state, predict the next token, and decode relative camera pose. The encoder is then frozen. Second, small per-layer projectors learn to map the video model's own feature transitions into that same frozen delta-token space; they are then frozen too. Third, LoRA adapters are inserted only at the diagnosed layers and trained with two losses — a token-matching loss pulling the adapted model's projected transitions toward the teacher's delta tokens, and a pose loss that decodes relative camera motion from those tokens and compares it to ground truth.

At inference nothing extra is needed: no 3D model, no projector, no cache, no camera trajectory. Video generation follows the standard Wan2.2 procedure from a first frame and a text prompt.

Why This Matters

Impact on research: The paper reframes inconsistent camera motion not as missing knowledge but as a retention failure — the model has already learned ego-motion during pretraining and loses it late in denoising. That suggests a family of cheap, inference-free fixes for generative video models, and it introduces a diagnostic toolkit (probe decodability plus norm-matched causal intervention, jointly profiled over layers and timesteps) that other groups can apply to any diffusion backbone. It also connects generative video quality to embodied policy performance, giving a concrete measure of why geometric consistency in a world model matters.

Real-world applications:

• Robotic mobile manipulation: Policies that predict actions while their viewpoint moves benefit from a world model whose implied camera motion is coherent; the paper reports a 10.7-point average success gain on RoboCasa365 Target100.
• Simulation and synthetic training data: More geometrically consistent generated video is more usable as training or replay data where camera trajectories must correspond to physical motion.
• Content creation and virtual production: Text- and image-conditioned generation with stable camera paths reduces the need for manual per-shot fixes or explicit trajectory specification.
• AR/VR and 3D scene capture: Videos whose implied geometry holds together across frames are easier to reconstruct, re-render, or composite into 3D environments.

Industry relevance: The method is a lightweight post-training step on top of an existing open backbone (Wan2.2-14B-I2V) using LoRA, and it adds no runtime cost. That profile fits organizations that already have a video generator in production and want better camera behavior without retraining or deploying a second model.

Future Directions

• Automatic target selection at finer granularity. The current pipeline picks layers and timesteps from decline rates on a diagnostic subset; whether the selection can be made per-head or adaptive during training is unexplored.
• Longer horizons and other backbones. The diagnosis is reported for Wan2.2, and the appendix extends experiments to text-to-video and camera-trajectory-conditioned generation. Whether the same rise-and-fall profile and the same target locations transfer to other architectures (for example Cosmos3) is an open question.
• Combining with explicit control. CausalAlign requires no camera trajectory at inference, while 3D-cache methods do; whether the internal alignment composes additively with trajectory conditioning is not reported.
• Beyond ego-motion. Delta tokens currently carry 64D ego-motion, 40D geometry, and 24D visual components; whether the same teacher-alignment recipe helps with other transient states (object motion, deformation, contact) is untested.

Target Audience

Researchers and engineers working on video diffusion models, 3D-aware generation, or world models for robotics. It is most useful to readers already comfortable with diffusion transformer internals and representation probing who want a practical recipe for improving geometric consistency without inference-time overhead. Roboticists building video-pretrained action policies will find the Fast-WAM transfer results directly relevant, while those new to the area should treat the diagnostic sections as the accessible entry point.

Authors’ abstract

Diffusion-based video generation models (VGMs) can produce visually plausible videos, yet their camera ego-motion often drifts over long horizons. Camera ego-motion determines how the viewpoint changes between frames and is essential for maintaining consistent geometry and trajectories. To address this problem, we propose CausalAlign, a causal alignment framework with two components. First, we diagnose how precise ego motion is represented, used, and retained within VGMs using linear probes and causal interventions, revealing a rise-and-fall pattern during denoising. Second, we strengthen ego-motion state in VGM by aligning the representations at the identified layers to a 3D foundation model through delta-token space. This space encodes the changes between consecutive frames, allowing the alignment to focus on ego-motion dynamics rather than static scene content. With Wan2.2 fine-tuned on DL3DV and RealEstate10K, CausalAlign can largely improve motion accuracy, geometric consistency, and temporal smoothness, while preserving visual quality. For example, it improves geometric consistency by more than 20% relative to the base model. Moreover, with the fine-tuned VGM, CausalAlign improves the success rate of world action models (WAMs) by 10 points on mobile manipulation tasks of RoboCasa365 benchmark. Overall, CausalAlign improves ego-motion consistency in diffusion-based VGMs and yields noticeable gains in embodied models.