VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generation

AI summary

Overview

Research area: Computer vision and generative machine learning, specifically video diffusion models and 3D-geometric consistency.

Technical level: Advanced. The paper assumes familiarity with diffusion models (DDPM/flow matching, v-prediction parameterization), Direct Preference Optimization (DPO), LoRA fine-tuning, and geometry foundation models such as DUSt3R/MASt3R-style pointmap regressors. The core idea is intuitive, but the formal treatment of DPO in velocity space is dense.

Scope: A data-efficient, self-supervised post-training method that uses a geometry foundation model to produce preference signals, aligning pretrained video diffusion models toward 3D-consistent generation.

What This Paper Is About

Modern video diffusion models produce visually convincing footage but frequently violate 3D structure: objects deform, backgrounds drift, and scenes collapse over time. The authors argue this happens because standard denoising objectives only reward pixel-level statistical matching, with no incentive for geometric coherence. VideoGPA fixes this by using a pretrained geometry model to score how 3D-consistent a generated video is, then aligning the video model toward higher-scoring outputs with DPO — no human labels, no retraining from scratch.

Key Contributions

1. A self-supervised 3D consistency metric. The authors derive a scene-level score from re-projection error: a geometry foundation model predicts depth and camera poses for each frame, builds a point cloud, re-renders it back into every frame, and measures the reconstruction gap (MSE plus LPIPS). Low error means the video admits a coherent 3D explanation; high error means it does not.

2. Automatic construction of geometric preference pairs. Candidate videos are sampled from a pretrained model with different seeds, ranked by the 3D consistency score, and paired as winner/loser. Three filters (motion salience, a minimum score margin, and difficulty pruning) remove trivial or degenerate pairs, yielding roughly 2,500 usable preference pairs with no human annotation.

3. Adaptation of Diffusion-DPO to v-prediction video DiTs. The DPO objective is reformulated in terms of velocity-prediction error rather than noise-prediction error, giving a clean energy term for models trained with v-prediction — the dominant parameterization for current video diffusion transformers.

4. Demonstration that lightweight alignment surpasses heavier alternatives. LoRA fine-tuning on about 1% of parameters, for 1,500–10,000 steps, beats both supervised fine-tuning on curated data and epipolar-constraint DPO, and achieves stronger perceptual quality than GeoVideo, which uses explicit depth-supervised training over a much larger video set.

Main Findings

• I2V generation improves on every geometric axis. With CogVideoX-I2V-5B as the base, VideoGPA raises MVCS from 0.945 to 0.950, lowers 3DCS from 0.485 to 0.483, and reduces epipolar error from 0.585 to 0.539. The overall VideoReward win rate reaches 76.0%, against 66.0% for Epipolar-DPO and 35.0% for SFT.

• T2V gains are likewise consistent, without sacrificing visual quality. On CogVideoX-5B, VideoGPA posts the best SSIM (0.803), LPIPS (0.411), MVCS (0.953), 3DCS (0.422), and epipolar error (0.548), with an overall win rate of 60.33% versus 48.67% (Epipolar-DPO) and 15.33% (SFT).

• Scene-level geometry beats local pairwise constraints. Epipolar metrics operate on frame pairs and can produce false positives on degenerate outputs — collapsed textures or frozen regions still satisfy sparse epipolar relations. The global re-projection constraint rejects these cases because all frames must jointly support one 3D scene, which prevents spatial drift from accumulating.

• Geometry acts as a regularizer for motion. Although training uses predominantly static scenes, Motion Quality win rates improve, and object structure and motion continuity are better preserved. The authors frame this as projecting generation back onto the physically plausible subspace of the video manifold, letting the model disentangle camera motion from object motion.

• Perceptual gains are human-visible. In a blind study with 25 participants and 20 video groups each, VideoGPA received 53.5% of first-place votes; Epipolar-DPO was second at 22.4%.

• Far less training data than comparable alternatives. Against GeoVideo on CogVideoX1.5-5B, VideoGPA trains for only 1,500 steps and still achieves epipolar error 0.567 (vs. 0.875), MVCS 0.982 (vs. 0.819), and OVL 57.64% (vs. 18.06%) — while GeoVideo trained on roughly 10,000 depth-supervised videos.

• Qualitative improvements span several artifact types. Suppressed object splitting under camera motion, reduced flickering in high-frequency regions (building facades, fan grilles), more stable specular reflections and low-light scenes, and better color and material constancy across frames.

Methodology in Plain English

The pipeline has three stages.

Probe. Take a generated video, sample ten frames, and feed them to a geometry foundation model. That model outputs a depth map and camera pose for each frame. Back-project every pixel into 3D using its depth and pose, producing a colored point cloud of the scene.

Score. Re-project that point cloud into each of the original camera views and compare the re-rendered images against the real frames, using MSE and LPIPS. If the video was geometrically coherent, the point cloud reconstructs the frames well and the error is low. If the video had drifting geometry or deformation, the single global 3D scene cannot explain all frames at once and the error spikes. This error is the 3D consistency score.

Align. Generate several candidate videos per prompt with different random seeds. Rank them by score, pair the best against the worst within each group, and keep only pairs with a meaningful score gap that also pass motion and quality filters. Then run DPO on the video model — adapted to v-prediction, trained through LoRA adapters on about 1% of parameters — so the model learns to assign higher likelihood to geometrically consistent outputs.

For image-to-video training, the authors add scripted camera-motion prompts (combinations like "pull back away from the scene, then roll gently to one side") with an explicit static-scene constraint, so that camera trajectory is the dominant variable and any geometric failure is attributable to geometry rather than scene content. Evaluation uses natural captions instead, to check for prompt overfitting.

Why This Matters

Impact on research. The paper reframes 3D inconsistency in video generation as an objective-alignment problem rather than an architectural or data-scale problem. That is a meaningful shift: it suggests that models pretrained on billions of frames already contain latent 3D knowledge that simply is not being reinforced by the pixel-matching loss, and that a small, targeted post-training step can surface it. It also establishes that a geometry foundation model can serve as an automated reward model, which is a reusable recipe for other generative settings where supervising signals are hard to specify by hand.

Real-world applications.

• Film, advertising, and content creation: fewer reshoots and manual fixes for deforming objects or drifting backgrounds in AI-generated footage.
• Data engines for embodied AI and robotics: synthetic video used to train policies must obey physical geometry, or downstream agents learn inconsistent spatial priors.
• Novel view synthesis and 3D reconstruction: stable multi-view video feeds directly into scene reconstruction pipelines.
• Simulation and autonomous driving: camera-motion-heavy footage used for perception training needs correct projective geometry to be useful.

Industry relevance. The method is deliberately cheap: no human annotation, roughly one percent of parameters trained, a few thousand preference pairs, and a short training run. That makes it practical as a post-training step on top of existing video model releases rather than a from-scratch training effort. The fact that it composes with different base models (CogVideoX 5B, CogVideoX1.5-5B, and Wan2.2 in the appendix) is what makes it commercially interesting — it can be applied as a bolt-on refinement stage.

Future Directions

• Scalability of the geometry probe. The authors flag this as the main limitation: reconstruction runtime and memory grow with video length. Lightweight or streaming geometry foundation models would directly address it.

• Extending beyond predominantly static training scenes. Training pairs emphasize static scenes with camera motion. Whether explicit dynamic-object geometry supervision would compound the observed motion-coherence gains is open.

• Broader architecture coverage. Only CogVideoX variants and Wan2.2 were tested. Whether the alignment transfers to other parameterizations, longer-context models, or autoregressive video generators is unresolved.

• Tighter coupling of geometric and perceptual objectives. VideoGPA improves geometry without hurting perceptual quality, but the two are still optimized separately. A joint formulation might achieve further gains, and the trade-off frontier between them is not fully mapped.

Target Audience

Researchers working on video diffusion models, diffusion alignment, and preference optimization; 3D vision researchers interested in geometry foundation models as supervision sources; and applied ML engineers building video generation pipelines who need 3D-consistent output without retraining a base model. Readers without a background in diffusion objectives or DPO will find Section 3 heavy going, but the high-level pipeline and the experimental results are accessible to anyone familiar with generative video.

Authors’ abstract

While recent video diffusion models (VDMs) produce visually impressive results, they fundamentally struggle to maintain 3D structural consistency, often resulting in object deformation or spatial drift. We hypothesize that these failures arise because standard denoising objectives lack explicit incentives for geometric coherence. To address this, we introduce VideoGPA (Video Geometric Preference Alignment), a data-efficient self-supervised framework that leverages a geometry foundation model to automatically derive dense preference signals that guide VDMs via Direct Preference Optimization (DPO). This approach effectively steers the generative distribution toward inherent 3D consistency without requiring human annotations. VideoGPA significantly enhances temporal stability, geometric plausibility, and motion coherence using minimal preference pairs, consistently outperforming state-of-the-art baselines in extensive experiments.