Learning to Correct Geometry in Generated Videos

AI summary

Overview

Research area: Computer vision — generative video models, 3D/geometric consistency, and post-hoc video-to-video refinement.

Technical level: Intermediate. The paper assumes familiarity with video diffusion models, camera pose representations (position plus quaternion rotation), optical flow, and standard video reconstruction metrics (PSNR, SSIM, LPIPS).

Scope: The paper proposes Video Geometry Corrector (VGC), a post-hoc video-to-video diffusion model trained on a synthetically constructed paired dataset (VG-Pairs) to repair camera-motion-induced geometric distortion in generated videos without modifying the source generator.

What This Paper Is About

Video generators such as Sora, CogVideoX, HunyuanVideo and Wan produce visually convincing clips, but once the camera moves, rigid structures warp, parallax becomes inconsistent, scene layout drifts across frames, and continuous camera motion can collapse into shot-like cuts. The authors argue this is largely a data problem: web-scale video corpora mostly show dynamic objects filmed by static or weakly moving cameras, so models get little supervision for how appearance should change as the viewpoint changes, and they end up explaining camera motion by deforming the scene instead. The goal is to build a separate corrector that takes an already-generated video and makes it geometrically consistent while preserving its content, camera intent, and appearance.

Key Contributions

1. Problem framing and diagnosis. The authors identify post-hoc geometric correction (as opposed to pixel-level restoration or modifying the generator) as an underexplored task, and attribute camera-induced distortion to insufficient supervision for viewpoint-induced appearance change in existing video data.

2. VG-Pairs, a synthetic paired-supervision dataset. Real videos with diverse camera motion and pose annotations are treated as geometry-consistent targets; distorted counterparts are synthesized with a pretrained image-to-video inpainting model (Wan 2.1-Fun-InP) using a trajectory-adaptive keyframe strategy that inserts extra keyframes at abrupt pose changes and stitches shorter subsegments.

3. VGC, a flow-augmented video-to-video corrector. A diffusion model trained to map distorted input to the geometry-consistent target, with optical flow computed from the distorted input as an auxiliary conditioning signal to help disentangle camera motion from object motion. No modification of the source generator is required.

4. Cross-generator transfer and scalability. Demonstrated improvements on pose-conditioned and prompt-driven benchmarks across multiple source generators (DFoT, three CogVideoX variants, HunyuanVideo 1.5, Wan 2.1), plus a larger-backbone version (Wan 2.2-14B) that extends the framework toward general video refinement, including dynamic human and object motion.

Main Findings

• Pose-conditioned results on RealEstate10K (100 sampled test videos, source generator DFoT): VGC applied to DFoT reaches PSNR 14.87, SSIM 0.4545, LPIPS 0.4395, RVE 168.9, versus 13.61 / 0.4006 / 0.4487 / 184.2 for DFoT alone. The paper describes VGC as having the strongest PSNR, SSIM, and RVE among the direct baselines; its LPIPS (0.4395) is higher—worse—than Geometry Forcing's 0.4127.

• VGC stacks on top of geometry-aware generators: Applied to Geometry Forcing, results improve to PSNR 15.50 (+0.74), SSIM 0.4715 (+0.0232), RVE 158.8 (-22.2); applied to Geometry Forcing + REPA, to PSNR 15.25 (+1.09), SSIM 0.4675 (+0.0458), RVE 160.9 (-10.9).

• Optical flow is a modest but positive ablation: Removing flow gives PSNR 14.76 / SSIM 0.4509 / RVE 169.4 for DFoT + VGC, versus 14.87 / 0.4545 / 168.9 with flow. The authors conclude that flow improves local structural consistency and that the main gains come from the paired supervision itself.

• Prompt-driven results on DL3DV (200 prompts from the held-out test set, reconstruction via Depth Anything 3 following VideoGPA): PSNR improves for all three source generators — CogVideoX-5B T2V 15.6668 → 16.6940 (+1.0272), CogVideoX 1.5-5B T2V 15.9062 → 17.7342 (+1.8280), CogVideoX-5B I2V 17.3189 → 17.9989 (+0.6800) — with corresponding MSE reductions (0.0333 → 0.0264; 0.0289 → 0.0192; 0.0218 → 0.0184).

• Gains also hold on top of VideoGPA: VideoGPA on CogVideoX-5B T2V goes from PSNR 14.6318 to 15.9092 (+1.2774) with VGC; on CogVideoX 1.5-5B T2V from 14.4002 to 16.8430 (+2.4428); on CogVideoX-5B I2V from 15.7203 to 17.3053 (+1.5850), with 3DCS improving in each case.

• Not every metric moves the same direction: In Table 2, LPIPS increases (worse) for every VGC variant, e.g. 0.5285 → 0.5733 for CogVideoX-5B T2V; for CogVideoX-5B I2V, SSIM decreases (0.5543 → 0.5456) and 3DCS increases (0.7944 → 0.8121, worse since the arrow indicates lower is better). The authors' stated claim is consistent improvement in PSNR, SSIM, and MSE.

• General video refinement with a 14B backbone (100 held-out SpatialVID videos, Wan 2.2-14B, 480p input to 720p output): On CogVideoX-I2V, VGC reaches dynamic degree 0.66, imaging quality 72.7654 and DOVER 10.50 versus 0.30 / 69.6945 / 9.54 for the baseline, 71.1767 imaging quality and DOVER 8.31 for SeedVR2, and DOVER 10.31 for VEnhancer v2. On HunyuanVideo 1.5-I2V, VGC obtains the highest imaging quality (75.1367) and DOVER 9.30 (baseline 10.10, VEnhancer v2 9.85). On Wan 2.1-Fun-InP, VGC obtains DOVER 11.47 and imaging quality 72.6499, but aesthetic quality 0.5141 is below SeedVR2's 0.5431.

• Qualitative failures corrected: Restored consistent lighting and coherent sofa structure on RealEstate10K; fixed a duplicated streetlight (CogVideoX baseline) and a distorted, tilted streetlight (VideoGPA) on DL3DV; for dynamic scenes, preserved a man in a leather jacket across frames and restored a missing cyclist.

• The data pipeline's key trick matters: For clips with large, nonlinear camera motion, conditioning only on first and last frames can make the inpainting model take a shortcut trajectory of about 45° rotation versus about 225° in the ground truth; adaptive keyframes keep the distorted video close to the target trajectory.

Methodology in Plain English

1. Use real footage as the "correct" answer. The authors start from real videos that genuinely have camera motion and come with pose annotations (camera position plus rotation per frame), drawn from DL3DV and SpatialVID. Each real clip is treated as the geometry-consistent target.

2. Deliberately break a copy of it. A pretrained image-to-video inpainting model is asked to fill in the frames between a start and an end frame. Because that model has the same geometric weaknesses as other generators, the resulting clip keeps the scene and roughly the same camera path but introduces warping, inconsistent parallax, depth-order errors, and shot-like jumps — exactly the errors to be fixed. The pair (broken copy, real clip) is a training example.

3. Decide where to place keyframes automatically. The pipeline sums rotation (geodesic distance between quaternions) and translation over the clip, then computes how many segments are needed given per-segment budgets (120° rotation, 10 m translation), capped at 4 segments and reduced further if any segment would fall below 15 frames. Interior keyframes are placed at moments of peak trajectory inflection, scored by the angle between consecutive translation vectors plus a weighted angle between consecutive rotation increments, chosen greedily with non-maximum suppression and a minimum temporal separation; leftover keyframes fill the largest gaps roughly uniformly. Segments are generated independently and stitched, with duplicate boundary frames removed.

4. Add motion structure as a hint. Optical flow (estimated with SEA-RAFT) is computed between consecutive frames of the distorted input and fed in alongside the distorted video and the text prompt, via channel concatenation for video and flow and cross-attention for text. Because flow comes from the input, it is available at inference too. Flow is not a direct supervision signal; it tells the model which motion is globally coherent (camera) and which is local (objects).

5. Train a corrector, not a generator. VGC is built on Wan 2.1-1.3B and trained with a standard noise-prediction objective to denoise toward the real target while conditioned on the distorted video, text, and flow. Training used 10K steps on 32 H100 GPUs with AdamW, learning rate 1×10⁻⁵, weight decay 0.01, global batch size 64, on 81-frame clips at 832×480. The dataset comprised 9K DL3DV videos (first 1K reserved for evaluation) plus 63K filtered SpatialVID videos with large camera motion.

6. Scale up for the general case. A version based on Wan 2.2-14B was trained and evaluated on dynamic videos, where it also performs super-resolution from 480p to 720p and is judged with no-reference VBench metrics plus DOVER, since dynamic clips have no geometry-consistent target to reconstruct against.

Why This Matters

Impact on research. The paper reframes geometric inconsistency as a supervision gap rather than an architectural flaw, and shows that a separate, generator-agnostic corrector trained on synthetic pairs can be layered onto any model — including closed ones. This offers a complementary path to approaches that require retraining with depth, epipolar, or 3D-foundation-model priors, and it introduces a reusable paired dataset construction recipe where real pairs are essentially impossible to collect.

Real-world applications:

• Post-production and VFX cleanup: repairing camera moves in AI-generated or AI-assisted footage without regenerating the shot and losing the approved look.
• Virtual production and previsualization: turning rough generated camera fly-throughs into stable layouts suitable for set planning or compositing.
• Real-estate and architectural visualization: the pose-conditioned RealEstate10K setting maps directly onto walkthrough generation where structure must stay believable under large viewpoint change.
• Marketplace, advertising, and social content pipelines: a drop-in refinement stage that improves clips from multiple vendors' generators without re-training or re-licensing each model.

Industry relevance. Because the method treats the video generator as a black box and is distributed as a post-hoc module, it fits the common industrial reality where foundation models are accessed only as checkpoints or inference APIs. The scaling result — the same recipe improving dynamic human and object motion at 14B and adding 480p-to-720p super-resolution — positions it as a candidate general video-refinement stage rather than a narrow geometry tool.

Future Directions

• Closing the LPIPS and aesthetic-quality gap. VGC raises LPIPS on the DL3DV benchmark and scores lower aesthetic quality than SeedVR2 at the 14B scale, so combining geometric correction with perceptual/appearance objectives is an open problem.
• Characterizing when flow helps. The flow ablation shows only modest gains on RealEstate10K; understanding whether stronger motion representations or explicit disentanglement losses would help more is unresolved.
• Extending paired-supervision construction. VG-Pairs depends on videos with pose annotations and on an inpainting model whose own distortions define the error distribution; broadening to scenes without pose labels, or to synthesized distortions beyond inpainting shortcuts, is a natural next step.
• Unifying correction and enhancement. The 1.3B model fails to recover dynamic human motion while the 14B model does, leaving open how capacity, resolution, and the correction objective interact, and whether one model can serve geometry correction, restoration, and super-resolution simultaneously.

Target Audience

Researchers and engineers working on video diffusion models, video post-processing and enhancement pipelines, and 3D-consistent generation. It is also useful for practitioners who consume third-party generative video APIs and need a refinement layer they can apply without retraining, and for readers interested in synthetic paired-data construction as a way around uncollectable real-world supervision.

Authors’ abstract

Generated videos often lose geometric consistency under camera motion: structures warp under viewpoint change, parallax becomes inconsistent, and smooth camera movement may collapse into shot-like transitions. We argue that this failure is largely a data problem. In web-scale video corpora, most visible motion comes from dynamic objects captured by static or weakly moving cameras, leaving only sparse supervision for viewpoint-induced appearance changes. To address this, we propose Video Geometry Corrector (VGC), a post-hoc video-to-video model trained on synthetic paired supervision. Starting from real videos with diverse camera motion and pose annotations, we construct VG-Pairs with a synthetic paired-supervision data generation pipeline. Each video clip is treated as a geometry-consistent target and paired with a distorted counterpart synthesized by a pretrained image-to-video inpainting model using trajectory-adaptive keyframes. VGC learns to map the distorted input to the geometry-consistent target without modifying the source generator, and uses optical flow as an auxiliary cue to help disentangle camera motion from object motion. Experiments on pose-conditioned and prompt-driven benchmarks show that VGC improves geometric consistency across multiple source generators, and scaling to a larger backbone further extends the framework toward more general video refinement.