GeoFix: Geometry Fixing for 4D Consistent Generation

AI summary

Overview

Research area: Computer vision — generative video modeling, 4D scene reconstruction, and reward-based alignment of diffusion models.

Technical level: Advanced. The paper assumes familiarity with video diffusion models, flow-matching samplers, Gaussian splatting, camera pose/intrinsics geometry, and reinforcement-learning-style reward alignment (DiffusionNFT, DPS guidance).

Scope: A single geometric reward, FOCAL, computed directly on video latents through a stitched 4D reconstructor, used for both training-time and inference-time alignment of video diffusion models to improve 3D consistency without hurting — and in fact improving — visual quality.

What This Paper Is About

Video diffusion models can generate visually impressive clips, but the underlying 3D structure often deforms or drifts as the camera moves, which limits their use as priors for 4D scene generation. Prior fixes attach geometric rewards to post-training, but those rewards tend to degrade texture and detail, forcing researchers to bolt on extra appearance rewards or filter preference data, which adds weighting and threshold choices. GeoFix argues that geometric inconsistency is itself a visual defect, and shows that a single, properly calibrated geometric reward can improve both geometric consistency and video quality at once.

Key Contributions

1. FOCAL, a FOV-calibrated geometric reward. The reward measures cross-view depth reprojection agreement restricted to static scene regions (obtained via a pretrained motion segmentation network), so valid object motion is not penalized. It is then corrected against the depth agreement expected at a given camera field-of-view overlap, removing an "overlap bias" the authors identify in raw depth-consistency scores.
2. Latent-to-4D model stitching. A pretrained 4D reconstructor (NeoVerse) is stitched to the video model's latent space with a lightweight stitching layer plus a trainable copy of the remaining reconstructor layers, trained by output distillation. This yields depth, camera poses, intrinsics and an explicit 4D Gaussian representation directly from latents, bypassing VAE decoding.
3. A reward-alignment-agnostic framework. FOCAL requires no VAE decoding and is differentiable in the latent, so GeoFix is instantiated with reward fine-tuning (DiffusionNFT), zeroth-order selection (Best-of-N), and first-order guidance (DPS), all without auxiliary rewards.
4. Empirical gains across tasks. Reported improvements include 39% on WorldScore 3D consistency and a 4% increase in VBench Quality Score, plus better camera accuracy and more coherent 4D Gaussian scene generation.

Main Findings

• Training-time alignment wins on geometry and quality. With Wan2.2-TI2V-5B, GeoFix reaches 61.60 WorldScore 3D Consistency and 51.06 Photometric Consistency versus 44.39 and 33.55 for the base model, 58.32 and 47.00 for VIST3A, and 42.26 and 31.64 for World-R1. It also posts the best average WorldScore (33.01 vs 31.06 for VIST3A and 29.99 for base).
• Supervised fine-tuning alone explains little. SFT reaches only 50.11 3D Consistency and an average of 31.02, indicating most of the gain comes from the reward rather than the supervised data.
• Larger relative VBench gains than VGGRPO. GeoFix improves every VBench dimension by a larger relative margin than VGGRPO, with the largest gaps in visual quality — even though no video-quality metric is used as a reward.
• Inference-time steering also works. Best-of-N (N=20) raises 3D Consistency from 75.43 (unguided) to 78.00 and Photometric Consistency from 76.06 to 78.33; DPS guidance raises them to 76.49 and 76.59. Camera accuracy improves too: rotation error drops from 2.040 to 1.751 (Best-of-N) and 1.977 (DPS), translation error from 34.97 to 34.57 and 34.63, ATE norm from 0.2563 to 0.2507 and 0.2562. Motion accuracy and motion magnitude drop only marginally.
• Better image-to-4D scenes than MoGe4D and 4DNeX. On 100 VBench-I2V images rendered along a common ReCamMaster trajectory, GeoFix achieves the highest consistency scores (reported in Figure 6). 4DNeX has the highest dynamic degree, which the authors attribute largely to frame-to-frame structural inconsistency rather than valid motion; MoGe4D has the lowest dynamic degree.
• The reward tracks geometric corruption. Under four perturbations a valid camera motion cannot explain (warp, squeeze, depth-layer warp, pose drift), FOCAL decreases monotonically, confirming it measures geometry rather than image fidelity (95% bootstrap confidence intervals reported).
• Calibration matters. Removing FOV calibration (γ=0) lowers VBench average to 0.6802 and dynamic degree to 0.395, because the model reduces camera motion to collect the easier reward; γ=0.7 matches γ=1 on dynamic degree (0.425) while giving the best VBench average (0.6861 vs 0.6823).
• Directly optimizing VBench rewards degenerates. Training on five VBench rewards produces saturated videos, with longer training adding blur and grid artifacts — the authors' contrast case for why a geometric reward is the better route to quality.

Methodology in Plain English

The team starts from the observation that a good reward should judge structure, not appearance. So instead of comparing colors across views, they take the depth map and camera pose estimated for one frame, use the camera motion to project those depths into another frame, and check whether the projected depths match that frame's depth map. Only static regions are compared — a motion segmentation network masks out moving objects — so a walking person or swinging arm is not punished as an error.

The catch they discovered is that this score is not purely about geometry: even clean real videos and multi-view 4D datasets score better when the two cameras see more of the same thing. A model could therefore raise its reward simply by moving the camera less. They quantify overlap as an area-weighted IoU of the two camera frustums on a sampled reference sphere, fit the relationship between raw score and log-overlap on reference data, and subtract that trend out. The result is a score that says "how good is this depth agreement relative to what this camera motion should give." The calibration coefficient is estimated once and frozen.

To get depth and poses cheaply, they splice the video model's latent into a pretrained 4D reconstructor: a small trainable layer maps the latent into the reconstructor's intermediate feature space, and the reconstructor's remaining layers are copied and trained by distilling the original RGB-input model's outputs (depth, poses, intrinsics, Gaussian attributes). This avoids decoding to pixels and, per prior work, is more robust to latent noise.

Because the reward is differentiable in the latent, it plugs into whichever alignment machinery you have. For training they combine a supervised loss on MultiCamVideo and UltraVideo with a DiffusionNFT loss scored by the reward (64 rollouts per step from VideoUFO prompts). At inference they either generate 20 samples and keep the best-scoring one, or add the reward gradient to the latent update during denoising steps 3–17, using a soft, differentiable relaxation of the overlap term so gradients reach the predicted cameras.

Why This Matters

Research impact. The paper reframes geometric alignment: instead of a multi-reward balancing act with appearance terms, it argues that a well-designed geometric signal alone is sufficient and even beneficial for quality. The overlap-bias analysis is a diagnostic that likely applies to other depth-, epipolar-, and RGB-reprojection rewards, and the latent-space stitching lets geometry supervision reach the diffusion model without a VAE decode bottleneck.

Real-world applications

• Generating dynamic 4D assets for AR/VR and games from a single image or text prompt, where background stability under novel viewpoints is essential.
• Camera-controllable video generation for virtual production and previsualization, where the rendered camera path must match a specified trajectory.
• Robotics and simulation, where video models act as priors for building coherent dynamic environments for training and planning.
• Video post-production and VFX, where consistent scene structure across shots is needed for compositing and re-shooting.

Industry relevance. The method is a bolt-on to existing post-training and sampling pipelines and uses publicly available components (Wan2.2 models, NeoVerse), so the barrier to adoption is low. The absence of auxiliary reward weights and preference-data thresholds removes a major tuning cost that has kept reward alignment finicky in production settings.

Future Directions

• Extending calibration beyond overlap. The correction addresses camera FOV overlap only; other confounders of depth agreement — scene depth range, textureless regions, occlusion density — are not modeled.
• Robustness of the static-region mask. The reward depends on a pretrained motion segmentation network; how errors in that mask propagate into the reward and into trained models is not reported.
• Dependence on the stitched reconstructor. Depth and pose quality come from the 4D reconstructor, so failure modes of that model (for example, the scale ambiguity the authors handle heuristically in evaluation) carry into the reward.
• Scaling and generalization. The paper reports a 1,000-step LoRA fine-tune on Wan2.2-TI2V-5B; whether the same calibration coefficient and benefits hold for larger backbones, longer videos, or non-photorealistic domains is left open.

Target Audience

Researchers and engineers working on video diffusion, 4D content generation, and reward-based model alignment who already understand diffusion samplers and multi-view geometry. It is also valuable for practitioners building camera-controllable video or image-to-4D pipelines, and for anyone designing reward functions who wants a concrete case study in how an uncalibrated metric can be gamed by the model. Beginners will find the high-level motivation accessible but will need background in Gaussian splatting and diffusion guidance to follow the method.

Authors’ abstract

Video diffusion models increasingly serve as priors for 4D generation, but the generated videos often lack geometric consistency. Recent approaches address this through post-training by designing geometric rewards, yet this can degrade texture and fine detail, requiring auxiliary appearance rewards and careful reward balancing. To address this challenge, we propose GeoFix, a unified framework that improves geometric consistency and visual quality with a single geometric reward, applicable to both training and inference-time reward alignment. Our reward measures cross-view depth consistency in static scene regions, targeting structural inconsistencies without directly constraining appearance or penalizing valid object motion. Crucially, depth agreement increases with camera field-of-view overlap even in real videos and 4D datasets, allowing a smaller camera movement to be mistaken for better geometry. We account for this effect by calibrating the reward against the overlap-dependent trend. GeoFix achieves a 39% improvement in WorldScore 3D consistency and a 4% increase in VBench Quality Score.