FixAnything: 3D-Consistent Rendering Refinement via Video Generative Priors

AI summary

Overview

Research area: Computer vision, specifically novel view synthesis, 3D scene reconstruction, and generative video models.

Technical level: Advanced. The paper assumes familiarity with diffusion and flow-matching generative models, 3D representations such as Gaussian Splatting and NeRF, and preference optimization.

Scope: FixAnything is a single finetuned video diffusion model that cleans up artifact-ridden renderings from four different 3D representations while preserving camera trajectories and supporting downstream 3D reconstruction.

What This Paper Is About

Every 3D scene representation — Gaussian Splatting, NeRF, meshes, or point clouds — produces visual artifacts when the input photographs are sparse or when a novel viewpoint sits far from the captured views. Prior fixes are specialists: each one targets a single representation and needs its own architecture, conditioning design, and large training set. This paper asks whether one generalist model can remove artifacts across all of them, and answers by repurposing a pretrained video diffusion model with only lightweight adaptation.

Key Contributions

1. A generalist, representation-agnostic cleanup framework. FixAnything takes a video rendered along a camera trajectory from any of four 3D representations (3DGS, NeRF, mesh, sparse SfM point cloud) and outputs a cleaned video, using one model with no architectural changes to the base video diffusion model.

2. Mask-aware conditioning that separates "trust" from "fix." A per-frame binary mask marks which frames come from training viewpoints (clean) and which are degraded. The clean frames act as anchors that the model propagates appearance, lighting, and structure from, rather than hallucinating over already-correct content.

3. Geometry-aware preference optimization. Camera pose accuracy recovered by structure-from-motion (COLMAP with SuperPoint features and LightGlue matching, scored by AUC@5° over relative rotation and translation accuracy) is used as a reward signal, with Flow-DPO steering the model toward outputs that downstream 3D reconstruction can actually use.

4. Extreme data efficiency with no inference overhead. Effective cleanup is achieved with as few as 20 paired training videos, and the geometry prior is baked into the LoRA adapter during training, so no pose estimation runs at inference time.

Main Findings

• One model, four representations. The same model cleans up NeRF, 3DGS, mesh, and sparse point cloud renderings to comparable quality. On DL3DV-10K at 6 training views, PSNR/SSIM/LPIPS are 17.01/0.522/0.329 (NeRF input), 17.65/0.561/0.289 (3DGS), 17.95/0.583/0.269 (mesh), and 17.74/0.568/0.271 (sparse SfM points).

• Sparse point clouds rival full 3D representations. Sparse COLMAP keypoint renderings, with the only dense visual information coming from the few clean training views the trajectory passes through, perform on par with or better than 3DGS and NeRF inputs. At 9 views, the sparse point cloud input reaches 19.72/0.624/0.241 versus 19.76/0.632/0.269 for 3DGS input.

• Competitive against specialist pipelines. FixAnything with 3DGS input reaches 15.18 PSNR at 3 views, 17.65 at 6 views, and 19.76 at 9 views, against post-hoc enhancement baselines 3DGS-Enhancer (14.33, 16.94, 18.50), Xu et al. (14.62, 17.35, 19.19), Difix3D (12.85, 14.84, 16.76) and Difix3D+ (12.37, 14.41, 16.39). The authors describe this as competitive or superior performance on the 3DGS input while being simpler to implement.

• Flow-DPO improves geometry substantially at higher cost of nothing at inference. SFT-only scores 17.51 PSNR / 0.554 SSIM / 0.296 LPIPS / 61.12 AUC@5°. After DPO: 17.65 / 0.561 / 0.289 / 68.32, a 7.2% AUC@5° gain. Image quality metrics improve only modestly; the gain is concentrated in geometric consistency.

• The mask matters by 1.3 dB. Without the mask, PSNR is 16.37 with SSIM 0.525 and LPIPS 0.311; with the mask, 17.65 / 0.561 / 0.289. Without it, the model hallucinates over clean frames (the paper's example is a wet floor sign disappearing), and the corruption spreads to nearby frames.

• Training data requirements are tiny. 20 paired videos give 16.70 PSNR / 0.531 SSIM / 0.309 LPIPS; 50 give 17.20 / 0.548 / 0.297; 100 give 17.45 / 0.556 / 0.292; 500 give 17.65 / 0.561 / 0.289. The authors compare this to prior methods requiring 80K–150K image pairs.

• Denoising steps trade quality for speed. At 50 steps: 17.65 PSNR / 0.561 SSIM / 0.289 LPIPS in 309 s. At 25 steps: 17.75 / 0.564 / 0.289 in 155 s. At 10 steps: 17.91 / 0.570 / 0.296 in 62 s. At 5 steps: 18.02 / 0.574 / 0.313 in 31 s. A 61-frame clip at 480×832 resolves in 31 seconds on a single H100 at 5 steps.

• Uncertainty can be estimated for free. Running inference 5 times with different seeds and taking per-pixel standard deviation yields an uncertainty map. On DL3DV at 6 views, mean PSNR over the most-confident 25% of pixels is 25.7 dB versus 14.4 dB over the least-confident 25%. Sky- and ground-like regions show low uncertainty, while regions with multiple plausible completions (such as buildings) show high uncertainty.

• Hallucination is reframed as a feature, not purely a failure. The authors define hallucination as content the model must invent when it is not seen in the input views, and argue it becomes a failure only when it contradicts existing observations.

• Cross-dataset generalization is reported in supplementary material. Evaluation on MipNeRF-360 and LLFF shows FixAnything with 3DGS input achieving comparable performance to state-of-the-art methods with a notable improvement in LPIPS.

Methodology in Plain English

The core idea is that a degraded rendering and a clean video differ in appearance, but both follow the same camera path through the same rough scene. That shared structure lets the problem be framed as video-to-video translation rather than as 3D generation.

Base model and adaptation. The researchers start from Wan2.1-I2V-14B, a large pretrained image-to-video diffusion model built on a DiT architecture and trained with rectified flow. They leave the architecture untouched. The degraded rendering is encoded into latent space by a frozen VAE, then concatenated channel-wise with the noised target latent and a spatially broadcast binary mask. Only a LoRA adapter of rank 64 is trained, updating less than 1% of total parameters. The model predicts a velocity field and is trained with a standard flow-matching objective.

The mask. A per-frame binary flag marks which frames come from training viewpoints. Setting a frame's mask entry to 1 tells the model to trust and preserve it; 0 tells it to fix it. Because degradation severity typically grows with distance from the nearest anchor, the model implicitly learns to modulate the strength of its refinement from the mask's temporal arrangement.

Training data. Pairs are built from DL3DV-10K, which ships with precomputed COLMAP reconstructions. For each scene, between 3 and 12 frames are sampled as training views, and 61-frame trajectories passing through at least two of them are extracted. Each trajectory is rendered four ways: Nerfacto for NeRF (blur and fog), gsplat initialized from a random point cloud and deliberately underfit for 7K iterations for 3DGS (floaters), MapAnything plus a fitted triangular mesh for meshes (holes at ambiguous depth), and COLMAP keypoints only for sparse point clouds (scattered patches). For meshes and point clouds, frames at training viewpoints are replaced with the original captured images because depth at sky regions and occlusion boundaries is unreliable. Training uses 500 paired videos, first at 288×512, then at 480×832 with 61 frames, for 3000 iterations on a single H100.

Preference optimization. Flow matching alone does not enforce multi-view consistency, so the SFT model sometimes invents structures that look fine per frame but shift across frames. For 1,000 separate DL3DV scenes, the researchers generate five candidate outputs per scene with different random seeds, rank them by how accurately COLMAP can recover their camera poses, and keep only pairs with an AUC gap of at least 0.2. They then apply Flow-DPO with the SFT checkpoint as the reference model for 2000 additional iterations.

Inference. The user supplies a rendering video and a mask. The model samples Gaussian noise and integrates the learned velocity field using the standard flow matching ODE, 50 denoising steps by default, splitting longer videos into overlapping chunks of 61 frames.

Why This Matters

The paper argues that the field's habit of building a bespoke generative pipeline for each new 3D representation does not scale, and that a single generalist video prior can replace that growing family of specialists. It also makes a stronger claim: because sparse COLMAP point clouds clean up as well as fully trained NeRFs or Gaussian Splatting models, the intermediate 3D representations in typical pipelines may not be necessary at all. A conventional workflow estimates poses with COLMAP, learns NeRF or 3DGS, then refines with a generative model; the results here suggest rendering sparse point clouds and cleaning them directly gets comparable results. The observation that the real difficulty lies in weakly observed or unobserved regions — where the problem becomes plausible completion rather than geometric inference — points toward a division of labor: reconstruct reliably where observations exist, generate where they do not.

Real-world applications:

• Content creation and visual effects, where novel views of a captured scene must be usable rather than riddled with floaters and holes.
• Robotics, where downstream perception and planning degrade when rendered views are unusable.
• Consumer 3D capture, where users provide only a handful of photographs and expect plausible results.
• Virtual and augmented reality asset production from sparse photo sets.

Industry relevance: Because the adaptation requires no architectural changes, updates less than 1% of parameters, and can be trained in a single H100 run with only a few dozen paired videos, the recipe can be reapplied to stronger video foundation models as they appear without redesign. The 5-step variant's 31-second runtime for a 61-frame 480×832 clip is a practical throughput figure for production pipelines, and the authors note distillation could push it further.

Future Directions

• Distillation for real-time cleanup. The authors explicitly note that the 5-step result opens the door to further acceleration through distillation, potentially enabling real-time rendering cleanup.

• Feeding generative predictions back into reconstruction. The discussion proposes that generative completions could be fed back into reconstruction pipelines to produce more complete and consistent 3D scene models, rather than being a terminal post-processing step.

• Making uncertainty actionable. The training-free uncertainty estimate correlates with reconstruction error, but the paper presents it only as preliminary analysis. Turning it into a reliability signal for downstream systems is an open step.

• Adopting stronger video backbones. The framework is designed so newer video foundation models can be dropped in with only a new LoRA training run on academic-scale compute, leaving open how much further quality improves as those base models advance.

Target Audience

Researchers and practitioners in 3D computer vision, novel view synthesis, and generative modeling — particularly those working on sparse-view reconstruction, 3D Gaussian Splatting, NeRF, or video diffusion. Engineers building production 3D capture or rendering pipelines will find the data-efficiency and inference-cost numbers directly relevant. Readers without a background in diffusion models, rectified flow, or structure-from-motion will find the method sections demanding, though the framing of hallucination, the mask mechanism, and the representation-agnostic claim are accessible without that background.

Authors’ abstract

Rendering views using 3D scene representations such as Gaussian Splatting (3DGS), Neural Radiance Fields (NeRF), meshes, or even point clouds produces artifacts when input views are sparse or target views lie far from the input. Recent work mitigates these artifacts using diffusion-based generative priors, but is specialized to individual representations and require custom architectures or extensive retraining. We present FixAnything, a single model for fixing a wide range of rendering artifacts. It does so by repurposing a pretrained video generative model, leveraging its implicit multi-view priors with only minimal modification and lightweight finetuning. Our key insight is that even noisily-rendered sequences preserve camera motion and coarse scene structure, allowing cleanup to be formulated as video-to-video translation. To control what scene structure should be preserved, we introduce a binary mask denoting the clean pixels, enabling the model to anchor its output to high-quality inputs (e.g. training views) while refining the rest. To encourage FixAnything to produce 3D-consistent renderings that support downstream reconstruction, we use camera pose accuracy (recovered via structure-from-motion) as a reward signal for direct preference optimization (DPO). Across four distinct 3D representations, FixAnything consistently improves rendering quality with lightweight finetuning, demonstrating that a single generalist video prior can replace multiple specialist refinement pipelines. The simplicity of the framework enables immediate adoption of stronger future video models without architectural redesign.