Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support Token Refinement

AI summary

Overview

• Research area: Computer vision, specifically 3D scene inpainting, feed-forward 3D reconstruction, and pose-free novel view synthesis.
• Technical level: Advanced. The paper assumes familiarity with vision transformers, cross-view attention, 3D Gaussian Splatting, diffusion priors, and Structure-from-Motion pipelines.
• Scope: A single feed-forward framework (FreeInpaint) that inpaints masked regions in unposed multi-view image collections while jointly estimating camera poses and building a 3D Gaussian scene, validated on three inpainting benchmarks.

What This Paper Is About

3D scene inpainting fills in missing or removed regions of a 3D scene so the result stays geometrically and texturally consistent across many viewpoints. Existing methods require camera poses that are usually computed by a slow, failure-prone Structure-from-Motion preprocessing step (such as COLMAP), which breaks down on casual captures, sparse views, untextured areas, and already-masked images. FreeInpaint instead takes one reference image plus several unposed, masked source images and directly produces a complete, multi-view-consistent 3D Gaussian field, estimating the camera parameters itself.

Key Contributions

1. FreeInpaint, a feed-forward framework that performs 3D scene inpainting directly from unposed, masked image collections, eliminating any dependence on pre-computed camera poses.
2. Learnable Mask Attention, a per-token, key-side attention bias initialized from the input occlusion mask and updated across layers through learnable residuals, so masked regions cannot corrupt the foundation model's native pose and geometry estimation while still eventually absorbing useful context.
3. Support Token Refinement, an inference-time strategy that detects under-observed "blind spots," generates a diffusion-based support view using SDEdit plus an IP-Adapter, and re-injects it as confidence-weighted auxiliary tokens for progressive refinement.
4. Extensive experiments on SPIn-NeRF, 360-USID, and LLFF showing superior inpainting quality without camera poses, plus ablations on each component, view-count scalability, and sensitivity to the reference view.

Main Findings

• Against per-scene optimization methods (Table 1): FreeInpaint reaches 17.79 PSNR / 0.2819 LPIPS on SPIn-NeRF, 18.58 PSNR / 0.2526 LPIPS on 360-USID, and 0.5613 C-KID / 343.97 C-FID on LLFF, beating SPIn-NeRF, NeRFiller, InFusion, GScream, and AuraFusion360 on most metrics.
• Speed: A single forward pass takes about 0.4 seconds on an A6000 GPU, versus roughly 30 minutes to 5 hours for the optimization-based baselines. When severe occlusions trigger Support Token Refinement, the time rises to about 1.8 seconds, which the authors state remains orders of magnitude faster than the baselines.
• Against feed-forward unposed baselines (Table 2): FreeInpaint outperforms LAMA+DA3 and MVInpainter+DA3 on all evaluated metrics; the authors attribute the baselines' failures to multi-view inconsistency in 2D inpainting corrupting pose estimation and to pose misalignments.
• Ablation of components on 360-USID (Table 4): Baseline DA3 gives 15.58 PSNR / 0.7399 LPIPS / 392.83 FID; adding fine-tuning gives 16.96 / 0.2854 / 259.26; adding Mask Attention gives 18.26 / 0.2628 / 209.86; adding Token Refinement gives 18.58 / 0.2526 / 199.61.
• Mask attention type (Table 5): No mask attention gives 16.96 PSNR and ATE 0.0462, RPE_t 0.0803, RPE_r 0.7766; a hard binary mask gives 17.97 PSNR and ATE 0.0377, RPE_t 0.0764, RPE_r 0.7430; the learnable mask gives 18.58 PSNR, LPIPS 0.2526, and ATE 0.0340, RPE_t 0.0644, RPE_r 0.7069. Adaptive masking improves pose accuracy, not just pixels.
• Object replacement (Table 3): FreeInpaint achieves CLIP_dir 0.1942, C-KID 0.2589, and C-FID 153.25, surpassing non-reference DiGA3D (0.1928 / 0.2607 / 169.08) and reference-based NeRFiller (0.1917 / 0.2628 / 198.25) and InFusion (0.1922 / 0.2647 / 236.75).
• Scalability to more views (Table 6): On SPIn-NeRF, PSNR rises from 17.79 with 4 total input views to 18.32 at 8, 18.33 at 16, and 18.47 at 32; on 360-USID, PSNR rises from 17.19 at 4 views to 18.58 at 8, 18.66 at 16, and 18.69 at 32. Perceptual metrics stay stable with minor fluctuations.
• Sensitivity to the reference view (Table 7): Over ten diffusion seeds, PSNR is 17.68 ± 0.16, LPIPS 0.2808 ± 0.0168, FID 149.85 ± 7.55; across four reference-view choices, PSNR is 17.61 ± 0.24, LPIPS 0.2867 ± 0.0144, FID 152.18 ± 12.76, indicating moderate stability.
• Refinement iteration budget (Table 8a, averaged over 360-USID and IMFine): 1 iteration gives 18.02 PSNR / 0.2738 LPIPS in 0.41s; 2 give 18.29 / 0.2703 in 1.12s; 3 give 18.46 / 0.2623 in 1.83s; 4 give 18.48 / 0.2620 in 2.54s, with only +0.02 dB gained from iteration 3 to 4 at linearly increasing cost.
• Visibility threshold (Table 8b): Thresholds of 0.93, 0.95, 0.97, and 0.99 yield 18.12 / 18.25 / 18.46 / 18.30 PSNR with average iteration counts of 1.33, 1.50, 2.67, and 4.33 respectively.
• Support view generation ablation: Using the IP-Adapter without SDEdit repairs basic geometry but introduces structural artifacts from multi-view inconsistency; using SDEdit without the IP-Adapter preserves spatial structure but produces textures that do not match the reference; only combining both repairs distortion while keeping textural identity.

Methodology in Plain English

FreeInpaint builds on Depth Anything 3 (DA3), a visual geometry foundation model that reads an arbitrary number of images and predicts camera poses, depth, ray maps, and pixel-aligned 3D Gaussian parameters using a DINOv2 vision transformer backbone with alternating within-view and cross-view attention. The authors keep that backbone and its geometric priors but adapt it to masked inputs.

The first adaptation is the mask attention bias. Each input binary mask is downsampled to the transformer's patch resolution to give a token-level vector, converted into a logit-scale suppression state, and then updated at every cross-view attention layer by a small learned network conditioned on the current tokens. The resulting bias is subtracted from the attention logits. This means masked tokens are strongly suppressed early on, when they carry no information, but can gradually earn attention in deeper layers as they accumulate context. In the appendix, each such lightweight network is described as a 3×3 convolution, a depthwise 3×3 convolution, and a 1×1 projection with GELU activations, a hidden dimension of 8, a residual scaling factor α of 1.5, and a suppression strength β of 10; the final projection is zero-initialized so training starts from the binary-mask prior.

The second adaptation handles blind spots, regions the reference view simply cannot see. After the first pass, the predicted Gaussians are rendered back into the masked source views and a visibility score is computed from the rendered opacity inside each mask; the view with the lowest score becomes the anchor. A diffusion model then generates a support view using SDEdit on the rendered image canvas, conditioned on the reference image through an IP-Adapter so that both spatial and textural priors are preserved. This support view is encoded into auxiliary tokens and injected with a confidence weight map derived from the anchor mask and rendered opacity, Gaussian-smoothed and normalized to [0,1], so true blind spots get high confidence and already-explained areas are suppressed. Refinement is triggered automatically when the visibility score drops below 0.97, with at most 3 total feed-forward passes.

Training uses DL3DV-10K with 15-frame clips, a hybrid masking scheme mixing random box/brush masks and geometry-projected masks at a 0.4:0.6 ratio, two-stage fine-tuning (first the DPT and GS-DPT heads, then the learnable mask attention modules with the cross-view attention layers), 8 NVIDIA RTX A6000 GPUs, per-GPU batch size 2, gradient accumulation 2, and images resized to 280×504. A frozen copy of the pre-trained DA3 acts as a teacher, producing depth priors on the fly from unmasked views as pseudo ground truth, which the student matches through a masked L1 distillation loss. The total objective combines a photometric loss (MSE plus LPIPS) computed only on held-out novel views with the depth loss applied to both input and novel views. At inference, PowerPaint v1 supplies the 2D diffusion prior when a clean reference is unavailable, and Lang-SAM extracts object masks.

Why This Matters

The paper removes the multi-stage "run COLMAP first, then inpaint" pipeline that bottlenecks user-facing 3D editing. By folding pose estimation, geometry recovery, and inpainting into one feed-forward pass, it makes interactive editing from casual captures plausible.

• Virtual and augmented reality content creation: quickly removing unwanted objects or completing scenes from casual phone captures without a calibration step.
• Interactive gaming and scene authoring: near-instant edits where a 0.4-second pass (up to ~1.8 seconds with refinement) is compatible with interactive workflows, unlike 30-minute to 5-hour optimization runs.
• Real estate, retail, and 360-degree virtual tours: the 360-USID results show the method works in 360-degree settings where users want objects removed and backgrounds filled consistently behind them.
• Robotics and embodied AI simulation: generating complete, multi-view-consistent 3D scenes from sparse, unposed, partially occluded captures for downstream simulation or training data.
• Industry relevance: the approach is directly relevant to tooling vendors and platforms that need on-device or server-side scene editing without a fragile SfM preprocessing dependency, and it demonstrates that existing 3D foundation models can be adapted to generative completion rather than replaced.

Future Directions

• Raising robustness in extreme blind spots: the refinement loop saturates quickly (only +0.02 dB PSNR from the third to the fourth iteration), so better ways to supply missing evidence or to avoid relying on a single diffusion support view remain open.
• Removing the diffusion dependency at inference: support view generation currently depends on an external 2D diffusion model (PowerPaint v1 for references), and more tightly integrating or distilling that generative prior is a natural next step.
• Better confidence calibration for support tokens: the confidence map is currently a smoothed, normalized function of mask and rendered opacity; an explicitly learned reliability estimator could reduce the risk of injected artifacts.
• Broader evaluation scope: the main quantitative results cover SPIn-NeRF, 360-USID, and LLFF, with IMFine used only in appendix ablations; extending systematic benchmarking to more in-the-wild and dynamic scenes would test the claims more broadly.
• Training efficiency and data scaling: the paper reports no training wall-clock time or compute budget beyond the 8×A6000 setup, leaving open how the approach scales with larger training corpora and bigger backbones.

Target Audience

Researchers and graduate students working on 3D vision, neural rendering, and scene editing, particularly those interested in feed-forward reconstruction foundation models and pose-free pipelines. It is also useful for practitioners in AR/VR, gaming, and 360-degree content tools who need practical, fast 3D inpainting, and for engineers evaluating whether a feed-forward foundation model can replace a per-scene optimization stack. Readers should already understand vision transformers, 3D Gaussian Splatting, and diffusion models; the paper assumes all three.

Authors’ abstract

3D scene inpainting aims to recover missing or occluded regions in edited 3D scenes, while ensuring geometric and textural consistency. Existing approaches, however, typically require accurately calibrated camera poses, which restricts their applicability in casual, in-the-wild scenarios and introduces additional preprocessing overhead. To overcome this limitation, we present FreeInpaint, a novel feed-forward framework that generates complete and 3D-consistent scenes directly from unposed multi-view images with masked regions. At its core, FreeInpaint extends a 3D foundation model to propagate masked regions from a reference view to other unposed views, bridging 3D reconstruction and scene inpainting while preserving the model's native ability to recover camera poses and scene geometry. Our method addresses two key challenges in adapting feed-forward 3D foundation models to masked inputs. First, masked regions can corrupt cross-view correspondence reasoning, degrading pose estimation and geometry recovery. To address this, we introduce a Learnable Mask Attention mechanism that preserves the spatial anchoring of reliable observations while allowing masked regions to progressively absorb useful context in deeper layers. Second, under severe occlusions, a single forward pass often lacks sufficient appearance evidence for high-fidelity completion. Therefore, we propose a Support Token Refinement strategy, which injects diffusion-generated support evidence as confidence-weighted auxiliary tokens to refine under-observed regions while preserving the original spatial anchor. Extensive experiments across diverse datasets demonstrate that FreeInpaint achieves superior inpainting quality, eliminating the reliance on pre-computed camera poses while keeping a fast inference speed. The project page is https://rorisis.github.io/FreeInpaint/.