Transformer-Based Inpainting for Real-Time 3D Streaming in Sparse Multi-Camera Setups

AI summary

Overview

Research area: Computer Vision — multi-view 3D reconstruction, video inpainting, and real-time neural rendering for AR/VR streaming.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, rotary positional embeddings, attention mechanisms, generative adversarial training, and multi-view geometry (visual hulls, reprojection, novel view synthesis).

Scope: The paper proposes a transformer-based, multi-view-aware 2D video inpainting module that fills holes and missing textures in rendered novel views produced by a sparse multi-camera 3D streaming pipeline, evaluated on the DNARendering and RIFTCast datasets against DSTT, FuseFormer, E2FGVI, and RGVI.

What This Paper Is About

Real-time 3D streaming from a small number of cameras leaves gaps in the rendered images, because viewpoints that were never captured cannot be reconstructed. Existing pipelines fill these holes with simple heuristics, which produces inconsistencies and visual artifacts. This paper replaces those heuristics with a learned inpainting network that runs as an image-based post-processing step after novel view rendering, aiming to restore missing textures quickly enough for live streaming.

Key Contributions

1. A novel multi-view-aware transformer-based inpainting network designed for real-time video inpainting, usable as a general post-processing step in 3D streaming pipelines.
2. A spatio-temporal embedding that improves feature propagation from multi-view information by using a geometry proxy for reprojection of context patch coordinates into the target view.
3. A patch filtering strategy based on spatio-temporal locality (top-k token selection after the first transformer in each group) that adjusts how many patches are used during inference, allowing an explicit speed-versus-accuracy trade-off.
4. A public implementation released at https://github.com/vc-bonn/transformer-based-inpainting.

Main Findings

• Best overall quality/speed trade-off on DNARendering: The proposed method reaches a whole-image PSNR of 32.616, SSIM of 0.9851, LPIPS of 0.0262, and VFID of 1.6671, compared with 31.532/0.9827/0.0332/3.2139 for pretrained-default DSTT, 31.884/0.9832/0.0316/3.1830 for pretrained-default FuseFormer, and 31.834/0.9831/0.0320/3.1350 for pretrained-default E2FGVI.

• Large gains inside inpainted regions: In the inpainted regions the method reports PSNR 42.184, SSIM 0.99911, and LPIPS 0.00224, versus 36.051/0.9967/0.0109 (DSTT), 37.361/0.9975/0.0095 (FuseFormer), and 37.535/0.9976/0.0092 (E2FGVI) on the RIFTCast dataset, and versus 35.091/0.99733/0.00761 (DSTT pretrained-default) on DNARendering.

• Fastest inference among compared methods: The method runs at 41.55 FPS on DNARendering, compared with 12.82 FPS for DSTT (pretrained, def), 8.64 FPS for FuseFormer (pretrained, def), and 6.14 FPS for E2FGVI (pretrained, def), while the offline method RGVI runs at 3.09 FPS.

• Competitive with an offline reference: RGVI, which is an offline method inferred with ground-truth frames from the target view up to the current inpainting frame, achieves a higher inpainted-region PSNR (42.582) and VFID (1.6203) than the proposed method, but a worse inpainted-region SSIM (0.99834) and LPIPS (0.00678).

• Multi-view variants improve baselines but cost speed: Giving baselines multiple views (the "mul" variant) improves their perceptual quality but slows inference substantially — for example, pretrained DSTT drops from 12.82 FPS (def) to 1.71 FPS (mul), and pretrained E2FGVI from 6.14 to 0.90 FPS.

• Fine-tuning baselines did not help: Fine-tuning DSTT, FuseFormer, and E2FGVI on the same DNARendering subset used to train the proposed model did not yield improvements and in most metrics resulted in a slight decline in quality.

• Generalization without retraining: On the RIFTCast dataset, which the authors describe as substantially harder (multiple actors, human-object interactions, animals, complex occlusions, extensive subject movement), the model without any retraining or fine-tuning reaches a whole-image PSNR of 33.059, LPIPS of 0.0287, VFID of 0.9953, and inpainted-region PSNR/SSIM/LPIPS of 42.192/0.9989/0.0031. The reduction in inference speed to 37.01 FPS is attributed to the higher image resolution of that dataset.

• Ablation confirms each component matters: Removing components lowers inpainted-region PSNR from 42.184 (full model) to 40.004 (single camera only), 39.838 (no temporal context), 38.920 (no RoPE), and 34.426 (single camera without masks). Notably, the "Single cam w/o masks" variant is much faster (87.46 FPS) but far less accurate.

• Failure mode under fast motion: Output quality degrades when the assumption that screen-space patch coordinates from past frames can be reprojected using the current geometry proxy no longer holds, as happens with fast-moving content. Inaccurate foreground masks in the dataset add further difficulty.

Methodology in Plain English

The authors treat inpainting as a post-processing step that sits on top of an existing streaming pipeline and does not care how the underlying 3D representation was built. At each timestep the pipeline supplies a rendered novel view, the original camera images, a geometry proxy, foreground masks, and an error map marking which pixels need repair.

The system encodes both the target view and the context images (original camera views plus past frames) into feature maps using a CNN encoder. Feature maps are cut into small overlapping patches; patches that are entirely background are thrown away, and patches from the target view that contain no missing pixels are moved over to the context set. Each surviving patch carries a spatio-temporal coordinate describing where and when it came from, and context patches are reprojected into the target camera's view through the geometry proxy so the network knows where they land.

A stack of transformer groups then updates the patches that need inpainting by attending to the context patches, with coordinates injected as a 3D rotary positional embedding so attention can reason about relative position in space and time without computing pairwise distances. After the first transformer in each group, a top-k filter keeps only the most relevant context tokens, which is where the speed comes from. A decoder turns each updated patch back into RGB, patches are blended back into place, and the result is combined with the original rendered image using the error mask. Training combines L1 reconstruction losses on the intermediate image with an adversarial loss on the final output; no explicit cross-view consistency loss is used.

Why This Matters

Impact on research: The work reframes inpainting for 3D streaming as a 2D post-processing problem that is deliberately decoupled from the 3D representation, and shows that reprojection-aware spatio-temporal attention over original camera views can outperform both online video inpainting baselines and, in several perceptual metrics, an offline reference. It also demonstrates that resolution-independent, multi-view-aware design can transfer to a harder dataset without retraining.

Real-world applications:

• Telepresence and immersive remote communication, where low-latency correction of rendered frames is critical.
• Live entertainment broadcasting such as sports, concerts, arts, and film, where sparse camera rigs must produce complete views.
• AR/VR experiences on consumer hardware, where real-time constraints force the use of few input views.
• Medical applications, which the paper names among the domains that template-free 3D streaming could serve.

Industry relevance: The method is designed as a standalone module compatible with any calibrated multi-camera system, reports 41.55 FPS on DNARendering and 37.01 FPS on RIFTCast, explicitly targets consumer hardware and AR/VR devices, and is released as open source — all of which lower the barrier to integrating it into existing streaming products. The comparison against offline RGVI at 3.09 FPS frames the central commercial trade-off between quality and throughput.

Future Directions

• Handling fast motion, where the reprojection assumption in Equation 2 breaks down and quality drops; the paper identifies this as its main observed failure case.
• Reducing the dependence on accurate foreground masks and error maps, since mask inaccuracies in the ground-truth data were shown to create artifacts (visible as green screen leakage between arms).
• Extending beyond human-centric, foreground-matted sequences, which the paper names as the specific focus of its evaluation.
• Further exploring the memory-versus-computation trade-off in feature caching, which the paper notes can significantly reduce recomputation if enough memory is available.

Target Audience

Researchers and engineers working on multi-view 3D reconstruction, real-time neural rendering, and video inpainting; practitioners building telepresence or AR/VR streaming systems on sparse camera rigs; and readers with a background in transformer architectures and multi-view geometry who want to understand how learned inpainting can be made fast enough for live pipelines.

Authors’ abstract

High-quality 3D streaming from multiple cameras is crucial for immersive experiences in many AR/VR applications. The limited number of views - often due to real-time constraints - leads to missing information and incomplete surfaces in the rendered images. Existing approaches typically rely on simple heuristics for the hole filling, which can result in inconsistencies or visual artifacts. We propose to complete the missing textures using a novel, application-targeted inpainting method independent of the underlying representation as an image-based post-processing step after the novel view rendering. The method is designed as a standalone module compatible with any calibrated multi-camera system. For this we introduce a multi-view aware, transformer-based network architecture using spatio-temporal embeddings to ensure consistency across frames while preserving fine details. Additionally, our resolution-independent design allows adaptation to different camera setups, while an adaptive patch selection strategy balances inference speed and quality, allowing real-time performance. We evaluate our approach against state-of-the-art inpainting techniques under the same real-time constraints and demonstrate that our model achieves the best trade-off between quality and speed, outperforming competitors in both image and video-based metrics.