WorldMirror: Universal 3D World Reconstruction with Any-Prior Prompting

AI summary

Overview

Research area: Computer vision, specifically feed-forward 3D scene reconstruction and multi-task geometric learning.

Technical level: Advanced. The summary below is written to be accessible, but the underlying work assumes familiarity with transformer architectures, multi-view geometry, point maps, and 3D Gaussian Splatting.

Scope: WorldMirror is a single feed-forward transformer that accepts multi-view images with any optional combination of geometric priors (camera intrinsics, camera poses, depth maps) and simultaneously outputs point clouds, depth maps, camera parameters, surface normals, and 3D Gaussians.

What This Paper Is About

Most 3D reconstruction models assume RGB images are the only available input, and each model typically targets one output — either depth, or camera pose, or point clouds, or novel view rendering. In practice, partial geometry is often already known from sensors or calibration. WorldMirror asks whether a single model can both absorb whatever geometric priors are available and produce every major 3D representation at once, without extra architectural branches or task-specific models.

Key Contributions

1. A unified any-prior, multi-task framework. WorldMirror is presented as the first feed-forward model to combine flexible prior conditioning (images plus optional intrinsics, poses, and depth) with comprehensive multi-task prediction (point maps, depth, cameras, normals, and 3D Gaussians) in one pass.

2. Multi-modal Tokenization. Rather than building modality-specific fusion modules, the authors convert every input type into tokens of a single sequence. Poses and intrinsics become compact single tokens that are concatenated with image tokens, while dense depth maps are patch-embedded and added element-wise to image tokens. Each prior token is independently dropped with probability 0.5 during training, so any subset of priors works at inference.

3. Unified Spatial Prediction with decoupled sequential training. A VGGT-style transformer backbone with DPT decoders handles geometry (point maps, depth, normals) and cameras, while a separate Gaussian head handles appearance. Training proceeds in three stages — prior-aware geometry, then normals, then a frozen-backbone 3DGS stage — because jointly optimizing rendering and geometry degrades both.

4. Broad state-of-the-art results and prior-injection synergy. The model outperforms VGGT, pi-cubed, StableNormal, GeoWizard, AnySplat, and DepthSplat across point map reconstruction, camera pose estimation, surface normal estimation, and novel view synthesis, and demonstrates that adding any prior improves every output, not just the one it directly informs.

Main Findings

• Priors help everything at once. Injecting camera poses, intrinsics, or depth does not merely improve the corresponding task; it universally improves all predictions. The authors attribute this to the shared spatial representation imposing 3D geometric consistency.

• Point map reconstruction leads without priors and pulls further ahead with them. On 7-Scenes, NRGBD, and DTU, the no-prior model already beats VGGT and pi-cubed (roughly 10.4 percent and 17.8 percent accuracy gains on 7-Scenes and DTU respectively). Using all three priors cuts 7-Scenes accuracy error from 0.043 to 0.018 and NRGBD error from 0.041 to 0.016.

• Camera pose estimation is state-of-the-art or competitive. WorldMirror reaches 99.99 RRA@30 and 95.81 RTA@30 on RealEstate10K, and the best ATE on TUM-dynamics (0.010). On the dynamic outdoor Sintel benchmark it trails pi-cubed, which the authors attribute to limited outdoor dynamic data in training.

• Surface normals improve substantially over both regression and diffusion baselines. Mean angular error on ScanNet drops to 13.8 degrees versus 16.0 for StableNormal, with similar margins on NYUv2 (15.1 vs 18.5) and iBims-1 (16.6 vs 17.9). This suggests a multi-task shared backbone can beat specialized single-task models.

• Novel view synthesis scales better with more views. On a multi-resolution DL3DV benchmark, the model outperforms DepthSplat at 8, 24, and 64 views, and the gap widens as views increase — 20.30 PSNR at 64 views versus 16.80 for DepthSplat. Gains also appear in post-optimization fine-tuning, where predicted point clouds make a better Gaussian initialization than random ones.

• Compact single-token priors beat dense per-pixel encodings. Ablations show a single pose or intrinsic token outperforms dense Plücker raymaps and raymap embeddings while using roughly six to nine times fewer extra parameters (1.06M versus 9.02M or 6.65M).

• Three design choices are essential for the Gaussian head. Rendering with ground-truth cameras instead of predicted ones, supervising both input and novel views, and letting the Gaussian head predict positions independently rather than reusing depth outputs each measurably improve rendering quality.

• Prior-guided competitors are surpassed. Against Pow3R (extended to multi-view via Procrustes alignment) and MapAnything, WorldMirror wins under most prior configurations, which the authors credit to multi-view-friendly embeddings and fine-tuning from VGGT weights.

Methodology in Plain English

The authors start from VGGT, an existing multi-view transformer, and add a front end that turns every kind of geometric input into tokens the same transformer can digest. Camera poses are converted into quaternions plus normalized translations and squashed into one token per view. Intrinsics become a small normalized 4D vector, also squashed into one token. These single tokens are simply concatenated onto the image token sequence. Depth is different: it is spatially dense, so the authors patch-embed it and add it directly to the image tokens, which keeps the sequence short and avoids quadratic attention blowup.

During training, each prior token is randomly zeroed out half the time. This teaches the model to work with whatever combination is available at test time — none, one, two, or all three — without needing separate models.

The outputs come from two families of decoder heads sharing the same backbone features. Geometry heads produce point maps, depth, and surface normals through DPT-style decoders, while a small MLP regresses cameras. Because ground-truth normals are scarce, the authors supplement labeled data with pseudo-normals computed from ground-truth depth via plane fitting.

Training is staged deliberately. First the prior-aware geometry tasks are trained from VGGT initialization. Then normal prediction is folded in. Finally, the backbone is frozen and only the Gaussian head is trained to render images. The authors found that training everything jointly confuses geometry and appearance learning. To keep the Gaussian head honest, rendering is supervised on both the input views and a set of held-out novel views, camera parameters are taken as ground truth during rendering rather than predicted, and the Gaussian head predicts its own positions so that rendering accuracy and geometric accuracy can trade off internally instead of corrupting one another. A voxel-based pruning step removes redundant overlapping Gaussians.

Why This Matters

Research impact. The paper argues that input flexibility and multi-task prediction are mutually reinforcing, and provides evidence: priors improve all tasks, and shared representations let one task benefit from another. This challenges the prevailing pattern of building one specialized model per 3D task and offers a concrete recipe — tokenize everything, train geometry and appearance in separate stages — for future generalist 3D foundation models.

Real-world applications:

• Augmented and virtual reality, where headsets and phones often have partial calibration or inertial data that WorldMirror can consume directly.
• Robotics and autonomous navigation, where depth sensors and known camera rigs are standard and dense 3D structure must be recovered in real time.
• Content creation and gaming, where the model converts real photos or AI-generated videos into editable 3D Gaussians and meshes without per-scene optimization.
• Scene understanding and spatial AI pipelines that need depth, normals, and cameras in one consistent coordinate frame rather than from four disconnected models.

Industry relevance. The work comes from Tencent's Hunyuan group with CUHK and Zhejiang University, and code plus weights are released publicly. Because a single feed-forward pass replaces a stack of task-specific models, deployment and maintenance costs drop substantially — a meaningful advantage for products that need on-device or low-latency 3D reconstruction. The paper's own impact statement flags privacy, misrepresentation, and bias risks as concerns to address as the technology spreads.

Future Directions

• Adding more input modalities. The tokenization scheme is task-agnostic by design; the authors note that optical flow or semantic masks could be incorporated by adding tokenizers without touching the architecture, and this remains untested.

• Closing the dynamic outdoor gap. Sintel performance lags the best baseline, pointing to a need for better handling of dynamic scenes and more outdoor training data.

• Reducing reliance on ground-truth cameras during 3DGS training. Rendering currently uses ground-truth camera parameters to prevent error accumulation, which limits fully pose-free training and suggests a joint pose-refinement approach.

• Extending beyond static scenes and single feed-forward passes. Scaling to longer sequences, streaming video input, and temporally consistent dynamic Gaussians are natural but unaddressed extensions, as is a deeper analysis of exactly why prior injection transfers gains across unrelated tasks.

Target Audience

Researchers and engineers working on 3D reconstruction, neural rendering, and multi-task vision foundation models will get the most from this paper. It is also valuable to practitioners in AR/VR, robotics, and content generation who need to know whether a single feed-forward model can replace their existing cascade of specialized geometry models. Readers without a background in multi-view geometry or transformer decoders will find the high-level framing accessible, but the architectural details and ablations assume intermediate to advanced familiarity with the field.

Authors’ abstract

We present WorldMirror, a unified feed-forward model for comprehensive 3D geometric prediction tasks. Unlike existing methods constrained to image-only inputs or customized for a specific task, our framework flexibly integrates diverse geometric priors, including camera poses, intrinsics, and depth maps, while simultaneously generating multiple 3D representations: dense point clouds, multi-view depth maps, camera parameters, surface normals, and 3D Gaussians. Remarkably, prior injection yields universal gains across all tasks, suggesting that input flexibility and multi-task prediction are mutually reinforcing. WorldMirror achieves state-of-the-art performance across diverse benchmarks from camera, point map, depth, and surface normal estimation to novel view synthesis, while maintaining the efficiency of feed-forward inference.