Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images

AI summary

Overview

Research area: Computer vision, specifically unified multimodal models (UMMs) for 3D scene understanding and generation from multiview images.

Technical level: Advanced. The paper assumes familiarity with multimodal large language models, diffusion/flow-matching generative models, novel view synthesis, and 3D scene benchmarks.

Scope: The paper introduces Omni-View, a unified 3D model built on BAGEL-7B that couples 3D scene understanding with texture generation (novel view synthesis) and geometry estimation (depth and camera pose), and shows that the generative tasks measurably improve the understanding performance.

What This Paper Is About

Unified models already combine 2D image understanding and generation, and prior work such as Metamorph showed that understanding can help generation. The reverse direction — whether generation can improve understanding — is underexplored, especially in 3D. This paper builds a single model that jointly performs 3D scene understanding, novel view synthesis, and geometry estimation from multiview images, testing the hypothesis that "generation facilitates understanding" because 3D understanding tasks inherently need the geometric and spatiotemporal modeling that these generative tasks provide.

Key Contributions

1. A unified 3D understanding-and-generation architecture. Omni-View splits the generation model into two specialized modules — a texture module for appearance synthesis and a geometry module for depth and camera pose — both wired into a shared understanding model, rather than using one monolithic generative pathway for both texture and geometry.
2. A two-stage, dense-to-sparse training recipe. Stage 1 trains the understanding model, texture module, and geometry module simultaneously, with the number of reference images progressively reduced across iterations (dense-to-sparse, D2S). Stage 2 freezes the understanding model and fine-tunes generation with joint RGB-Depth-Pose (RGBDP) learning.
3. Evidence that generative tasks transfer to understanding. Ablations isolate the separate contributions of texture generation, depth estimation, camera pose estimation, autoregressive generation, and the D2S curriculum on understanding benchmarks.
4. State-of-the-art unified results across three task families. Omni-View reports a VSI-Bench score of 55.4 while also producing novel view synthesis and 3D scene generation results, with code and pretrained models released at https://github.com/AIDC-AI/Omni-View.

Main Findings

• VSI-Bench state of the art: Omni-View-7B reaches an average of 55.4 on VSI-Bench, ahead of Spatial-MLLM-4B (48.4), BAGEL-7B-FT (46.3), VG-LLM-4B (46.1), Gemini-1.5 Pro API (45.4), and GPT-4o API (34.0).
• Gains concentrated in geometry- and time-dependent subtasks: Relative to Spatial-MLLM, Omni-View improves Absolute Distance by 11.6, Room Size by 9.6, and Relative Distance by 24.6. Relative to VG-LLM, it improves Absolute Distance by 9.8, Object Size by 13.4, Relative Distance by 25.1, Relative Direction by 11.0, and Appearance Order by 9.5.
• Strong 3D question answering and localization without 3D input: On SQA3D test, Omni-View scores 59.2 EM (61.9 EM-R); on ScanQA val, 103.0 CIDEr, 16.2 BLEU-4, 20.1 METEOR, 49.0 ROUGE, and 29.5 EM; on 3DOD, 46.4 F1; on ScanRefer, 50.8 Acc@0.25 (32.5 without proposal refinement) and 45.0 Acc@0.5 (9.9 without refinement).
• Improvement over the fine-tuned base model: Versus BAGEL-7B-FT, Omni-View gains 2 points EM on SQA3D and 7.5 CIDEr on ScanQA. On SQA3D it exceeds SpatialMLLM by 3.3 EM, and on ScanQA by 11.2 CIDEr.
• Generation performance: For novel view synthesis from a single view, Omni-View scores 23.22 PSNR, 0.817 SSIM, and 0.114 LPIPS, the best among compared methods. For scene generation, it scores 23.12 PSNR, 0.801 SSIM, and 0.146 LPIPS. The authors note that pixel-level fidelity improvements over popular scene generation models are slight, which they attribute to challenges in precise camera pose control.
• Texture module aids spatiotemporal tasks: Adding the texture module raises Appearance Order by 4.1 points in the ablation.
• Geometry module aids relative position tasks: Adding depth and camera pose estimation markedly improves Relative Distance, but improvements in absolute metric tasks such as Absolute Distance are constrained because the synthesized depth maps lack an absolute metric.
• Autoregressive generation helps: Using autoregressive generation instead of bidirectional generation boosts Absolute Distance by 5.8 and Appearance Order by 4.4.
• Dense-to-sparse beats alternatives: The D2S curriculum outperforms random masking (the Ross3D-style visual reconstruction approach), an all-dense reference set, and an all-sparse reference set across the reported benchmarks.
• Stage 2 improves generation: Without stage 2, novel view synthesis scores 21.90 PSNR / 0.705 SSIM / 0.265 LPIPS; with stage 2 it reaches 23.22 / 0.817 / 0.114.

Methodology in Plain English

The model starts from BAGEL-7B, a unified framework where understanding and generation share the same multimodal self-attention. Images are encoded with SigLIP, text with a frozen vocabulary, and the understanding model produces an answer by next-token prediction while also emitting intermediate features.

Because 3D generation needs both appearance and structure, the generative side is split in two. The texture module takes reference images (encoded with a FLUX-VAE encoder), a textual description, and a target camera pose, and synthesizes a novel view using flow matching, with camera pose injected through Plücker-Ray encoding following MV-AR. The geometry module takes the texture module's output latent, random depth noise, and a learnable camera pose query, and produces depth maps plus camera intrinsics and extrinsics decoded by VGGT's camera decoder; it also receives features from the understanding model through cross-attention, and gradients from geometry estimation flow back into the understanding model.

Training happens in two stages. In stage 1, all three components train together with a weighted sum of the understanding loss (next-token prediction), the texture loss (MSE on predicted noise), and the geometry loss (MSE on depth noise plus Huber loss on camera pose), with weights 1, 1, and 0.1. Diffusion forcing is used to improve 3D consistency. Across iterations the reference images shrink from all input images, to all except the first, to only the first — the dense-to-sparse curriculum. In stage 2, the understanding model is frozen and the generation side is fine-tuned with RGB-Depth-Pose joint learning, using point clouds reconstructed from a single reference image and its depth map as conditions (following Voyager), and the geometry module no longer uses understanding-model features as cross-attention conditions.

Training details: AdamW with β1 = 0.9 and β2 = 0.95, peak learning rate 1×10⁻⁵, warm-up covering 5% of iterations, one epoch over the understanding dataset. The geometry module has the same dimensions as the texture module with a depth of four layers. The understanding data is a filtered set of 780k valid items drawn from SQA3D, ScanQA, 3DOD, ScanRefer, VLM-3R, a 234k subset of SPAR, and a 64k subset of llava-hound4. Novel view synthesis training uses 61k video clips from re10k, with depth maps synthesized by the Voyager data pipeline and captions by QwenVLMax. Images from the scene understanding task are deliberately excluded from generation training, to show that the understanding gains do not come from memorizing understanding data. Inference uses 32 frames for 3D understanding, and VSI-Bench evaluation uses 8 to 32 frames at 640p resolution.

Why This Matters

The paper supplies concrete evidence for the "generation facilitates understanding" direction in the 3D setting, showing that depth-and-pose estimation and autoregressive view synthesis can raise the performance of a model that never receives explicit 3D input. It also shows a unified model can match specialist understanding models on question answering, while acknowledging a remaining gap on 3D grounding tasks.

Real-world applications implied by the work:

• Spatial assistants and robots that reason about object distances, sizes, and directions from ordinary multiview camera feeds without depth sensors.
• 3D content creation pipelines that synthesize consistent novel views of indoor scenes from few reference images.
• Scene reconstruction from video, using generated views and predicted depth/camera pose to build 3D representations such as 3D Gaussian Splatting.
• Mixed-reality or simulation environments where a viewer's viewpoint changes and unseen regions must be plausibly filled in.

Industry relevance: the work originates from Alibaba International Digital Commerce Group with academic collaborators, and the code and pretrained models are released, making it directly usable as a backbone for downstream spatial-intelligence products that rely on camera arrays rather than specialized 3D capture hardware.

Future Directions

• 3D visual grounding. The authors state that Omni-View's grounding capabilities remain to be substantiated and that methodologies without 3D scene input still lag those with it on 3D grounding tasks.
• Long-range world generation. The current generation model is described as lacking long-range world generation capability, to be pursued in future work.
• More accurate geometry prediction. The geometry module's training data is synthetic depth maps, so its real geometry prediction ability may not be accurate enough; the lack of an absolute metric in synthesized depth maps also constrains absolute distance tasks.
• Camera control and outdoor robustness. Failure cases show artifacts when outdoor scenes involve substantial camera movement, pointing to two needs the authors name: more precise camera control mechanisms and improved inter-frame texture consistency. Reinforcement learning is mentioned as a planned route for both grounding and long-range generation.

Target Audience

Researchers and engineers working on unified multimodal models, 3D scene understanding, spatial reasoning benchmarks, and novel view synthesis. It is most useful for readers already comfortable with multimodal LLM training and diffusion or flow-matching generative models who want to see how generative objectives can be repurposed to boost perception tasks, and for practitioners looking for an open-sourced 7B-scale backbone for multiview 3D applications.

Authors’ abstract

This paper presents Omni-View, which extends the unified multimodal understanding and generation to 3D scenes based on multiview images, exploring the principle that "generation facilitates understanding". Consisting of understanding model, texture module, and geometry module, Omni-View jointly models scene understanding, novel view synthesis, and geometry estimation, enabling synergistic interaction between 3D scene understanding and generation tasks. By design, it leverages the spatiotemporal modeling capabilities of its texture module responsible for appearance synthesis, alongside the explicit geometric constraints provided by its dedicated geometry module, thereby enriching the model's holistic understanding of 3D scenes. Trained with a two-stage strategy, Omni-View achieves a state-of-the-art score of 55.4 on the VSI-Bench benchmark, outperforming existing specialized 3D understanding models, while simultaneously delivering strong performance in both novel view synthesis and 3D scene generation. The code and pretraiend models are open-sourced at https://github.com/AIDC-AI/Omni-View.