G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning

AI summary

Overview

Research area: Computer vision and multimodal foundation models, specifically spatial intelligence — the intersection of feed-forward 3D reconstruction and spatial understanding/reasoning in vision-language models.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, Mixture-of-Experts, 3D reconstruction formulations (point maps, camera poses), and spatial reasoning benchmarks.

Scope: The paper presents G$^2$VLM, a 2B-parameter vision-language model that unifies 3D visual geometry prediction with spatial understanding and reasoning inside a single architecture with two specialized experts.

What This Paper Is About

Current vision-language models are weak at spatial understanding because they learn about the physical world only implicitly, by training on large unstructured 2D image-text data — they never explicitly learn the visual geometry that lifts 2D pixels into a coherent 3D representation. The authors propose G$^2$VLM, a geometry grounded vision-language model that natively predicts 3D attributes (point maps and camera poses) and uses those learned geometry features to improve spatial reasoning through in-context learning and interleaved reasoning. The goal is a single model that is competent at both low-level 3D reconstruction and high-level spatial question answering.

Key Contributions

1. A unified model. G$^2$VLM is described as the first unified model that bridges spatial 3D reconstruction and high-level spatial understanding within a single vision-language model.
2. A dual-expert architecture. The model uses a Mixture-of-Transformer-Experts design with a geometric perception expert ("where pathway") for visual geometry learning and a semantic perception expert ("what pathway") for multimodal understanding, interacting through shared self-attention.
3. Learning 3D geometry from 2D data. The design lets the model learn to reason about 3D geometry from pure 2D image inputs, removing reliance on hard-to-collect 3D annotations and allowing scaling on in-the-wild multi-view images and videos.
4. Competitive results on both task families. G$^2$VLM achieves results comparable to state-of-the-art feed-forward 3D reconstruction models and better or competitive results across spatial understanding and reasoning tasks, at a 2B parameter size.

Main Findings

• Depth estimation improvement over VGGT. On the Sintel benchmark, G$^2$VLM reduces monocular depth estimation Absolute Relative Error from VGGT's 0.335 to 0.297. The paper reports $\pi^3$ at 0.277 on the same metric, and describes G$^2$VLM's overall depth performance as on par with state-of-the-art multi-frame feed-forward reconstruction approaches.
• Point map estimation. On ETH3D, G$^2$VLM reports Accuracy 0.414 and Completion 0.309 (VGGT: 0.28 and 0.305; $\pi^3$: 0.194 and 0.210). On 7-Scenes, G$^2$VLM reports Accuracy 0.046 and Completion 0.029 (VGGT: 0.022 and 0.026; CUT3R: 0.023 and 0.027). The paper characterizes this as on-par completion performance with VGGT and comparable accuracy.
• Camera pose estimation. On Co3Dv2, G$^2$VLM reports RRA@30 of 97.91, RTA@30 of 95.20, and AUC@30 of 74.81, against VGGT's 98.96, 97.13 and 88.59, and $\pi^3$'s 99.05, 97.33 and 88.41. The authors describe the RRA and RTA results as on-par and the AUC result as comparable, noting the model does not use camera tokens or require fine-tuning from pretrained weights.
• Best SPAR-Bench result. G$^2$VLM-SR achieves the best results on SPAR-Bench among all existing works, scoring 54.87 against GPT-4o's 36.39 — a gap of 18.48 points (stated as 18.5 in the abstract). The paper also reports that it surpasses human performance in the low-difficulty category of SPAR-Bench.
• Broad spatial benchmark strength at small scale. G$^2$VLM-SR achieves the best results on all four spatial benchmarks when compared with spatial expert models, despite its 2B size, and the best results among open-source models on SPAR-Bench, MindCube, and OmniSpatial. On OST-Bench, the much larger Qwen2.5-VL-72B performs best, which the authors attribute to online spatio-temporal scene understanding requiring models to store significant knowledge.
• Large gains over the base model. Both G$^2$VLM and G$^2$VLM-SR show significant improvement over their base model Qwen2-VL-2B across all tasks.
• Dual encoders beat a single encoder. The dual-encoder design using DINO for visual geometry and CLIP for multimodal understanding gives the best performance on both visual geometry and spatial understanding tasks.
• Global attention is best for the geometry expert. Among frame attention, global attention, and mixed attention mask variants, global attention consistently outperforms the others and is compatible with LLM frameworks.
• Geometry and reasoning reinforce each other. The global attention design, which is best for geometry, also yields the strongest spatial reasoning results — confirming a positive interplay where better geometric features produce greater spatial reasoning improvements.
• Geometry pretraining is essential. The full model significantly outperforms a variant finetuned only on spatial understanding data, showing the learned visual geometry representations are essential to the model's effectiveness.

Methodology in Plain English

The model takes a sequence of RGB images as input and splits the work between two transformer experts that share attention. A geometric perception expert uses a DINOv2 vision encoder and lightweight 3D geometry heads to predict a pixel-aligned 3D point map and a camera pose for each image. A semantic perception expert is built on a pretrained vision-language model (Qwen2-VL-2B) and handles multimodal understanding and question answering.

Training happens in two stages. In stage one, the semantic expert is frozen with its pretrained Qwen2-VL weights, and the geometric expert is trained from scratch on large-scale 3D-annotated datasets using a visual geometry loss that is a weighted sum of a point reconstruction loss, a camera pose loss (rotation plus translation), and a normal loss. In stage two, the semantic expert is unfrozen and jointly trained on spatial understanding data using standard cross-entropy language modeling loss.

The authors test three joint-training variants. CE Loss Only freezes the geometric expert and updates only the semantic expert, which preserves geometry performance and scales using abundant video data; this is chosen for the main G$^2$VLM. CE + CE Loss fine-tunes the geometry expert with language loss and is the most effective specifically for spatial reasoning, so it is released as G$^2$VLM-SR. VG + CE Loss supervises the geometry expert with both losses and yields the strongest results on both tasks, but requires a large-scale 3D annotated dataset and therefore limits scalability. The study was conducted on the ScanNet dataset with semantic annotations.

Geometric pretraining itself is done in two sub-stages: 100K iterations at 224x224 resolution with a learning rate of 2e-4 under a cosine scheduler, then 20K steps at a learning rate of 5e-4 at 518x518 resolution with randomized aspect ratio between 0.5 and 1.0. Each batch samples 2 to 24 frames from a random training scene, following VGGT. The low-resolution stage ran on 32 A800 GPUs over 7 days and the high-resolution stage on 64 A800 GPUs over 3 days. Joint training used AdamW for 16K iterations at a learning rate of 2e-5 on 64 A800 GPUs over 3 days. Gradient norm clipping with a threshold of 1.0, bfloat16 precision, and gradient checkpointing were used throughout.

Architecturally, both experts have 28 layers mirroring Qwen2-VL-2B. The model omits register tokens, uses only global attention layers, and removes the camera token design used in VGGT in favor of a permutation-equivariant design as in $\pi^3$. A linear projection layer maps DINOv2 features into the expert's input space. Geometry heads are lightweight 5-layer transformer decoders for camera poses, local point maps, and global point maps; the global point head only stabilizes training and is excluded at inference.

Why This Matters

Impact on research. The paper offers a concrete alternative to the standard approach of

Authors’ abstract

Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.