3D-Aware VLMs with Implicit and Explicit Geometries

AI summary

Overview

Research area: Computer vision and multimodal learning — specifically 3D-aware vision-language models (VLMs) that operate on RGB video only.

Technical level: Advanced. The paper assumes familiarity with VLM architectures, 3D geometry encoders, cross-attention, depth and point-map reconstruction, and 3D Gaussian splats.

Scope (one sentence): The paper proposes VLM-IE3D, a framework that injects both implicit and explicit 3D geometry derived from RGB video into a pretrained VLM to improve 3D detection, grounding, captioning, and spatial reasoning.

What This Paper Is About

Most VLMs are built on 2D visual inputs and struggle with 3D tasks that need fine-grained spatial understanding. Prior work tried to fix this by plugging a 3D geometry encoder into a VLM so it learns implicit 3D representations — but those representations are compressed and latent, capturing only coarse, global scene layout and leaving precise geometric detail inaccessible to the language model. This paper argues that VLMs need both kinds of geometry: the abstract, global "3D cognitive map" and the structured, measurable "3D reconstruction map," and builds a framework that supplies both from RGB video alone.

Key Contributions

1. VLM-IE3D framework: a unified framework that injects strong 3D inductive biases into a VLM using only RGB video sequences, with no additional 3D input at inference or training time.
2. Two complementary geometry representations: Implicit Geometry Tokens (IGTs), which encode high-level 3D priors from a 3D geometry encoder, and Explicit Geometry Tokens (EGTs), which encode detailed 3D structures from reconstructed explicit attributes such as depth maps, point maps, and 3D Gaussian splats.
3. A 3D-aware adapter: a fusion module, including an implicit–explicit attention (IEA) module, that aligns IGTs and EGTs and merges them with 2D visual tokens into unified 3D-aware visual embeddings.
4. Extensive evaluation: experiments across 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning showing competitive performance without 3D inputs, plus ablations over fusion strategies, explicit attribute types, embedding designs, and geometry encoders.

Main Findings

• 3D dense captioning (Scan2Cap): VLM-IE3D reaches 80.4 C@0.5 and 28.8 M@0.5 — the best C@0.5 among all compared methods in the table (including 3D-input methods such as Video-3D LLM at 80.0) and the best M@0.5 among methods with only 2D visual input. This is a 22.4 gain in C@0.5 over Qwen2.5-VL-3B (58.0) and 1.8 over VG LLM (78.6).
• 3D visual grounding (ScanRefer): VLM-IE3D achieves 43.2% Acc@0.25 and 16.9% Acc@0.50, improving 9.2 and 6.3 points respectively over Qwen2.5-VL-3B (34.0 / 10.6) and 6.8 / 5.1 over VG LLM (36.4 / 11.8). With proposal refinement it reaches 55.4% and 48.9%, narrowing the gap with 3D-scene-input methods. ScanRefer contains 36,665 object descriptions with axis-aligned bounding boxes across 562 indoor scans.
• 3D video detection: VLM-IE3D scores 44.2 P25, 41.9 R25, and 42.8 F1-25, gains of 12.1, 11.8, and 11.9 over Qwen2.5-VL-3B (32.1 / 30.1 / 30.9), and 2.5, 6.2, and 4.6 over VG LLM (41.7 / 35.7 / 38.2). The benchmark is curated from EmbodiedScan with 958 training and 243 evaluation scenes, four consecutive frames at 1 FPS, and 20 object categories, evaluated at an IoU threshold of 0.25.
• Spatial reasoning (VSI-Bench): VLM-IE3D-4B achieves the best average of 47.6%, above proprietary models including Gemini-1.5-Pro (45.4%) and much larger open models such as LLaVA-NeXT-Video-72B (40.9%). It scores 67.5% on object counting (1.5 above VG LLM) and 47.7% on relative distance, a +3.1 gain over VG LLM (44.6%).
• Small efficiency cost: VG LLM has 3.13B trainable parameters and VLM-IE3D has 3.23B — an addition of 0.10B, or 3.2%. Inference drops only 1 FPS (7 to 6 FPS on a single H100 GPU). The explicit embedding module itself adds just 0.008B parameters, 0.25% of the 3.23B trainable total.
• Both geometries matter, and they are complementary: starting from a 30.9 F1-25 baseline, EGTs alone raise it to 34.7 and IGTs alone to 40.5; combining them reaches 42.8 — 11.9 above baseline, 8.1 above the EGTs-only model, and 2.3 above the IGTs-only model. IGTs contribute more individually, which the authors attribute to their richer and more general 3D priors.
• The proposed fusion wins: IEA (42.8 F1-25) beats concatenation (41.5), element-wise addition (42.4), and learnable weighted addition (41.2). For merging 2D and 3D tokens, plain non-parametric addition was found sufficient.
• Choice of explicit attribute barely matters: point maps (42.6), depth maps (42.8), and 3D Gaussian splats (42.5) all land within a narrow F1-25 band of 42.5 to 42.8, suggesting any physically meaningful spatial measurement works as a complement. Depth maps are chosen as default because they are the most accessible and parameter-efficient.
• A lightweight explicit embedding beats a heavy one: removing it gives 40.5, average pooling gives 42.0, pooling with sinusoidal positional embedding gives 42.5, and the full method gives 42.8 — but treating depth maps as images with DepthAnything V2 as a deep encoder drops performance to 35.9. The authors hypothesize that deep encoders over-abstract features and overlap redundantly with IGTs.
• The framework generalizes across geometry encoders: π³ reaches 42.1 and VGGT 41.7 versus AnySplat at 42.8; using AnySplat geometry with DepthAnything V2 depth reaches 42.5, showing IGTs and EGTs can even come from separate models.

Methodology in Plain English

The system takes an RGB video and a natural language question and runs them through three parallel streams:

1. 2D visual tokens from the same 2D visual encoder used in Qwen2.5-VL.
2. Implicit Geometry Tokens (IGTs) from a pretrained 3D geometry encoder (AnySplat), taken from the output of its fusion decoder. These carry the coarse, global sense of how a scene is laid out — like knowing a bedroom's typical structure.
3. Explicit Geometry Tokens (EGTs), built by taking the reconstructed 3D attributes the geometry encoder already predicts (depth maps, camera poses, 3D Gaussian splats), back-projecting depth into 3D point maps using the reconstructed camera poses, and passing them through a deliberately simple one-layer patch embedding plus a two-layer MLP. The point is to preserve raw spatial measurements rather than abstract them away.

A 3D-aware adapter then does the joining. All three token types are spatially compressed by concatenating 2×2 neighboring features and passing them through a two-layer MLP. An implicit–explicit attention module then uses multi-head cross-attention with the implicit tokens as queries and the explicit tokens as keys and values, plus a residual connection, so explicit geometric detail can be written into the implicit representation. Finally the 3D tokens are added element-wise to the compressed 2D tokens, and the result goes into the VLM backbone alongside the text.

Training uses Qwen2.5-VL-3B as the backbone with the 2D visual encoder and the 3D geometry encoder frozen; only the VLM backbone and the 3D explicit embedding are trained. The setup is one epoch with Adam, a warmup ratio of 0.03, a learning rate rising to 1e-5 and decaying to zero, and batch size 1 per GPU across 8 H100 80G GPUs. Inputs are rescaled and cropped to 392×518, with patch size 14, channel dimension 2048, maximum sequence length 32, 1036 tokens before compression and 252 after. The first frame serves as the reference coordinate system for all tasks except 3D visual grounding, where boxes are in each frame's local coordinates. Models are trained separately for scene understanding versus spatial reasoning tasks for fair comparison.

Why This Matters

The work shows that a VLM does not need depth sensors or point-cloud inputs to gain useful 3D competence — it can extract the geometry it needs from ordinary video. That is significant because it removes a hardware and data-collection bottleneck that has held back 3D-capable VLMs, while keeping the extra compute cost small (0.10B extra parameters, 1 FPS slower inference). It also reframes the design question: rather than one geometry stream, the paper argues for pairing an abstract global geometry with a concrete measurable one, and demonstrates empirically that the two are complementary rather than redundant.

Real-world applications:

• Embodied navigation and robotics: the paper motivates its work through embodied navigation, vision-language-action systems, and robotic reliance on explicit depth and point-cloud representations for localization. RGB-only 3D grounding could let such systems derive geometric cues from cameras they already have.
• Indoor scene understanding and assistive tools: precise 3D bounding boxes and dense descriptions of objects in scans (ScanRefer, Scan2Cap) support applications that describe rooms or locate objects for a user.
• Video analytics and spatial question answering: 3D video detection across consecutive frames and VSI-Bench-style reasoning (object counting, relative distance and direction, route planning) map onto tasks like querying recorded footage about where things are and how far apart they sit.
• Cheap 3D annotation pipelines: since EGTs come from reconstructed attributes rather than ground-truth 3D, the approach can exploit ordinary RGB video collections for large-scale training.

Industry relevance: the authors are affiliated with Nanyang Technological University, DAMO Academy at Alibaba Group, HuPan Lab, and Alibaba Group, and code and models are released publicly. Progress toward 3D-capable VLMs that avoid specialized sensors is directly relevant to companies building robotics, embodied AI, AR/VR scene understanding, and video-based spatial assistants, where camera-only pipelines are far cheaper to deploy at scale.

Future Directions

• Better explicit geometry, not just more of it: results show the choice of explicit attribute (depth, point map, Gaussian splat) barely changes performance, and that a deep encoder hurts. What explicit representation would add genuine new information beyond what depth maps already provide remains open.
• Closing the remaining gap to 3D-input methods: even with proposal refinement, VLM-IE3D trails methods that receive explicit 3D scene input on precise localization, so better reconciliation of RGB-derived geometry with native 3D supervision is unresolved.
• Exploiting future geometry encoders: the ablation shows the framework works with π³, VGGT, and AnySplat, and even with depth from a different model, suggesting the design could absorb stronger 3D geometry encoders as they appear — the limits of that plug-in flexibility are not established.
• Scaling the backbone and data recipe: the reported model is 3B-class (listed as VLM-IE3D-4B on the spatial reasoning benchmark) trained for one epoch on partial data subsets (234K SPAR-7M samples and 63K LLaVA-Video samples), leaving open how the implicit-plus-explicit recipe behaves at much larger scale.

Target Audience

Researchers and practitioners working on multimodal foundation models, 3D scene understanding, and embodied or robotic AI will get the most from this paper. It is also useful for engineers who want 3D spatial capability from camera-only pipelines, and for readers already familiar with RGB-only spatial VLMs (such as VG LLM) who want to understand why an explicit geometry channel, fused with an implicit one, improves performance. Readers without grounding in VLM architectures and 3D reconstruction will find the technical sections demanding.

Authors’ abstract

Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.