GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models

AI summary

Overview

• Research area: Computer vision and multimodal learning — specifically spatial reasoning in vision-language models (VLMs) and the integration of geometry foundation models with VLMs.
• Technical level: Advanced. The paper assumes familiarity with VLM architecture, fine-tuning, token/embedding injection, and geometry foundation models such as Depth Anything 3.
• Scope: The paper introduces GeoReader, a two-stage framework that learns geometric tokens which a receiving VLM can both read and use for spatial question answering.

What This Paper Is About

VLMs handle general multimodal understanding well but still struggle with metric and multi-view spatial relationships that are only implicit in RGB images, such as camera motion, viewpoint change, and object distance. Geometry foundation models can recover this structure, but the paper argues that existing ways of connecting them to VLMs fail in different ways: fusing geometry features and learning from question answering alone may not teach the model to interpret those features, and aligning VLM latents with geometry features may capture geometry without the model ever using it. The goal is a geometric representation that is both readable and actually used by the answering model.

Key Contributions

1. A framing of the problem as an interface problem. The authors propose explicitly connecting geometry information to spatial relationships expressed in language, separating the step of adapting geometry to the VLM from the step of applying it to spatial reasoning.
2. The GeoReader framework. A two-stage pipeline in which lightweight Geometric Readers are first trained to produce continuous tokens supervised by geometry descriptions decoded by a frozen VLM, then frozen while the VLM and a question-conditioned geometry injector are fine-tuned for spatial QA.
3. Pair-supervised readability training. Instead of supervising descriptions of all views jointly (group supervision), the Reader encodes only a reference view and one target view, removing ambiguity about which token should carry each geometric relation.
4. Empirical validation with geometry interventions. Evaluation across seven spatial reasoning benchmarks and three VLM families, plus ablations on Reader training and freezing, injection design, text-versus-continuous geometry, and controlled interventions showing that the model's answers actually depend on the supplied geometry.

Main Findings

• Headline MindCube-Tiny result. With geometry predicted from RGB, GeoReader reaches 94.3% overall on MindCube-Tiny with Qwen3-VL-4B (Rot. 94.5, Among 96.2, Around 89.6). It achieves the highest accuracy in four of the five backbone blocks on that benchmark.
• Six-benchmark average. With a multi-source QA mixture, GeoReader averages 72.5% across six additional benchmarks with Qwen3.5-9B, the highest average in every backbone-size block. The 9B model scores VSI 68.6, SPBench 79.5, CV 87.7, SPAR 71.0, ViewSpatial 81.3, MMSI 47.1.
• Gains over 3DThinker at every backbone size on MindCube-Tiny. GeoReader-2B scores 86.5, GeoReader-3B 81.1, GeoReader-4B 94.3, GeoReader-7B 93.2, and GeoReader-8B 93.4 (in the Qwen3-VL-8B block, GAMSI-8B is the bold best at 94.1).
• Gains over vanilla SFT are concentrated on metric-geometry questions. With a matched Qwen2.5-VL-3B setup, GeoReader improves object and room size questions by +4.8 (1,862 questions), camera motion by +4.4 (967 questions), distance and depth by +2.2 (6,613 questions), and metric-geometry questions overall by +2.9 (9,442 questions), while all other questions change by only +0.3 (13,577 questions). Reported gains reach +14.3 on SPBench-SI relative distance and +13.5 on MMSI camera motion.
• Both training stages are necessary. Stage 1 alone without QA training scores 41.4%, QA training alone scores 54.7%, and combining them reaches 89.2%.
• Readability pretraining beats the alternatives. Readability pretraining reaches 76.4%, above QA-only training of a random Reader (54.7%) and feature alignment with a released 3DThinker checkpoint (62.7%, same backbone and questions).
• Pair supervision beats group supervision. With frozen Readers, pair supervision improves accuracy from 77.8% to 89.2%.
• Freezing the Readers matters. Tuning the Reader during QA training gives 76.4%; freezing the same pair-supervised Stage 1 Reader improves accuracy to 89.2%.
• The question-conditioned injector matters. Replacing it with direct injection gives 78.6% and a learned fixed scale gives 68.5%, neither matching the full injector at 89.2%. Adapted VLM-3R- and SpaceMind-style fusion on dense VGGT features achieve 86.1% and 81.1%, and adding annotated pose descriptors to image-token embeddings gives 73.8%.
• The model actually uses the geometry. In a controlled MindCube-Tiny experiment with annotation-derived camera geometry, zeroing GeoReader's geometric residual lowers accuracy from 89.2% to 77.2%, and shuffling poses across views drops it to 63.1%.
• Feature alignment does not. For the released feature-alignment checkpoint on the same questions, zeroing latents gives 62.9% and swapping them across questions gives 61.5%, both near its 62.7% baseline, indicating answers barely depend on the latents. Answer-token attention also attends more to GeoReader's geometric tokens than 3DThinker's tokens attend to its latents.
• Descriptions help more as Reader supervision than as text. As extra SFT targets descriptions score 67.0%, below vanilla SFT at 68.6%; generating them at inference yields 72.9% and supplying them in the prompt 81.0%, while GeoReader, using them only as Reader targets, achieves 89.2%.
• Token efficiency. With precomputed geometry, GeoReader generates 16.2 tokens in 0.5 s versus 119.3 tokens and 2.5 s when generating geometry descriptions.

Methodology in Plain English

The approach treats geometry as a translation problem. In Stage 1, a small "Geometric Reader" network takes a geometric signal for each view — a camera pose relative to view 1, or a depth map — and compresses it into a single continuous token per view on the same patch grid as the RGB tokens. These tokens are inserted into a frozen VLM along with the images and a fixed instruction, and the frozen VLM is asked to produce a text description of the geometry, such as "compared with image 1, the observer in image 2 moves 0.4 meters right and 0.1 meters backward, and turns left by about 23 degrees." Gradients flow through the frozen VLM to update only the Reader, so the Reader learns to express geometry in a form the target model already handles through its language abilities. Descriptions supervise only geometric content positions (distances, directions, angles), not template wording. To avoid ambiguity about which view's token carries a relation, supervision is applied pair by pair: for each target view, only the reference view and that target view are encoded.

In Stage 2, the Readers are frozen so this learned interface is preserved, and the VLM is fine-tuned on spatial QA. Because questions differ in how much they rely on each signal, a geometry injector first runs a "read pass" over images and question only, takes the final-layer hidden state of the last question token, and uses a routing head to compute a gate per signal. That gate scales the adapted Reader output, which is added to learned placeholder embeddings at the geometric-token positions. If a signal is missing, its gate is set to zero and the slot embeddings alone mark it absent. The answer pass reuses the read pass's key–value cache, and geometry descriptions are never generated at inference — the model answers directly from the continuous geometric tokens.

Training used about 65K geometry descriptions from 1,550 ScanNet and ScanNet++ training scenes in Stage 1 and roughly 135K multi-source QA examples, or 10K MindCube examples for ablations, in Stage 2, with no test-set examples. Readers train for 10 epochs and Stage 2 for 2 epochs on multi-source data or 10 on MindCube, at a learning rate of 2 × 10⁻⁵ with greedy decoding. Main models use Depth Anything 3 (the DA3NESTED-GIANT-LARGE-1.1 checkpoint) for both training and evaluation, while ablations use annotated geometry to exclude estimation error.

Why This Matters

Impact on research. The paper reframes a common assumption: that recovering geometry is the hard part. It shows a released feature-alignment model whose aligned latents can be zeroed without changing MindCube-Tiny accuracy, and contrasts this with a design where removing geometry drops accuracy from 89.2% to 77.2%. This suggests representation quality should be measured by whether the receiving model can use it, not by whether geometry can be reconstructed from it — a distinction relevant to latent-reasoning and geometry-augmented VLM work more broadly.

Real-world applications:

• Embodied agents and robots that need to reason about viewpoint changes, camera motion, and object distances from ordinary cameras.
• Navigation and spatial assistance systems that must relate objects across multiple views.
• Augmented and virtual reality applications where users move between views of a scene.
• Multimodal assistants answering metric questions about images, such as object size or relative position.

Industry relevance. The method requires only RGB at inference, keeps the VLM's input and output interfaces intact, and uses lightweight Readers that are trained once and then frozen. The reported token efficiency (16.2 tokens in 0.5 s versus 119.3 tokens and 2.5 s for generated descriptions) is a practical argument for continuous geometric tokens over text-based geometry reasoning in deployed systems. The paper also notes explicitly that benchmark performance does not establish reliability for safety-critical uses such as navigation or robotic control, which would require application-specific validation.

Future Directions

• Closing the gap between annotated and predicted geometry. Ablations use annotation-derived geometry, while main models use DA3 predictions, whose errors vary by source (evaluation AbsRel 5.9% on ScanNet versus 26.1% on SUN RGB-D training images). Quantifying how these estimation errors affect the readability and use of geometric tokens is an open question.
• Broadening beyond the evaluated domains. Matterport3D, nuScenes, and Omni3D images are omitted from the geometry error table because they lack pixel-aligned metric ground truth, leaving outdoor and large-scale scenes less characterized.
• Extending to signals and settings beyond pose and depth. The formulation is stated to apply to any per-view signal, but only camera pose and depth are demonstrated; other geometric signals and settings such as the single-image case where inter-view geometry is absent (handled by setting the gate to zero) are not explored in depth.
• Understanding failure modes. The paper acknowledges that predicted geometry and answers can be wrong, for example under reference-frame confusion, and points to additional successes and failures in the appendix; systematic characterization of these failure modes is left open.

Target Audience

Researchers and engineers working on vision-language models, multimodal spatial reasoning, and 3D/geometry-aware AI. It is most useful to readers already comfortable with VLM fine-tuning and embedding-level fusion who want to understand how to make external geometric information genuinely usable by a language model, and to practitioners deciding between text-based geometry supervision and continuous geometric tokens. Readers seeking a beginner-level introduction to spatial reasoning benchmarks would find the paper's dense benchmark tables and appendix-heavy detail demanding.

Authors’ abstract

Vision-language models (VLMs) struggle with metric and multi-view relationships that remain implicit in RGB observations. Geometry foundation models recover this structure from RGB, but existing methods either fuse the estimated features with visual tokens and learn to use them from question answering alone, or align VLM latents with geometry features. Capturing geometry, however, does not ensure its use: zeroing the aligned latents of a released feature-alignment model leaves its accuracy unchanged. We introduce GeoReader, a framework that learns geometric representations for the VLM that reads them. First, we train lightweight Geometric Readers to produce continuous input tokens, supervising them with geometry descriptions decoded by a frozen VLM, so that the Readers express geometry in a form the target model can already read. We then freeze the Readers to preserve this interface while fine-tuning the VLM on spatial QA, with geometry injection conditioned on the question. Geometry descriptions serve only as training targets: at inference, the VLM answers directly from the continuous geometric tokens without generating a geometry chain of thought. With geometry predicted from RGB images, our approach reaches 94.3 on MindCube-Tiny with Qwen3-VL-4B and, with a multi-source QA mixture, an average of 72.5 across six additional spatial benchmarks with Qwen3.5-9B. These results suggest that a useful geometric representation is one the receiving model can use, not merely one from which geometry can be reconstructed.