SplatReasoner: Enhancing Embodied Reasoning and Grounding by Novel View Synthesis

AI summary

arXiv: 2601.13132v2 [cs.CV] · Published: 2026-01-19 (v2: 13 Jul 2026) Authors: Kim Yu-Ji, Dahye Lee, Kim Jun-Seong, Nam Hyeon-Woo, GeonU Kim, Yongjin Kwon, Yu-Chiang Frank Wang, Jaesung Choe, Tae-Hyun Oh Affiliations: POSTECH, KAIST, ETRI, NVIDIA · Project page: https://splatreasoner.github.io/

Overview

Research area: Computer vision / embodied AI — combining Vision-Language Models (VLMs), 3D Gaussian Splatting (3DGS), and novel view synthesis for embodied question answering and 3D visual grounding.

Technical level: Advanced. The paper assumes familiarity with 3DGS, CLIP-style language embeddings, and VLM prompting pipelines, though the core idea is intuitive.

Scope: This paper proposes a framework that lets a VLM decide which camera viewpoints to render from a 3D Gaussian scene in order to answer a question and localize the objects it refers to, instead of being limited to a fixed set of pre-recorded images.

What This Paper Is About

Embodied agents typically reason over a fixed episodic memory of RGB-D images captured from whatever viewpoints happened to be recorded. Those views may be poorly composed — objects occluded, truncated at image boundaries, or simply not visible. SplatReasoner addresses this by rebuilding the scene as language-embedded 3D Gaussians, retrieving the Gaussians relevant to a query, and then synthesizing and selecting new viewpoints that actually reveal the evidence needed, before passing the chosen views to a VLM for answering and 3D grounding.

Key Contributions

1. A framework (SplatReasoner) that integrates novel view synthesis of 3DGS into the VLM reasoning loop for embodied reasoning and 3D grounding — injecting synthesized views rather than relying only on pre-captured observations.

2. A VLM-as-Judge mechanism over synthesized novel views, which the authors argue is critically important for spatial reasoning tasks and which overcomes the strict viewpoint limitations of existing fixed-view memory approaches.

3. A visibility-based initial view selection operating directly in 3D Gaussian space: an LLM extracts "evidence categories" from the query, matching Gaussians are activated, HDBSCAN clusters them into instance-level candidates, and a visibility score (which accounts for occluding Gaussians) picks the best input pose per cluster. The authors state this avoids 2D-pixel-space biases from occlusion and camera distance.

4. Strong empirical results on embodied question answering and precise 3D visual grounding using two memory representations: a closed-set category label per Gaussian (following 3D-Mem) for embodied reasoning, and an open-set CLIP feature per Gaussian (following Dr. Splat) for 3D localization and referring segmentation.

Main Findings

• Embodied question answering (EM-EQA) — best overall: SplatReasoner reaches 57.8 LLM-Match with a closed model (GPT-4o) and 51.6 with an open model (Qwen3-VL-8B), using 2.6 average frames. This compares to 3D-Mem at 54.6 / 50.8 with 2.7 frames, Multi-Frame at 49.1 / 48.2 with 3.0 frames, SVM Captions 38.9, CG Captions 36.5, BlindLLM 34.8 / 29.4 with 0 frames, and Frame Captions 24.1 / 31.7 with 0 frames. The Human score is 86.8.

• Text-only memory is insufficient: BlindLLM and the caption-based agents score lowest, which the authors read as evidence that textual abstraction alone cannot support embodied reasoning.

• 3D object localization (ScanRefer, single target): SplatReasoner scores 11.12 3D mIoU / 35.84 Acc@8 / 32.75 Acc@10, versus Dr. Splat (category) 8.73 / 34.53 / 28.53, Dr. Splat (sentence) 9.44 / 34.28 / 29.35, and ReferSplat 3.14 / 10.68 / 7.32.

• Multi-target localization (Multi3DRefer): SplatReasoner scores 6.44 3D mIoU / 31.24 Acc@8 / 25.27 Acc@10, versus Dr. Splat (category) 4.76 / 18.75 / 13.94, Dr. Splat (sentence) 4.19 / 15.70 / 13.04, and ReferSplat 1.27 / 2.19 / 1.40. The Acc@8 gap here is the largest reported margin.

• 3D referring segmentation: SplatReasoner scores 12.46 3D mIoU / 45.14 Acc@8 / 45.14 Acc@10, versus Dr. Splat (category) 10.03 / 41.71 / 33.86, Dr. Splat (sentence) 10.56 / 45.21 / 43.21, and ReferSplat 2.34 / 2.04 / 0.00.

• Visibility score beats volume score (184 questions): visibility score 48.2 LLM-Match at 2.7 frames vs. volume score 47.8 at 2.7 frames vs. 3D-Mem 45.4 at 3.1 frames. The authors attribute the volume score's weakness to large outlier Gaussians dominating the metric.

• Each pipeline stage adds value (184 questions): 3D-Mem 45.4, initial view selection alone 48.2, final view selection 50.5 (all at 3.1 → 2.7 frames).

• Text-based judging beats image-based judging (184 questions): Text-only VLM-as-Judge 50.5, image-only 47.1, image + text 46.9 — the authors note their choice is also more computationally efficient and lightweight.

• Verification matters (1,623 questions): final view alone 54.5, final view + verification 57.8 (both at 2.6 frames).

• Model-scale caveat: the performance gap with the smaller open-source Qwen3-VL-8B is described as "relatively constrained"; the authors state they show in the supplementary material that the gap widens significantly with more capable, larger-scale models.

Methodology in Plain English

The pipeline has four stages.

1. Build a searchable 3D memory. The scene is reconstructed as a set of 3D Gaussians (the same primitive used in 3D Gaussian Splatting). Each Gaussian gets a semantic tag. For embodied reasoning, this is a closed-set category label from an object detector, following 3D-Mem. For 3D grounding, it is an open-set CLIP feature produced by segmenting images with SAM and registering each segment's feature onto the Gaussians, following Dr. Splat. This yields a memory that can be searched directly in 3D rather than through pixels.

2. Initial view selection. An LLM reads the query and extracts a short list of "evidence categories" — e.g. for "Where can I take a nap?" it returns pillow, cushion out of a larger vocabulary. Gaussians matching those categories are activated (for grounding, by cosine similarity against the category's text embedding with a threshold of 0.5). HDBSCAN clusters the activated Gaussians spatially into instance-level candidates. For each cluster, the system renders a cluster activation map from every input pose and computes a visibility score = the fraction of cluster Gaussians that are the dominant contributor at their pixel. This means an occluded target does not receive credit, unlike simply counting projected Gaussians. The highest-scoring input pose becomes that cluster's initial view.

3. Novel view synthesis. Around each initial pose, the system renders V = 4 new viewpoints corresponding to left, right, forward, and backward perturbations, producing L × V synthesized views. Each initial pose plus its novel views forms a local candidate group.

4. Final view selection and verification. Within each candidate group, every rendered image is fed separately to a VLM together with the query — deliberately one at a time, since VLMs struggle with long visual sequences. Each view yields a textual answer candidate; an LLM then compares the text candidates within the group and picks the answer that best addresses the query, and that view becomes the selected view. A verification step compares answers from the initial and final views and prioritizes the final view, especially when the initial view produced no definitive answer (e.g. "I don't know"). For EQA, all L final views go to the VLM. For grounding, the VLM also returns the image indices of the referred targets: L′ = 1 for single-target localization, and L′ ≤ L determined dynamically for multi-target. Grounding itself is done by intersecting the activated Gaussians' 3D centers with the viewing frustums of the task poses.

Why This Matters

Impact on research. The paper argues that fixed-view episodic memory is a structural bottleneck, not just a data bottleneck — no amount of better VLM prompting fixes an occluded camera angle. It also positions itself against language-embedded 3DGS methods (Dr. Splat, ReferSplat) that rely on static embedding similarity, which the authors say handles "chair" but fails on "What color are pillows in the kitchen?" because reasoning happens at the level of individual Gaussians without iterative grounding or contextual aggregation. Notably, SplatReasoner uses 2D-pretrained VLMs and requires no additional fine-tuning, in contrast to 3D VLM approaches.

Real-world applications (as the paper frames them):

• Smart glasses and assistive agents that answer questions from a history of past observations — the paper's stated motivating EM-EQA scenario.
• Embodied exploration and navigation, where the authors note fine-grained localization is particularly important for precisely reflecting user queries.
• Home robotics and service assistants that must resolve spatial references like "the pillow in the kitchen" or "the bed closest to the window."
• Augmented-reality and scene-understanding interfaces that need to point at, or segment, a named object in 3D.

Industry relevance. The work targets multimodal assistants and robotics stacks that already have 3D reconstruction pipelines. Efficient frame use matters: SplatReasoner matches or beats baselines while using 2.6 average frames versus 3D-Mem's 2.7 and Multi-Frame's 3.0, and its text-only judging scheme is described as computationally lighter than image-based judging. Support from IITP, NRF, and InnoCORE grants indicates national Korean research-program backing.

Future Directions

• Active, long-horizon exploration. The paper explicitly states the system is designed for passive episodic-memory scenarios rather than active robotic navigation, and names sequential decision-making for active exploration as a promising future avenue.
• Scaling to larger VLMs. The authors report that the improvement gap over baselines is constrained with Qwen3-VL-8B and state the gap widens with more capable, larger-scale models — an open question is how far the method scales.
• Better linking of language and Gaussian features. The paper notes ReferSplat's rendering-based feature updates are poorly suited to direct 3D search; improving the alignment between fine-grained language and Gaussian features remains an open problem.
• Benchmark infrastructure for 3DGS grounding. Because existing benchmarks predominantly use point clouds, the authors had to curate their own 3DGS-based evaluation, manually annotate targets for question-form referring segmentation, and generate 49 spatial queries — suggesting a need for standard 3DGS-native grounding benchmarks.

Target Audience

Researchers and practitioners in embodied AI, 3D scene understanding, and multimodal reasoning — particularly those working on VLM-based agents, language-embedded 3D representations (3DGS, NeRF), or visual grounding. It is also relevant to robotics and AR/VR engineers who build agent memory systems on top of RGB-D capture and need to connect natural-language queries to 3D locations. Readers without a background in 3D Gaussian Splatting or CLIP-based feature lifting will need to read the related-work section first, as the method builds directly on those two lines of prior work.

Authors’ abstract

Vision-Language Models (VLMs) have demonstrated strong reasoning capabilities over images and videos, yet their application to embodied scene understanding often constrained by the fixed viewpoints stored in episodic RGB-D memories. These observations may fail to capture query-relevant evidence due to occlusions, object truncation, restricted fields of view, or suboptimal view composition. We present SplatReasoner, a framework that introduces novel view synthesis into the VLM reasoning process by leveraging 3D Gaussian Splatting (3DGS). Given a user query about a 3D scene, SplatReasoner retrieves relevant observations and synthesizes query-conditioned viewpoints that reveal the visual evidence needed to answer the query and ground the referred entities in 3D. Experiments show that query-conditioned novel view synthesis improves both embodied reasoning and 3D grounding over fixed-view memory and language-embedded 3DGS baselines.