RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generation

AI summary

Overview

• Research area: Computer vision and robotics, specifically open-vocabulary 3D scene graph (3DSG) generation, uncertainty estimation, and retrieval-augmented generation (RAG) with vision-language models.
• Technical level: Advanced (assumes familiarity with 3D reconstruction, CLIP/SAM-style foundation models, VLMs, LLMs, and scene graph evaluation protocols).
• Scope: This paper proposes RAG-3DSG, a pipeline that diagnoses and then rectifies semantic noise in multi-view 3D scene graphs by virtually "re-shooting" objects from optimal viewpoints and using low-uncertainty objects as retrieval anchors for a VLM.

What This Paper Is About

Open-vocabulary 3D scene graphs let robots represent a scene as objects (nodes) and their relationships (edges), but existing pipelines blindly aggregate per-image captions and embeddings across views, so occlusions and constrained viewpoints introduce semantic inconsistency—the same object gets conflicting descriptions. RAG-3DSG addresses this by first quantifying how ambiguous each object's semantics are, then using reliable objects as context to correct the unreliable ones. The goal is a cleaner, more trustworthy 3D scene graph for downstream robotics tasks.

Key Contributions

1. Re-shot guided uncertainty estimation: A mechanism that renders a synthetic "best view" of each reconstructed object from a sampled hemisphere and compares the resulting caption against the original passive crop captions to quantify that object's semantic ambiguity.
2. Object-level RAG for contextual rectification: Low-uncertainty objects are gathered into a retrieval document and used as semantic anchors; high-uncertainty objects trigger retrieval of the nearest anchor caption plus a concatenated re-shot/crop image prompt to refine the caption.
3. A complete three-stage 3DSG pipeline: Cross-image object mapping with scale-adaptive dynamic downsampling, node caption generation via the diagnose-then-rectify mechanism, and LLM-based edge caption generation over a constrained taxonomy of eight relation categories.
4. Extensive validation: Evaluation on SceneFun3D (20 scenes) and FunGraph3D (24 scenes) under the OpenFunGraph protocol, a fine-grained human evaluation on 8 Replica scenes with three expert annotators, ablation studies, a noisy-camera-pose stress test, and a real-world mobile robot deployment.

Main Findings

• Object classification gains: Ours-GPT reaches 90.2% R@10 on SceneFun3D, above OpenFunGraph's 87.8%. On FunGraph3D, Ours-GPT reaches 75.0% R@3 and 83.0% R@10, exceeding the prior state of the art by 4.3% and 3.9% respectively.
• Small local model competes: Ours-LLaVA (LLaVA-v1.5-7b) surpasses OpenFunGraph on FunGraph3D object R@3, 73.6% vs. 70.7%.
• Edge prediction improvement: Ours-GPT achieves 97.9% R@10 on SceneFun3D edges (vs. OpenFunGraph's 96.2%) and 65.7% R@5 on FunGraph3D edges (vs. 65.1%).
• Node quality drives edge gains: The Ours Node + CG Edge variant reaches 95.6% R@10 edges on SceneFun3D versus 97.9% for the full model, despite using the vanilla ConceptGraphs edge builder.
• Isolated re-shots fail: Using the re-shot caption alone without RAG or image concatenation (Re-shot VLM) drops to 59.4% Object R@3 on SceneFun3D, showing the re-shot and RAG components must work together.
• Passive baselines underperform: Passive VLM with GPT-4o reaches 67.7% R@3 and 72.9% R@10 on SceneFun3D objects; Passive VLM with GPT-5.2 reaches 78.9% and 85.6%.
• Robustness to pose noise: Under injected Gaussian noise (σ_trans = 0.02 m, σ_rot = 1°), Ours-GPT's SceneFun3D Object R@3 drops only 0.9% (83.0% to 82.1%), and Ours-LLaVA degrades by less than ~3.2% on FunGraph3D.
• Human evaluation on Replica: Average node precision of 0.82 for the method versus 0.68 for ConceptGraphs and 0.58 for ConceptGraphs-Detector; average edge precision of 0.91 versus 0.82 and 0.85. The method also reduces duplicate object predictions across the 8 scenes.
• Inter-annotator agreement: Fleiss' Kappa of 0.76 ("substantial agreement") over 100 randomly sampled evaluation units.
• Ablation impact: Removing re-shot guided uncertainty estimation causes the largest drop, with mF1 falling from 30.78 to 14.66 and f-mIoU from 54.26 to 35.56. Removing RAG, using random retrieval, or removing the concatenated re-shot image prompt all reduce performance.
• Runtime: Dynamic downsampling reduces per-iteration processing time from 6.65 s to 2.49 s on Replica compared with ConceptGraphs, roughly two-thirds. The GPT-4o variant costs approximately $0.50 per scene on FunGraph3D, with VLM reasoning at ~0.78 s/iteration and active verification rendering at ~0.26 s/object.
• Real-world deployment: On an Agilex SCOUT MINI platform with a RealSense D435, CH110 IMU, dual Livox MID 360 LiDARs, and an NVIDIA Jetson AGX Xavier, the system degrades only marginally under drift-prone pure IMU localization, and produces high-fidelity scene graphs when paired with robust state estimation such as Fast-LIO2.

Methodology in Plain English

The pipeline works in three stages. First, each RGB-D frame is segmented with SAM, embedded with CLIP, and projected into 3D; these local object lists are fused into a global list using a composite semantic-plus-spatial similarity score (θ = θ_sem + θ_spa) with a threshold δ_sim = 0.45. A scale-adaptive voxel size (δ_sample = 0.01 m scaled by the square root of each object's 3D bounding box diagonal) keeps small objects dense while making large ones cheap.

Second, for each global object, the top-k highest-confidence crop captions are gathered as a "passive baseline." The system then samples 64 virtual cameras on a hemisphere around the object and scores each candidate with a weighted sum of three terms—point visibility (α_vis = 0.6), a penalty for top-down/bottom-up angles (α_horiz = 0.2), and alignment with the average original viewing direction (α_prior = 0.2). Only the single best view is rendered and captioned, keeping overhead low. Cosine similarities between the re-shot caption embedding and each passive caption embedding are clustered with KMeans (K = 3); the cluster with the highest mean similarity forms the consensus caption, and its average similarity becomes the object's reliability score.

Third, objects are ranked by uncertainty (1 − ŝ). The top 50% lowest-uncertainty objects populate the retrieval document. For the rest, a 3D position-based retriever fetches the nearest document object's caption as environmental context, and a composite image stitching the re-shot point-cloud rendering with the best-matching RGB crop is fed to a VLM with the prompt "The picture is stitched from the point cloud image and the RGB image of the same indoor object. There is a [c_env] near the object. Briefly describe the object in the picture." Finally, an LLM infers edges using a fixed taxonomy of eight relation categories (on, in, part of, near, none and their bi-directional pairs) with few-shot in-context examples and a dynamic nearest-neighbor ratio for connectivity.

Why This Matters

• Research impact: The paper reframes 3DSG construction from passive multi-view aggregation to active verification, and shows that noisy object nodes—not weak relation models—are the main bottleneck for edge accuracy in open-vocabulary scene graphs.
• Real-world applications:
  • Service and assistive robots that must not confuse a medication bottle with a beverage container, a safety-critical example given in the paper.
  • Mobile robot navigation and manipulation that depend on compact, structured semantic scene representations.
  • Household or office robotics deployed in cluttered indoor facilities where occlusions and viewpoint constraints are unavoidable.
  • Offline semantic mapping backends paired with decoupled architectures such as DAAAM, where a low-frequency semantic layer guides high-frequency real-time exploration.
• Industry relevance: The LLaVA variant runs on a single local consumer GPU, and the GPT-4o variant costs roughly $0.50 per scene on FunGraph3D, making the approach viable for cost-sensitive deployment. Its resilience to tracking drift supports integration with commodity sensor suites and imperfect localization.

Future Directions

• Online, real-time operation: The system is currently offline; integrating it as a low-frequency semantic backend for high-frequency exploration, as the authors suggest with the DAAAM paradigm, remains open.
• Extending the relation taxonomy: The edge taxonomy is deliberately constrained to eight categories for stability; whether open-vocabulary relation spaces can be made equally reliable is untested.
• Broader benchmark coverage: Evaluation used SceneFun3D (20 scenes), FunGraph3D (24 scenes), and a human study on 8 Replica scenes; scaling to larger and more diverse environments is an open question.
• Deeper integration of state estimation: Since localization quality directly affects point-cloud ghosting and re-shot rendering, joint design of the reconstruction and verification stages could be explored.

Target Audience

Robotics and computer vision researchers working on 3D scene understanding, scene graph generation, and embodied AI; practitioners building semantic mapping stacks for mobile manipulators or service robots; and engineers interested in applying retrieval-augmented generation and vision-language models to spatial perception under real-world noise and imperfect localization.

Authors’ abstract

Open-vocabulary 3D Scene Graph (3DSG) can enhance various downstream tasks in robotics by leveraging structured semantic representations, yet current 3DSG construction methods suffer from semantic inconsistencies caused by noisy cross-image aggregation under occlusions and constrained viewpoints. To mitigate the impact of such inconsistency, we propose RAG-3DSG, which introduces re-shot guided uncertainty estimation. By measuring the semantic consistency between original limited viewpoints and re-shot optimal viewpoints, this method quantifies the underlying semantic ambiguity of each graph object. Based on this quantification, we devise an Object-level Retrieval-Augmented Generation (RAG) that leverages low-uncertainty objects as semantic anchors to retrieve more reliable contextual knowledge, enabling a Vision-Language Model to rectify the predictions of uncertain objects and optimize the final 3DSG. Extensive evaluations across three challenging benchmarks and real-world robot trials demonstrate that RAG-3DSG achieves superior recall and precision, effectively mitigating semantic noise to provide highly reliable scene representations for robotics tasks.