Memory Tree Guided Key Frame Querying for Efficient 3D Question Answering

AI summary

Overview

• Research area: 3D / embodied question answering, vision-language models, efficient key-frame retrieval for long video.
• Technical level: Intermediate (assumes familiarity with VLMs, object detection/tracking, and 3D scene representations, but the core idea is explained conceptually).
• Scope: The paper proposes MemTree3D, a lightweight tree-structured 3D scene representation built online from 6-DoF camera poses, which lets a Large Language Model select question-relevant key frames for a Vision Language Model instead of running an exhaustive visual search over every video frame.

What This Paper Is About

Answering questions about a 3D scene from a scan video is expensive because high-FPS scans contain thousands of frames, and transformer-based VLMs scale their computation and memory quadratically with input length. Existing visual-search methods pick key frames by running detectors or vision models over the whole video for each query, which is slow, cannot handle multi-round queries efficiently, and breaks when the vision model misses the relevant object. This paper builds a compact, reusable 3D scene representation once, then has an LLM query it to generate spatial and temporal cues for retrieving a few informative frames.

Key Contributions

1. MemTree3D: a compact, reusable 3D scene representation supporting online, real-time construction (25+ FPS on a single GPU) that enables LLM-driven key-frame selection.
2. A MemTree3D-guided key-frame selection paradigm for 3D question answering that is significantly more efficient than existing visual search key-frame selection approaches.
3. Extensive experiments showing a 17.4% LLM-Match improvement for GPT-4o and a 69.2% key-frame retrieval speedup over prior visual search methods, plus robustness to perception failures through LLM reasoning.

Main Findings

• OpenEQA accuracy gains: MemTree3D improves GPT-4o by 17.4% (from 49.4 to 66.8 LLM-Match with 3 input frames) and LLaVA-OneVision-7B by 5.8% (from 49.2 to 55.0), using the same number of input frames.
• Outperforms visual search methods: With GPT-4o at 3 frames, MemTree3D reaches 66.8 average LLM-Match versus 63.0 for VLM-Grounder (6 frames), 57.2 for 3D-Mem (3.1 frames), and 50.6 for CLIP Retrieval (3 frames).
• Efficiency: Runtime of Detector-based Frame Selection increases with the number of video frames while MemTree3D Frame Selection stays relatively stable, with at least 69.2% runtime speedup, measured on OpenEQA starting from when the user query is received.
• Category-level gains on OpenEQA: MemTree3D improves uniform-sampling GPT-4o by more than 10% in every category: object recognition (+10.6), object localization (+21.2), attribute recognition (+10.6), spatial understanding (+15.7), object state recognition (+20.8), functional reasoning (+20.2), and world knowledge (+22.1).
• Larger scenes benefit more: On the ScanNet subset (avg. scene size 82.6 m³, avg. 5.9 LocNode) the gain is +13.6 LLM-Match; on the HM3D subset (avg. 556.0 m³, avg. 20.3 LocNode) the gain is +24.6.
• ScanQA and SQA3D: With LLaVA-OneVision-7B at 3 frames, MemTree3D reaches 28.0 EM@1 on ScanQA (+2.9) and 49.6 on SQA3D (+3.4). The paper attributes the more moderate gains here to the smaller scene size in ScanNet, where uniform sampling already gives decent coverage.
• Open-source models work well: Replacing GPT-4o as the LLM with Qwen3-4B or Qwen3-8B (paired with LLaVA-OneVision-7B) does not notably degrade performance; Qwen3-4B reaches 51.1 and Qwen3-8B 53.1 average LLM-Match, both exceeding uniform-sampling GPT-4o at 49.4.
• 6-DoF segmentation beats uniform segmentation: LocNode construction using 6-DoF pose achieves 49.1 LLM-Match versus 40.4 for uniform construction with 30 frames per node and 33.0 with 90 frames per node — an 8.7% improvement over uniform sampling.
• Robustness to perception failure: When query objects are absent from the constructed representation, LLM-Match is 65.9 versus 69.0 when they are present, only a 3.1-point drop.
• More frames help: Performance with MemTree3D-selected frames consistently improves as more frames are included, and MemTree3D outperforms uniform sampling at different frame counts.

Methodology in Plain English

The pipeline has three stages. First, as the agent moves through the scene, MemTree3D is built online and in real time using the camera's 6-DoF poses plus an object detector (YOLO-World) and multi-object tracker (BoT-SORT). The structure has three levels: LocNode (a segment of the scan, created whenever translation or rotation since the previous pose exceeds a threshold), ObjNode (an object tracklet with a compact trajectory, derived from detections and tracking), and DetNode (a leaf holding a single frame's detection: timestamp, bounding box, and confidence). Because camera motion defines the segments rather than semantic clustering, construction is cheap and each segment corresponds to a distinct physical location.

Second, the first two tree levels are serialized into a textual JSON format and given to an LLM along with the user question. The LLM reasons over what objects appear in which locations and produces two kinds of cues: temporal cues (which candidate location segments likely contain the answer) and spatial cues (a set of key objects directly relevant to the question, and a set of cue objects that tend to co-occur nearby). Notably, because the LLM reasons over symbolic structure rather than raw detections, it can still propose plausible locations when the relevant object was never detected.

Third, for each of the top-k selected locations, one representative frame is chosen using a scoring function that aggregates detection confidence scores of key and cue objects with different weights (a fixed 10:1 key-to-cue ratio). These k frames, drawn from different locations and viewpoints, are passed with the question to a VLM for the final answer. Default settings are k = 3, translation threshold 1.5 m, rotation threshold 45°, with all parameters held constant across scenes. ScanNet-200 common indoor object categories are used, and inference temperature is 0.0 for GPT-4o and LLaVA-OneVision-7B and 0.6 for Qwen3.

Why This Matters

• Research impact: It reframes key-frame selection from an expensive per-query visual search problem into a one-time scene-representation construction plus cheap LLM querying, and it introduces a representation that is lightweight enough for real-time use while still supporting high-level reasoning.
• Real-world applications:
  • Household robots that need to answer spoken questions about a home they have already walked through.
  • Embodied navigation agents that must locate target objects or regions on demand.
  • Augmented and virtual reality systems that need spatial reasoning about a scanned environment.
  • Assistive agents for users who need grounded descriptions of a physical space.
• Industry relevance: The method is designed for resource-constrained embodied hardware, works with open-source models (Qwen3-4B, Qwen3-8B, LLaVA-OneVision-7B) as well as proprietary ones, and produces consistent runtime regardless of video length — all of which matter for deployment where latency after a user question is the key metric.

Future Directions

• Novel-object localization: The paper states that the method can fail when a question asks about an object entirely absent from MemTree3D (such as a cardboard cat scratcher in its failure example); the LLM makes its best guess but the selected frames may lack a clear view, so improving recovery for totally unseen objects is an open problem.
• Adaptive parameter selection: The authors note that per-scene tuning of the segmentation and retrieval parameters could further improve temporal segmentation and spatial retrieval, but they deliberately used a unified setting to reflect real deployment.
• Tighter integration with reasoning quality: Whether stronger or differently prompted LLMs materially change location-selection accuracy is not fully explored beyond the model-size ablation reported.
• Extending beyond detection-based representations: The paper does not report experiments on modalities or representations that could complement the object-centric tree, leaving open how to capture scene attributes that never appear as detected objects.

Target Audience

Researchers and engineers working on embodied AI, 3D question answering, and long-video vision-language systems; practitioners building resource-constrained agents that need low-latency question answering over scanned environments; and readers interested in compact intermediate scene representations that bridge perception output and LLM reasoning.

Authors’ abstract

Answering questions accurately and efficiently in embodied scenarios presents significant challenges due to limited computational and memory resources for Vision Language Model (VLM) inference. Existing methods adopt visual search key frame retrieval method to select critical question-related key frames for VLM input. However, visual search methods are inefficient because they require visual search among thousands of video frames for each individual user query. In this work, we propose a memory tree guided key frame selection paradigm for efficient 3D question answering in embodied scenarios. Our method leverages a compact and reusable 3D scene representation, termed MemTree3D, which supports real-time online construction leveraging camera 6-DoF poses. MemTree3D captures multi-level 3D scene information, enabling a Large Language Model to efficiently query and retrieve question-relevant key frames through our scoring-based frame selection without reprocessing the entire video stream. On OpenEQA, our method improves the LLM-Match of GPT-4o by 17.4%, LLaVA-OneVision-7B by 5.8%, outperforms existing visual search methods. Our code is available at https://github.com/hsiangwei0903/MemTree3D