Vision-Language Memory for Spatial Reasoning

AI summary

Overview

Research area: Computer vision and embodied AI — specifically video-based 3D spatial reasoning with vision-language models (VLMs).

Technical level: Advanced. The paper assumes familiarity with transformer attention, cross-attention fusion, 3D geometry foundation models, and parameter-efficient fine-tuning (LoRA).

Scope: The paper introduces VLM², a video-language model that builds view-consistent 3D-aware representations from monocular 2D video and augments them with a bounded dual-memory system for temporally coherent spatial reasoning.

What This Paper Is About

Vision-language models can describe what they see, but they struggle badly at spatial questions like "How many chairs are in this room?" — even when the answer is plainly visible across a walkthrough video. The core problem is twofold: models fail to keep semantic and geometric information aligned across camera viewpoints, and they have no memory, so information from earlier frames is overwritten and lost by the time it becomes relevant. The authors set out to build a model that maintains a consistent 3D understanding of a scene purely from ordinary 2D video, and that can recall observations from earlier viewpoints without requiring depth sensors or point-cloud inputs.

Key Contributions

1. A view-consistent 3D-aware representation learned from 2D video alone. The model fuses semantic features from a 2D vision encoder with geometric priors from a 3D foundation model (π³), using three components: adaptive 3D position injection, viewpoint-aware geometry alignment, and cross-attention-based semantic-geometric fusion. No ground-truth depth or point clouds are required at inference.

2. A dual-memory module with bounded cost. A sliding-window working memory captures immediate context, while a fixed-capacity episodic memory stores salient observations from across the whole video. A learned gate fuses the two, and a similarity-based update rule keeps the episodic bank diverse and non-redundant. This keeps computation and storage bounded regardless of video length.

3. State-of-the-art results among video-based models on four benchmarks. VLM² leads on VSI-Bench, VSTI-Bench, ScanQA, and SQA3D, including beating several models that use explicit 3D or 2.5D inputs.

4. Demonstrated zero-shot generalization and robustness to video length. Without additional fine-tuning, the model tops average accuracy on CV-Bench, SPAR-Bench, and BLINK, and its advantage over the prior best model grows on longer videos.

Main Findings

• VSI-Bench: 68.8 average accuracy, +7.9 points over the previous best spatial-enhanced model (VLM-3R-7B at 60.9). Gains are largest on Relative Direction (+7.3), Route Plan (+7.2), and Appearance Order (+28.2) — tasks that specifically require remembering objects across changing viewpoints.

• VSTI-Bench: 65.3 average, +6.5 over VLM-3R (58.8), an 11.1% relative improvement. The biggest jump is on Object-Object Relative Position (+16.2), a task demanding consistent spatial layouts across frames.

• ScanQA: CIDEr 105.5 versus 91.8 for the best prior video-based model, and higher than 3D-input models like Video-3D LLM (102.1) and 3DRS (104.8). SQA3D exact-match reaches 60.4, competitive with point-cloud-based systems.

• Ablation confirms both ideas are needed. Adding the 3D-aware representation alone lifts the fine-tuned LLaVA-NeXT-Video baseline by 8.6 points; working memory adds 2.1 and episodic memory 2.3; combining all three yields +12.6 overall. Route Plan alone improves by 12.4 points with the full stack.

• Naïve fusion fails. Concatenating visual and geometry tokens through an MLP actually hurts, scoring 40.4–43.3 versus the 55.2 baseline. Cross-attention fusion alone reaches 61.0; adding the paper's viewpoint-aware alignment and adaptive injection pushes it to 63.8.

• Adaptive gating matters. Removing the learnable gate from 3D position injection drops performance from 61.6 to 58.9, showing that indiscriminately injecting predicted 3D coordinates into every patch adds noise.

• Memory length has an optimum. The best configuration is a working window of 8 frames and an episodic capacity of 32 entries (68.8 average). Enlarging the working window to 16 slightly hurts, because older recent frames dilute attention.

• Gains grow with video length. On both VSI-Bench and VSTI-Bench, VLM²'s advantage over VLM-3R is largest on videos longer than two minutes, while the baseline degrades.

Methodology in Plain English

The team started from an existing video-language model (LLaVA-Video-7B) and added two things that existing models lack.

First, a better way to combine "what" and "where." For each frame, they extract semantic features from a 2D vision encoder and geometric features plus camera-viewpoint information from a 3D foundation model called π³. Simply stapling these together doesn't work, because the geometric features don't know which direction the camera is facing — a chair's front legs and back legs can look geometrically similar. So they tag each geometric feature with information about the camera's viewpoint, both locally (per patch) and globally (per frame). Separately, they take the 3D coordinates predicted by the foundation model and add them to the visual features — but only through a learned gate, so the model can ignore unreliable or irrelevant coordinates rather than being forced to trust noisy depth estimates. Finally, the visual features query the geometry features through cross-attention, producing a per-frame representation that is both semantically meaningful and spatially grounded.

Second, memory. Because a video is a stream, the model needs a way to remember. Working memory is a short sliding window (the last 8 frames) that the current frame attends over — this handles immediate context. Episodic memory is a fixed-size bank of 32 slots storing older, more salient observations, also accessed by attention. A learned gate mixes the two streams. To keep the bank from filling with near-duplicates, each new representation replaces whichever stored entry it is most similar to, based on cosine similarity. The result is that memory cost stays constant no matter how long the video runs.

The whole system — the fusion modules and the memory — is trained end-to-end on question-answering tasks. Only lightweight LoRA adapters and the new modules are trained; the vision encoder and 3D foundation model stay frozen.

Why This Matters

Spatial reasoning is the gap between a model that can talk about a scene and one that can act in it. This work shows that a persistent, viewpoint-consistent 3D understanding can be built from cheap monocular video — no LiDAR, no depth camera, no ground-truth point clouds — which is exactly the sensor setup available on consumer robots and phones.

Real-world applications:

• Home and service robots that need to answer "how many chairs are at the table" or "where did I leave the keys" while moving through a house, where objects pass in and out of view.
• Warehouse and logistics automation, where a robot must maintain an accurate inventory of items across a long traversal with occlusions.
• Assistive technology for visually impaired users, providing narrated spatial descriptions of changing environments from a phone camera.
• AR/VR and mobile mapping, where a device must remember the layout of a room the user walked through minutes ago and reconcile it with what is currently visible.

Industry relevance: The approach is deliberately built on top of open-weight models and frozen pretrained components, keeping training costs modest (one epoch, LoRA, 8 GPUs) and deployment requirements low (a single RGB stream). The bounded-memory design is particularly relevant for embedded robotics, where compute and RAM are fixed regardless of how long a task runs. It also directly challenges the assumption in prior work that accurate depth input is needed for strong 3D QA.

Future Directions

• Beyond answering questions: acting on memory. The episodic bank currently supports QA. Whether the same representations can drive navigation, manipulation, or active exploration policies is untested.

• Smarter memory eviction. The similarity-based update rule is heuristic. Learned policies for what to store, merge, or discard — and how to handle objects that genuinely change state rather than merely reappearing — remain open.

• Scaling to long-horizon and multi-room settings. Gains grow with video length, but the fixed 32-slot episodic capacity is a hard ceiling. Hierarchical or spatially-indexed memory could extend this to hour-long or building-scale sequences.

• Reducing reliance on the 3D foundation model's predictions. The adaptive gate mitigates noisy predicted coordinates, but a systematic study of how much of the geometry signal is actually trustworthy — and whether it can be distilled or self-supervised — would clarify the method's true limits.

• Multi-agent and dynamic scenes. All benchmarks feature mostly static scenes observed by a moving camera. Reasoning about other moving agents, and about memory shared across multiple robots, is unaddressed.

Target Audience

Researchers and graduate students working on embodied AI, video-language models, 3D scene understanding, and robot perception, particularly those interested in spatial reasoning benchmarks like VSI-Bench and SQA3D. It is also relevant to robotics engineers evaluating whether monocular video plus memory can replace depth-sensor pipelines, and to practitioners building memory-augmented or long-context multimodal architectures who want a concrete, bounded-cost design to borrow from.

Authors’ abstract

Spatial reasoning is a critical capability for intelligent robots, yet current vision-language models (VLMs) still fall short of human-level performance in video-based spatial reasoning. This gap mainly stems from two challenges: a semantic-geometric misalignment that prevents consistent 3D understanding, and the absence of persistent memory to retain 3D representation and understanding across frames. To address these limitations, we present VLM$^2$, a Vision-Language Model with persistent Memory for spatial reasoning with a view-consistent, 3D-aware representation purely from 2D videos. Specifically, we incorporate a dual-memory module consisting of a working memory that operates as a sliding window to focus on immediate context, and an episodic memory that consolidates and stores critical information across frames. This design enables bounded and efficient spatial reasoning under a fixed computational cost. Extensive experiments on multiple benchmarks show that VLM$^2$ achieves state-of-the-art performance among video-based models, significantly advancing the frontier of visual-spatial intelligence.