REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories

AI summary

Overview

• Research area: Multimodal large language model (MLLM) evaluation, embodied AI, and spatial cognition benchmarks.
• Technical level: Intermediate. The paper's concepts (object permanence, viewpoint change, counting) are intuitive, but interpreting the scaling analysis and model comparisons assumes familiarity with MLLM evaluation practice.
• Scope: The paper introduces REM, a synthetic 3D benchmark that measures whether multimodal models can maintain stable spatial representations of objects across egocentric navigation trajectories, and reports that current state-of-the-art models degrade sharply under even moderate scene complexity.

What This Paper Is About

Humans navigate environments by building persistent, viewpoint-independent "cognitive maps" that let them track where objects are, how many exist, and how they relate spatially even after those objects leave their field of view. The authors argue that multimodal LLMs, despite extensive video training, lack this capability, and they build REM to measure exactly where and how models fail. The goal is to move beyond overall accuracy scores and pinpoint which specific spatial competences break down, at what complexity levels, and why.

Key Contributions

1. A controlled benchmark for embodied spatial reasoning. REM uses Blender-generated synthetic environments with deliberately simple, easily distinguishable objects (cuboids, spheres, cones in distinct colors) so that failures reflect spatial reasoning and memory rather than object detection or visual discrimination.

2. Three complementary datasets isolating different failure modes. Baseline (3,119 trajectories, 47,019 QA pairs) tests general capability across varied complexity; Single Frame (350 trajectories, 1,289 QA pairs) isolates pure single-image counting as a control; Full Rotation (100 trajectories, 2,424 QA pairs) probes object permanence and distinction under a 360-degree rotation with deliberately mirrored scenes.

3. Explicit action conditioning. Unlike video benchmarks that only supply frames, REM includes the discrete camera action between consecutive frames (e.g., "move forward 1m," "rotate right 15 degrees"), simulating an agent that knows its own movement. This lets the benchmark test whether models actually integrate motion cues with visual change.

4. Systematic diagnostic analysis. The paper varies trajectory length (2 to 64 frames), scene congestion (8 to 48 objects), and object duplication (0 to 46 duplicates), then reports how each factor independently affects counting, comparison, left/right positioning, and temporal ordering — plus a human comparison on a Mini-Baseline subset (18 trajectories, 154 QA pairs).

Main Findings

• Best models are promising but unreliable. OpenAI o3 reaches 80.0% overall on the full Baseline, far ahead of GPT-4o (61.7%) and the Gemini-1.5 family (~58-60%), while Nova-Lite-v1 and Llama-3.2-11B sit only slightly above chance (37-39%). Humans average 97.8% on the Mini-Baseline, and even the best model (o3, 80.1%) substantially trails this.

• Counting is the weakest capability. o3 scores only 60.9% overall on counting, dropping to 35.7% when the ground-truth target count is 2 or more. Models systematically undercount as the true count grows and essentially never overcount.

• Object permanence collapses under full rotation. In the Full Rotation dataset, when two target objects are present, o3 fails 86% (single duplication) to 90% (double duplication) of the time. Models conflate visually similar scenes at 0 and 180 degrees, treating distinct object sets as the same set — a direct failure to use motion cues and visual context to individuate objects.

• Undercounting is partly a single-frame problem. In the Single Frame control, o3's average prediction-to-actual ratio is 0.65, versus 0.73 in multi-frame trajectories. The roughly 13% higher multi-frame predictions suggest some limited aggregation across frames, but the core undercounting bias persists regardless of trajectory length.

• Comparison degrades when quantities are close. Accuracy on numerical comparison falls as the target object counts become more similar, hitting 66% when the difference is zero — near the 33% random baseline. The authors interpret this as a "fuzzy counting" heuristic shared across models.

• Temporal ordering behaves counterintuitively. Performance improves as duplicate counts rise but degrades with overall scene congestion. The authors attribute this to order-preserving questions more often targeting duplicates, which have less attention competition in the scene.

• Left/right positioning is comparatively robust. o3 scores 92.3% on relational questions in the full Baseline but degrades with both scene congestion (occlusion, attention competition, intervening non-target objects) and trajectory length.

• Complexity compounds failure sharply. Increasing scene congestion to 36 viewed objects and restricting to counting questions with ground truth greater than 1 drops SOTA performance from 80% to under 60%.

Methodology in Plain English

The researchers built simple 3D rooms in Blender and placed colored shapes in them. A virtual camera then "walks" through each room — moving forward and rotating — producing a sequence of egocentric images. Alongside every image sequence, models receive the discrete action taken between frames.

They generated thousands of these trajectories with systematically varied length, object density, and duplication rate, using Blender's ground-truth annotations to record exactly which objects appear in which frames and where they sit left-to-right. Questions were then created from templates covering four types: how many objects of a given type exist, whether there are more of one type than another, whether one object is left or right of another, and whether one object appeared before or after another.

Model answers were graded with keyword and numeric matching — extracting the first number for counting, a single direction word for positioning, and sequence markers for ordering. For the human comparison, participants received an interactive interface where they could freely scroll through frames and use a scratchpad, with unlimited time.

Why This Matters

• Impact on research. Most video benchmarks report aggregate scores on realistic but uncontrolled footage. REM's controllable environment lets researchers attribute failures to specific factors — trajectory length, clutter, duplication — rather than guessing. The paper reframes the problem as one of world modeling and object individuation rather than perception.

• Real-world applications.

  • Home and service robots that must remember where items were placed after looking away or moving through rooms.
  • Warehouse and logistics systems where inventory counts must stay consistent across a moving camera's viewpoints.
  • Assistive navigation tools that describe spatial layouts to users and must track objects outside the current view.
  • AR and VR systems that overlay persistent spatial information as a user physically moves and turns.

• Industry relevance. The paper targets a specific bottleneck for deploying multimodal models in embodied settings: without object permanence and integration of motion with perception, systems that work well on static VQA cannot be trusted for navigation, manipulation, or long-horizon tasks. The benchmark's focus on controlled diagnostics gives teams a way to measure targeted progress rather than chasing headline accuracy on saturated benchmarks.

Future Directions

• Architectures that explicitly maintain world state. The authors call for designs and training paradigms that foster robust object permanence and integrate spatiotemporal context, rather than treating frames as independent inputs.
• Understanding and fixing the undercounting bias. Why models systematically undercount and never overcount remains unresolved, and it appears to originate in single-frame perception rather than accumulating across frames.
• Better use of given motion cues. Models receive explicit actions yet fail to leverage them to disambiguate mirrored scenes; closing this gap is a concrete research target.
• Extending beyond simple synthetic scenes. REM deliberately uses simple shapes and colors to isolate reasoning from perception; a natural follow-up is testing whether findings transfer to realistic objects and messy, real-world environments.

Target Audience

Researchers and engineers working on multimodal LLMs, embodied AI, robotics, and spatial reasoning evaluation. The paper is also useful for practitioners deciding whether a vision-language model is reliable enough for a navigation or manipulation task, and for benchmark designers interested in controlled, factor-by-factor diagnostics rather than aggregate scoring.

Authors’ abstract

Humans build viewpoint-independent cognitive maps through navigation, enabling intuitive reasoning about object permanence and spatial relations. We argue that multimodal large language models (MLLMs), despite extensive video training, lack this fundamental spatial reasoning capability, a critical limitation for embodied applications. To demonstrate these limitations and drive research, we introduce REM (Reasoning over Embodied Multi-Frame Trajectories), a benchmark using controllable 3D environments for long-horizon embodied spatial reasoning. REM systematically evaluates key aspects like object permanence/distinction, spatial relationships, and numerical tracking across dynamic embodied viewpoints. Our evaluation shows that the best-performing current models exhibit promising overall performance, but become increasingly unreliable at even moderate complexity levels easily handled by humans. These findings highlight challenges MLLMs face in developing robust spatial representations from sequential visual input. Consequently, REM provides targeted metrics and diagnostics to foster improved spatial understanding in future models.