CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically spatial reasoning benchmarks and world models.

Technical level: Intermediate. The paper is readable without prior benchmark-construction experience, but assumes familiarity with MLLMs, chain-of-thought prompting, and video generation models.

Scope: The paper introduces CausalSpatial, a synthetic diagnostic benchmark for "causal spatial reasoning" (predicting the physical consequences of hypothetical object motion), and COW, an object-centric world model that generates simulated video to ground MLLM reasoning.

What This Paper Is About

Current multimodal models can describe what is in a static image and roughly where objects sit, but they cannot answer "what-if" questions such as whether a car will hit a vase if it moves forward. The authors argue that this ability — grounding causal prediction in 3D spatial structure — is unmeasured by existing benchmarks, which focus on 2D perception, static 3D relations, or passive motion tracking. The paper builds a benchmark to measure that gap and proposes a video-generation framework that supplies models with explicit visual simulations of the predicted dynamics.

Key Contributions

1. CausalSpatial, an object-centric causal spatial reasoning benchmark. It contains four task types — Collision, Compatibility, Occlusion, and Trajectory — each with two difficulty levels (L1 one-to-one reasoning, L2 one-to-multiple reasoning). It is described as the first open-source diagnostic benchmark of this kind, with all scenes rendered in Blender using its physics engine.

2. An extensive MLLM evaluation exposing a machine–human gap. The authors test spatial-reasoning MLLMs (Spatial-VLM, Spatial-MLLM), open-source MLLMs (LLaVA-OneVision-1.5-8B, the Qwen2.5-VL and Qwen3-VL families), and closed-source MLLMs (GPT-5 series, Gemini 2.5 series, Claude Sonnet 3.7), alongside three human evaluators and a random baseline.

3. A failure diagnosis. The analysis attributes model failures to over-reliance on textual chain-of-thought that drifts from the visual evidence, producing fluent but spatially ungrounded hallucinations and losing the geometric constraints present in the image.

4. Causal Object World Model (COW). COW is an object-centric, 4D-trajectory-controlled video generation pipeline. It simulates object trajectories, renders them into video via the ATI world model, and passes the generated frames to an MLLM as additional visual evidence.

Main Findings

• Large human–model gap. Humans score 84.49% on average (85.44% at Level 1, 83.54% at Level 2). The best model, GPT-5, reaches 54.17% (54.00% Level 1, 54.52% Level 2). The abstract states this as humans 84% versus GPT-5 at 54%.

• Random guessing is far below models. The random baseline averages 28.6% (30.34% Level 1, 24.50% Level 2), well under the strongest MLLMs.

• Closed-source models lead, but not decisively. GPT-5 ranks first (54.17%), Gemini 2.5 Pro second (50.55%), Qwen3-VL-30B-A3B-T third (50.50%), Gemini 2.5 Flash fourth (48.82%), Sonnet 3.7 fifth (48.07%), and GPT-5-mini seventh (45.44%).

• Specialized spatial MLLMs underperform general-purpose ones. Spatial-VLM averages 33.50% and Spatial-MLLM 40.00%, both below general-purpose models such as Qwen3-VL-4B-I (44.76%) and Qwen3-VL-30B-A3B-I (45.80%).

• A compact open-source model beats larger competitors from other families. Qwen3-VL-2B-I averages 40.60%, above Qwen2.5-VL-7B (39.61%) and LLaVA-OneVision-1.5-8B-I (34.62%).

• Collision is the hardest task for models. Humans retain 78.57% at Level 2, while LLaVA-OneVision-1.5-8B-I collapses to 15.38%, only modestly above random guessing.

• Occlusion separates frontier from open-source models. Open-source models hover around 20–30% at Level 2, while GPT-5 reaches 62.96%, which the authors read as a stronger internal representation of 3D structure.

• Trajectory is where the gap is narrowest. Humans score 62.50% at Level 2 and GPT-5 scores 64.96%, indicating models are relatively better at trajectory-path patterns than at interaction dynamics. The authors note human intuition for precise trajectory prediction is itself less reliable than human semantic understanding.

• Model scaling gives limited returns. Instruction-tuned Qwen3-VL variants plateau: 4B at 44.76%, 8B at 43.53%, 30B-A3B at 45.80%.

• Confidence decouples from accuracy. Not Sure Rate falls from 18.77% for Qwen3-4B-I to nearly zero (0.10%) for Qwen3-30B-A3B-T. Thinking mode lowers NSR further: on Qwen3-8B from 10.57% (I) to 2.37% (T), and on Qwen3-30B-A3B from 2.84% (I) to 0.10% (T). This increased decisiveness is not matched by accuracy gains.

• COW improves reasoning without retraining. Using only three initial frames from generated videos, accuracy rises 2.40% on Collision and 2.20% on Trajectory for Qwen3-VL-30B-A3B-Instruct. Occlusion shows a slight decrease, which the authors attribute to the task's continuous, long-horizon nature relative to the short futures the first frames depict.

• No dataset size figure is reported in the provided text for the total number of evaluation items; the paper references per-task counts in its Figure 2 but those numbers are not given in the content available here.

Methodology in Plain English

The authors built scenes in Blender from publicly available 3D assets covering more than 50 object categories, manually refining textures and standardizing scale, materials, and orientation. They sampled over 130 camera viewpoints across different elevations and azimuths, and annotated each scene with RGB images, depth maps, and oriented bounding boxes.

Each question gives a model one initial image plus a text describing a hypothetical motion of a specified object. The motion is physically simulated in Blender's Bullet Physics Engine, and ground-truth answers come from computing mesh intersections in the simulated result. The first frame of each rendered sequence becomes the question image, paired with its causal QA pair. All questions are multiple-choice with four candidates plus a "Not Sure" option, and answer order is randomized per sample. GPT-5 was used to draft object descriptions, which human annotators then reviewed; every evaluation instance was human-verified.

Evaluation uses two metrics: True Positive Rate (TPR, accuracy over all samples) and Not Sure Rate (NSR, the share of abstentions), to separate confident errors from acknowledged uncertainty. Images are resized to 1920×1080, models answer in a single turn with chain-of-thought prompting and a {"Reasoning":…, "Answer":…} JSON format, maximum output length is 8192 tokens, and each question is evaluated five times.

For COW, objects are represented as 6-dimensional vectors {x, y, z, vx, vy, vz}, with position and direction estimated by prompting an MLLM. Trajectories are built by iteratively updating positions under external physical factors such as gravity. These trajectories and motion descriptions condition a diffusion-based generator (ATI) to synthesize future frames, which the MLLM then interprets alongside the text query. COW evaluation used the three initial generated frames as extra cues, resized images to 512 pixels, tested Qwen3-VL-30B-A3B-Instruct, and repeated each experiment three times.

Why This Matters

Impact on research. The paper reframes spatial intelligence as a causal, 4D problem rather than a static perception problem, and provides a benchmark that separates confident hallucination from genuine uncertainty through the Not Sure Rate. Its diagnosis — that textual chain-of-thought drifts from visual evidence — is a concrete, testable claim about how current MLLMs fail, and the COW results suggest external visual simulation can substitute for internal physical modeling without retraining.

Real-world applications:

• Robotics, where a manipulator must predict whether moving an object will collide with or displace others.
• Autonomous driving, where a vehicle must anticipate interactions between moving objects rather than merely detect them.
• Warehouse and logistics planning, where feasibility questions such as whether an item fits in a container depend on geometry and containment constraints.
• Assistive and augmented-reality systems that must reason about occlusion and visibility changes as objects are repositioned.

Industry relevance. The benchmark offers a diagnostic target for teams building spatial-capable multimodal systems, and the finding that model scale does not translate into causal spatial gains challenges the assumption that larger models alone will close the gap. COW's use of an off-the-shelf world model plus MLLM prompting, with no extra training data, is a practical pattern for deployment.

Future Directions

• Longer-horizon simulation. The first three simulated frames help Collision and Trajectory but slightly hurt Occlusion, where the relevant events may unfold later. Extending simulation to longer futures is an open problem.
• Geometric consistency beyond position. The authors state that guaranteeing consistency in volume, shape deformation, and inter-object collision dynamics remains open.
• Rethinking chain-of-thought grounding. How to keep textual reasoning anchored to visual evidence, rather than drifting toward linguistic priors, is described as a central challenge without a settled solution.
• Whether world models can become the primary reasoning substrate. COW is evaluated as an assistive cue for one model; whether object-centric world models can be extended to broader causal spatial tasks is left for future work.

Target Audience

Researchers and engineers working on multimodal large language models, spatial and 3D reasoning, embodied AI, and world models; benchmark designers looking for evaluation methodology that uses abstention to separate uncertainty from hallucination; and practitioners in robotics, autonomous driving, or AR who need models to anticipate the physical consequences of object motion rather than just describe scenes.

Authors’ abstract

Humans can look at a static scene and instantly predict what happens next -- will moving this object cause a collision? We call this ability Causal Spatial Reasoning. However, current multimodal large language models (MLLMs) cannot do this, as they remain largely restricted to static spatial perception, struggling to answer "what-if" questions in a 3D scene. We introduce CausalSpatial, a diagnostic benchmark evaluating whether models can anticipate consequences of object motions across four tasks: Collision, Compatibility, Occlusion, and Trajectory. Results expose a severe gap: humans score 84% while GPT-5 achieves only 54%. Why do MLLMs fail? Our analysis uncovers a fundamental deficiency: models over-rely on textual chain-of-thought reasoning that drifts from visual evidence, producing fluent but spatially ungrounded hallucinations. To address this, we propose the Causal Object World model (COW), a framework that externalizes the simulation process by generating videos of hypothetical dynamics. With explicit visual cues of causality, COW enables models to ground their reasoning in physical reality rather than linguistic priors. We make the dataset and code publicly available here: https://github.com/CausalSpatial/CausalSpatial