Cube Bench: A Benchmark for Spatial Visual Reasoning in MLLMs

AI summary

Overview

Research area: Multimodal large language model (MLLM) evaluation, spatial reasoning, and sequential decision making.

Technical level: Intermediate. The paper assumes familiarity with benchmark design, accuracy/parser metrics, and standard MLLM terminology, but its core ideas are explained in plain terms.

Scope: The paper introduces Cube Bench, a simulator-based Rubik's Cube benchmark that decomposes MLLM spatial-sequential competence into seven tests and evaluates seven models under identical prompts, parsers, seed lists, and a single distance-to-solved metric.

What This Paper Is About

Current multimodal benchmarks mostly test static perception: recognizing objects, reading text in images, or answering a short question about a fixed scene. Those scores say little about whether a model can take an action, observe the consequence, and stay coherent over many steps. Cube Bench addresses this by using the Rubik's Cube as a fully observed, exactly solvable testbed where an oracle tells you the exact number of optimal moves remaining, letting the authors isolate which stage of the see-evaluate-act-reflect-recover loop breaks down.

Key Contributions

1. A generator-based benchmark, not a fixed dataset. Cube Bench is built on a VirtualCube simulator that deterministically emits, for a given target distance d and episode index i, a rendered cube-net image, an authoritative textual state, and four candidate actions (A–D). Per-depth seed lists are published so results are exactly reproducible across models, and episode i uses seed = i.

2. Seven tests covering the full decision loop. The abstract frames these as five skills, while the body defines seven tests: cube face reconstruction, cross-modal verification, optimal move prediction, reflection-guided re-answering, closed-loop step-by-step control, causal move-effect prediction, and learning-curve/recovery.

3. Strict fairness and scoring controls. A single move metric (HTM/FTM) is used for both scrambling and evaluation; oracle distance d(s) is computed once via IDA* with pattern databases; correct answers are kept approximately uniform over A–D by discarding duplicates and regenerating episodes; strict one-line output formats mean anything else counts as an error.

4. A comparative evaluation of seven MLLMs — Gemma 3-27B, GLM-4.5V, Llama 4-Scout-17B:16E, Qwen2.5-VL-7B, Qwen2.5-VL-32B, Qwen3-VL-30B-A3B (also written Qwen3-Thinking 30B), and Gemini 2.5 Pro — with a public code release at https://github.com/dana-23/cube-bench.

Main Findings

• Accuracy collapses with scramble depth. All models degrade as depth increases, and once a trajectory stalls or diverges, recovery is rare. The abstract states that high face-reconstruction accuracy does not guarantee competent action selection or multi-step execution.

• Face reconstruction splits recognition from spatial arrangement. At d=3, Gemini 2.5 Pro reaches 96.8% element-wise accuracy and 84.0% exact 3×3 matrix accuracy, while Qwen2.5-VL-32B drops to 71.3% and 19.0%. The gap between per-sticker and whole-matrix accuracy widens with depth across models.

• Near-perfect parsing does not mean good discrimination. On cross-modal Yes/No verification at fixed d=5, parse rates are roughly 100% for top models, yet balance (Bal = (TPR+TNR)/2) varies widely: GLM-4.5V 95.00%, Gemini 2.5 Pro 100.00%, Qwen3 84.00%, Qwen2.5-VL-32B 81.00%, Qwen2.5-VL-7B 71.30%, Gemma 3 58.00%, Llama 4-Scout-17B 47.00%. Residual Yes-bias persists (52–93%), with Llama 4-Scout-17B:16E at 93%.

• A large closed- vs open-source gap in move selection. On optimal move prediction (one move from solved), Gemini 2.5 Pro leads in every modality with perfect A–D parsing: 96.00 (Image+Text), 92.00 (Image), 80.00 (Text). Most open-weight models land at 20–35%. Qwen2.5-VL-7B is better in Text-only (26.00) than Image-only (18.00) or Image+Text (19.00); Qwen3-Thinking 30B shows weak format compliance (77% parsing in Text-only). GLM-4.5V drops from 31.00 (Image+Text) to 21.00 (Image-only).

• Reflection helps unevenly and can backfire. Under the label-safe "Guided (Redacted)" regime with Image+Text, Qwen2.5-VL-7B gains +14.00 (EFR 36.00%, OTR 65.00%), Qwen2.5-VL-32B +11.00 (EFR 34.71%, OTR 64.00%), Qwen3-Thinking 30B +11.00 (EFR 27.81%, OTR 50.00%), and Llama 4-Scout-17B:16E +6.00 (EFR 20.00%, OTR 40.00%). Gemma 3-27B shows 0.00 net change (EFR 22.00%, OTR 78.00%), and GLM-4.5v degrades by −15.00 because OTR 66.70% overwhelms EFR 6.20%.

• Causal move-effect prediction separates the frontier model from the rest. Only Gemini 2.5 Pro shows consistently positive Cohen's κ and the strongest Macro-F1 across depths (d=1: Acc 79.0, F1 0.78, κ 0.680; d=2: 65.0/0.632/0.475; d=3: 62.0/0.606/0.439). Open-weight models cluster near chance; Qwen2.5-VL-32B's κ is ≈−0.074 at d=1, ≈0.040 at d=2, and ≈0.086 at d=3, with Macro-F1 ≈0.25 at d=1 and ≈0.33 at d=3 because it mostly predicts INCREASE and rarely predicts DECREASE.

• Closed-loop control collapses with depth. Gemini 2.5 Pro falls from 90 to 10 TA% (Teacher-Adherence); open-weight models sit at 20–26% TA at d=1 and fall to 6–9% by d≥3: Qwen-2.5-VL-7B 26→6, Qwen-2.5-VL-32B 20→7.5, Qwen3-VL-30B 22→9, GLM-4.5V 20→9, Gemma-3-27B 20→7.

• Post-error recovery is weak. Starting at d=3 with six additional attempts allowed after the first deviation, Gemini 2.5 Pro leads at SR 40.00% (95% CI [11.76, 76.93]), P(≤3)=0.40, Med@Solved 3.00, Avg@All 4.8. Qwen2.5-VL-7B reaches SR 8.00%, Gemma 3-27B 4.00%, Llama 4-Scout-17B 6.00%, GLM-4.5v 2.00%, and Qwen2.5-VL-32B 0.00%. No system achieves immediate bounce-back: P(1)=0 for all.

• Calibrated abstention helps modestly without changing the depth pattern. At d=5 with an IDK option (teacher_on_abstain), Gemini 2.5 Pro has coverage 87.5%, selective accuracy 28.6%, IDK 12.5%, and APA 0.28 at λ=0.25; Qwen2.5-VL-32B has coverage 99.3%, selective accuracy 23.7%, IDK 0.0%, APA 0.24; Gemma 3-27B has coverage 100.0%, selective accuracy 17.3%, IDK 0.0%, APA 0.17.

• Leaky reflection inflates scores. In the "Guided (Unredacted)" ablation where the reflection prompt includes the correct option, GLM-4.5v and Llama 4-Scout-17B:16E rise from 31% and 18% to 100% (+69 and +82 points), Qwen models jump to 59–72% with gains of +42 to +50 points, and Gemma 3-27B rises from 22% to 81%.

• Pre-action causal evaluation tracks control. At d=1, Move-Effect κ correlates with Closed-Loop TA% at Pearson r=0.997 (95% CI [0.972, 1.000]); the relation weakens among models with κ≈0.

Methodology in Plain English

The authors build a small simulator of a Rubik's Cube instead of shipping a fixed image dataset. Given a target difficulty (the number of optimal moves from solved) and an episode index, the simulator always generates the same three things: a picture of the cube drawn as a flat, unfolded net with all six faces labeled and a neutral gray background, a text description of the cube state, and a menu of four candidate moves labeled A through D.

Because the cube has known optimal solvers, the authors can compute the exact minimum number of moves left to solve any state. This "oracle distance" is computed once using IDA* with pruning tables via the RubikOptimal library, in the face-turn metric (FTM). That single number serves as the yardstick for progress across every test, and the same metric is used for scrambling and scoring.

The tests walk through the stages of a decision loop. In face reconstruction, the model sees only an image and must output the front face as a 3×3 grid of color initials from {W, Y, R, O, G, B}. In cross-modal verification, the model gets an image plus a text grid and answers Yes or No for consistency, with items drawn from states at distance d=5 and balanced positives and negatives. In optimal move prediction, the model picks the best of four moves, shown in three prompt modalities. In reflection-guided re-answering, the model's first answer is passed through one reflection pass in either a redacted (no label) or unredacted (label revealed) regime. In closed-loop control, the model picks a move, the simulator executes it, the distance is recomputed, and the episode ends at the first non-optimal step or parse error. In causal move-effect, the model predicts whether a given move will DECREASE, leave NO_CHANGE, or INCREASE the distance, before the move is executed. In recovery, the episode restarts from the first failure point with a budget of additional attempts.

Fairness controls include strict one-line output formats (A–D, Yes|No, or the three causal labels), deterministic decoding at temperature 0, deterministic option shuffling, and regeneration of episodes until the correct option is approximately uniform across A–D. Text is treated as the source of truth; images never override the textual state. Experiments ran on NVIDIA A100-80GB GPUs using vLLM with bf16.

Why This Matters

Impact on research. Cube Bench offers what the authors describe as a compact, reproducible probe of sequential spatial reasoning. It supplies deterministic transitions, a small discrete action set, paired image/text state, and exact oracle progress signals — properties that let researchers attribute failures to perception, evaluation, selection, or reflection rather than to web-induced variance. The finding that chance-corrected κ tracks closed-loop adherence and recovery argues for decision-aware metrics over raw accuracy.

Real-world applications:

• Robotic manipulation and embodied control, where a policy must act, observe consequences, and stay coherent over long horizons.
• Web and computer-use agents, where the paper notes strong snapshot performance does not directly translate to reliable multi-step behaviour.
• Instruction-following systems with strict output contracts, since the benchmark shows non-conforming outputs directly convert into errors and can mask underlying competence.
• Self-critique and reflection pipelines, where the EFR/OTR trade-off — correcting wrong answers without flipping right ones — determines net benefit.

Industry relevance. The results give a concrete reason to separate perception quality from planning quality when selecting models: a model can parse cube faces well and still fail at action selection, and an open-weight model can match on reconstruction while clustering near chance on causal evaluation. The abstention analysis also speaks to when a system should defer rather than act.

Future Directions

• Extend to longer horizons. The authors state that evaluations are limited to shallow horizons because of poor MLLM performance, but note the framework could produce arbitrarily long-horizon tasks.
• Build explicit pre-action evaluation, selective control, and recovery mechanisms. The conclusion argues these are needed rather than better perception alone; the recovery results suggest short rollouts, self-checks before acting, or selective abstention/deferral.
• Probe deeper attempt budgets to locate saturation points in recovery, and test whether higher-depth settings change the κ-to-adherence relationship that weakens at d=2,3.
• Test generalization beyond the cube. The limitations section notes results may not transfer to robotics or web-agent tasks, and that low scramble depths plus multiple-choice formats limit insight into complex reasoning errors.

Target Audience

Researchers and engineers working on multimodal LLM evaluation, spatial and sequential reasoning, agentic control, or self-reflection and self-correction methods. The paper is also useful for practitioners choosing between closed- and open-weight models for multi-step interactive tasks, since it separates perception failures from planning and selection failures under matched conditions.

Authors’ abstract

We introduce Cube Bench, a Rubik's-cube benchmark for evaluating spatial and sequential reasoning in multimodal large language models (MLLMs). The benchmark decomposes performance into five skills: (i) reconstructing cube faces from images and text, (ii) choosing the optimal next move, (iii) predicting the outcome of a candidate move without applying it, (iv) executing multi-step plans while recovering from mistakes, and (v) detecting and revising one's own errors. Using a shared set of scrambled cube states, identical prompts and parsers, and a single distance-to-solved metric, we compare recent MLLMs side by side as a function of scramble depth. Across seven MLLMs, accuracy drops sharply with depth; once a trajectory stalls or diverges, models rarely recover, and high face-reconstruction accuracy does not guarantee competent action selection or multi-step execution. A pronounced closed- vs open-source gap emerges: the strongest closed model leads on both single-step perception tasks and multi-step control tasks, while open-weight models cluster near chance on the hardest settings; yet even the best MLLM degrades at higher cube complexity. A simple self-correction via reflective thinking yields modest gains but can also introduce overthinking. Cube Bench offers a compact, reproducible probe of sequential spatial reasoning in MLLMs.