MentisOculi: Revealing the Limits of Reasoning with Mental Imagery

AI summary

Overview

Research area: Multimodal AI, specifically the evaluation of visual reasoning in frontier models (multimodal large language models, unified multimodal models, latent visual reasoning models, and video generation models).

Technical level: Intermediate — the paper is readable without deep mathematical background, but it assumes familiarity with terms like chain-of-thought, multimodal models, and benchmark evaluation.

Scope: The paper introduces MentisOculi, a procedurally generated, stratified benchmark of five multi-step visual reasoning tasks, and uses it to test whether models can use "mental imagery" — self-generated or provided visualizations — to improve reasoning.

What This Paper Is About

Frontier models are moving from merely reading images to generating them, which raises the question of whether intermediate pictures can serve as a reasoning aid, similar to human mental imagery. The problem is that while models can often generate visuals and often can reason in text, it is unclear whether visual thoughts actually help them solve multi-step problems, and unclear whether failures come from bad generation, bad interpretation, or limited reasoning. The paper builds a controlled benchmark, MentisOculi, to separate these factors and measure whether visual reasoning works at all.

Key Contributions

1. MentisOculi benchmark. A procedural, stratified suite of five multi-step reasoning tasks designed to be difficult to textualize but intuitive to solve visually: Form Board, Hinge Folding, Paper Fold, Rush Hour, and Sliding Puzzle. Each task has five difficulty levels, with 30 samples per level in the initial release, plus ground-truth visual chain-of-thought solutions.

2. A spectrum-wide analysis of machine mental imagery. The paper evaluates MLLMs (Gemini 2.5 Flash, Gemini 3 Pro, GPT-5.1, Qwen3-VL 235B-A22B Thinking), latent visual reasoning models (Qwen2.5-VL-32B fine-tuned with the Mirage and LatentSketchpad frameworks on Rush Hour, using 200 samples per level), UMMs (Gemini 2.5-I, Gemini 3-I, Emu 3.5), and video models (Veo 3.1, Wan 2.6).

3. Diagnosis of UMM failure. By substituting self-generated images with oracle visualizations, the paper separates generation errors from interpretation errors and shows that both are present, with interpretation failures persisting even when correct visuals are supplied.

4. Human reference data. A psychophysical study on Rush Hour with five PhD students (2f/3m, mean age 27), providing an upper-bound comparison and revealing that models do not scale their reasoning effort with task difficulty the way humans do.

Main Findings

• Benchmark is far from saturated: Across all tasks, performance degrades noticeably as difficulty increases, validating the stratification. With the exception of Gemini 3, models fail to reliably exceed chance even at Level 1 on all tasks except Form Board. By Level 5, all models operate at or below chance.

• Consistent model ranking: Gemini 3 performs best across tasks, followed by GPT-5.1 and Qwen3-VL. The UMM variants Gemini 3-I and Gemini 2.5-I often lag behind their corresponding MLLMs.

• Explicit visual thought is currently ineffective: No visual intervention reliably outperforms text-only baselines. Interleaved image generation, latent visual tokens, and video rollouts all failed to consistently improve multi-step reasoning.

• Latent reasoning gives brittle gains: Mirage, fine-tuned on 200 samples per level, outperforms MLLMs on Levels 2–3, but at higher levels merely matches Gemini 3 and drops to near-chance at Level 5. Compared to text-only fine-tuning, the effect of latent reasoning was limited.

• Video models never exceed chance: Despite a lenient scoring policy, Veo 3.1 and Wan 2.6 never exceed chance performance on Rush Hour, though Veo 3.1's ability to match or exceed Gemini 2.5-I on lower levels is noted as evidence of potential for natively visual reasoning.

• Models have the competence but not the performance: When given a lossless textual transcription of Rush Hour (parking-lot size, exit location, and each car's center coordinates, spatial extent, orientation, and admissible motion axis), Gemini 3 and GPT-5.1 perform on par with humans. This shows the bottleneck is visual understanding and manipulation, not reasoning capability.

• Oracle visuals expose a dual failure: Providing ground-truth visualizations lets Gemini 3-I and Gemini 2.5-I achieve peak accuracy on Form Board, far exceeding chance and their underlying MLLMs. In Hinge Folding and Paper Fold, oracle visuals only close the gap to the underlying MLLM. On other tasks, oracle visuals are still not sufficient to reliably meet or exceed chance. Emu 3.5 shows similar trends but consistently trails the Gemini models.

• Image and text channels are weakly coupled: Even on the easiest level of Rush Hour, roughly half of all solved puzzles are solved by exactly one of the two modalities. At Level 1, image rollouts solve puzzles text reasoning misses; from Level 2 onward the text channel carries multi-step planning the image channel cannot. Gemini 3-I, with nearly twice Gemini 2.5-I's text accuracy, shows a higher fraction of disagreeing puzzles.

• Common reasoning enhancements do not transfer: In-context learning yields no systematic improvement beyond Level 1, with no difference between ICL examples containing images and those without. Prompt optimization using OpenEvolve (57 variants over 50 iterations) did not beat the default prompt. Increasing the reasoning budget did not improve accuracy — GPT-5.1 and Gemini 3 used on average 13 times more tokens under high reasoning settings with largely unchanged performance. Tool use yielded no meaningful gains, with models mainly using image preprocessing tools such as cropping and resizing.

• Humans outperform models, and adapt effort differently: Humans achieve more than 60 percent accuracy at Level 5 of Rush Hour. Gemini 3 performs like humans limited to 5 to 10 seconds. Humans reliably spend more time on harder puzzles, while Gemini 3 shows no increase in token usage from Level 3 to Level 5.

• Visual aids can align compute with human difficulty: In an appendix analysis, providing models with visual chain-of-thought (Gemini 2.5-I with oracle visual CoT and Qwen3-VL with in-context-learning examples containing visual CoT) induces a highly linear relationship (R² ≥ 0.98) between tokens spent and the time an average human requires. However, this alignment was not a definitive predictor of task success.

• Visualization is expensive: Generating a video reasoning trace with Veo 3.1 costs $3.2 per sample — over 21 times more than Gemini 2.5-I and over 60,000 times more than Gemini 2.5 — despite roughly similar performance.

Methodology in Plain English

The researchers first defined what a good test of mental imagery should look like. They listed six requirements: tasks must be genuinely spatial (about relations, geometry, and transformations, not common knowledge), information-dense enough that a short text description is impractical, sequential (requiring repeated updates to a mental state where each action depends on the last), procedurally generated, stratified by difficulty, and feasible for current image generators to represent. Existing benchmarks frequently fail one or more of these, so the team built five new tasks that satisfy all six.

The five tasks are Form Board (pick shapes that tile a silhouette without gaps or overlaps), Hinge Folding (predict the 90-degree-step rotation of each hinge in a chain to match a target silhouette), Paper Fold (identify the hole pattern produced by folding and punching a sheet), Rush Hour (drive a red vehicle out of a crowded lot by moving blocking vehicles forward and backward), and Sliding Puzzle (output up, down, left, right moves of the blank tile to restore a shuffled image drawn from ImageNet-1k). Difficulty is controlled by the minimum number of steps required, from one step at Level 1 to five at Level 5. Each generator also produces a ground-truth visual chain of thought, so the researchers can inject correct intermediate images into a model's reasoning and see whether that helps.

Evaluation spans four model families. MLLMs produce text only. Latent visual reasoning models interleave text with visually grounded latent tokens. UMMs generate actual images interleaved with text. Video models produce purely visual rollouts conditioned on a prompt and an initial frame. Text outputs are scored either by exact match (Form Board, Paper Fold) or by simulating the predicted action sequence and checking whether the terminal state satisfies the goal (Hinge Folding, Sliding Puzzle, Rush Hour). For video output on Rush Hour, an automatic rater processes frames one by one, uses color and spatial consistency to recover object trajectories, extracts a lenient implied action sequence, and verifies it by simulation. Invalid identifiers or out-of-bounds moves are scored as incorrect.

To isolate the source of UMM failures, the paper runs three additional experiments: a text-transcription version of Rush Hour to test whether models have the underlying reasoning competence, an oracle-visual condition that replaces self-generated images with ground-truth ones, and a comparison of which puzzles the text channel versus the image channel solves. Finally, they test whether four standard reasoning enhancements (in-context learning, prompt optimization, larger reasoning budgets, tool use) transfer to visual reasoning, and they collect human data on Rush Hour for comparison.

Why This Matters

Impact on research: The paper argues that the field lacks a rigorous framework for disentangling whether visual reasoning failures come from reasoning deficits, flawed image generation, or an inability to interpret self-generated cues. MentisOculi provides a controlled testbed for exactly that, and its procedural design allows future, harder instances to be generated, which also addresses data contamination over time. The finding that models fail to benefit even from ground-truth visualizations challenges the assumption that adding a visual modality to a reasoning trace is inherently useful.

Real-world applications:

• Navigation and spatial planning assistants: Rush Hour-style tasks mirror route planning through constrained spaces with dependent obstacles, relevant to any system that must plan physical maneuvers.
• Design and manufacturing workflows: Form Board and Hinge Folding test shape comparison and geometric constraint satisfaction under translation and rotation, the kind of reasoning involved in assembling panels, parts, or folding structures.
• Image restoration and editing: Sliding Puzzle tests whether a model can plan the reconstruction of a scrambled natural image, relevant to editing tools that must reason about spatial coherence.
• Video generation as a decision-making interface: The paper tests whether purely visual rollouts (Veo 3.1, Wan 2.6) can act as reasoning, and reports both the current performance ceiling and the cost overhead relative to text approaches.

Industry relevance: The paper reports a concrete cost comparison — $3.2 per sample for Veo 3.1 versus over 21 times less for Gemini 2.5-I and over 60,000 times less for Gemini 2.5, at roughly similar performance. For teams deciding whether to invest in unified multimodal or video-based reasoning pipelines, this is a direct economic signal: visual reasoning must justify its overhead through clear performance gains or qualitatively new capabilities, which the current results do not show.

Future Directions

• Closing the generation gap: Performance increases on some tasks when generation errors are curbed, so improving image-generation fidelity within a reasoning chain is a concrete next step.
• Fixing interpretation errors: Since models fail to leverage even oracle visualizations, they must learn to treat visual states as actionable evidence. The paper speculates this will require dedicated training data and a greater focus on multi-step visual reasoning by model developers.
• Task-specific rather than general visual aids: The results suggest the key question is not whether mental imagery helps in general but which visual aids are useful for which tasks, since effects are neither uniform nor reliable.
• Beyond the current difficulty ceiling: Level 5 already challenges current models, so higher-complexity instances can be generated from the released code to keep the benchmark informative as models improve.

Target Audience

Researchers and engineers working on multimodal models, visual reasoning benchmarks, and chain-of-thought methods; teams evaluating unified multimodal or video-generation models for reasoning applications; and cognitive science or psychophysics researchers interested in comparisons between human mental imagery and machine visual reasoning. The paper is most useful to readers who already know what MLLMs, UMMs, and chain-of-thought prompting are, but it does not require specialized mathematical background.

Authors’ abstract

Frontier models are transitioning from multimodal large language models (MLLMs) that merely ingest visual information to unified multimodal models (UMMs) capable of native interleaved generation. This shift has sparked interest in using intermediate visualizations as a reasoning aid, akin to human mental imagery. Central to this idea is the ability to form, maintain, and manipulate visual representations in a goal-oriented manner. To evaluate and probe this capability, we develop MentisOculi, a procedural, stratified suite of multi-step reasoning problems amenable to visual solution, tuned to challenge frontier models. Evaluating visual strategies ranging from latent tokens to explicit generated imagery, we find they generally fail to improve performance. Analysis of UMMs specifically exposes a critical limitation: While they possess the textual reasoning capacity to solve a task and can sometimes generate correct visuals, they suffer from compounding generation errors and fail to leverage even ground-truth visualizations. Our findings suggest that despite their inherent appeal, visual thoughts do not yet benefit model reasoning. MentisOculi establishes the necessary foundation to analyze and close this gap across diverse model families.