Reasoning with Image Generation

AI summary

Overview

Research area: Multimodal reasoning and agentic computer vision — specifically, using instruction-following image generation models as a reasoning tool for multimodal large language models (MLLMs).

Technical level: Intermediate. The paper assumes familiarity with chain-of-thought reasoning, multimodal LLMs, and tool-augmented agents, but its central idea is explained through worked visual examples.

Scope: The paper introduces ReImaGin, a training-free multimodal agent that interleaves text reasoning with calls to a general-purpose image generator, and evaluates it on six visual reasoning tasks with three MLLM backbones and three image generation models.

What This Paper Is About

Multimodal reasoning systems usually "think" in text: they describe an image in words and reason over that description, or they call narrow specialist tools (depth estimator, segmenter, object detector) that each perform one pre-specified operation. Both approaches are bad at tasks that need spatial or physical intuition, such as imagining an occlusion removed or synthesizing a view of a room that no single input image contains. The paper's goal is to replace that fixed toolbox with a single instruction-following image generator that accepts natural-language commands and can produce open-ended visual transformations — depth maps, floorplans, occlusion-removed scenes, solid-line redrawings — inside an iterative reasoning loop.

Key Contributions

1. A free-form image generation tool for multimodal reasoning. ReImaGin equips an MLLM with a generate_image function callable at any reasoning step with a natural-language prompt, returning a synthesized image into the agent's context. Unlike prior tool-augmented agents that depend on fixed specialist modules, one generalist generator covers many visual operations.

2. Evaluation across six diverse tasks with one unmodified tool. The same image generation model is used across depth reasoning, puzzle completion, occlusion counting, collision prediction, multi-view spatial reasoning, and a new path tracing task, with no task-specific specialization of the generator.

3. Demonstration that generative visual reasoning beats fixed tools. ReImaGin is reported to consistently outperform both text-only reasoning and the specialist-tool baseline Visual Sketchpad, with gains of up to 25% on path tracing and 40% relative on counting under partial occlusion.

4. Automated discovery of visual reasoning strategies via prompt optimization. Rather than handcrafting the strategy for when and how to generate an image, the authors run an iterative proposal-and-evaluation loop that searches over agent prompts using a small training and development set, recovering most of the gains of handcrafted strategies without human-authored examples.

Main Findings

• Generative tools beat fixed tools on most tasks. Across Gemini-3.1-Pro, GPT-5, and Qwen-3.5-27B (Table 1), ReImaGin outperforms No Tools and Visual Sketchpad on the majority of tasks. With Gemini-3.1-Pro, ReImaGin reaches 42.3 (puzzle, MIRA), 7.1 sMAPE (occlusion counting, CAPTURE), 69.7 (collision, Spatial457), 59.0 (spatial, MMSI), 89.0 (path tracing), and 94.6 (depth, BLINK).

• Sketchpad sometimes degrades below no tools at all. On puzzle completion with GPT-5, Visual Sketchpad scores 16.7% versus 29.5% for No Tools, which the authors attribute to these tasks requiring generative capability that specialist vision modules lack.

• One clear exception: depth reasoning on Gemini-3.1-Pro. ReImaGin improves over No Tools (94.6% vs. 91.7%) but trails Visual Sketchpad (99.2%), which benefits from a dedicated depth estimation specialist.

• Test-time scaling of image generation helps where generation is high-variance. On MMSI spatial reasoning, ReImaGin stitches two views together and generates a top-down map. Sampling N = 10 candidates and selecting the most faithful one raises Gemini-3.1-Pro from 51.0% to 59.0% and Qwen-3.5-27B from 42.0% to 54.3%. Even when baselines receive matched compute (majority vote over 20 sampled answers), ReImaGin still leads (59.0% vs. 57.0% on Gemini-3.1-Pro; 54.3% vs. 46.7% on Qwen-3.5-27B). The other five tasks show little variance across samples, so test-time scaling is not applied there.

• Open-weights generators help but lag overall. With Gemini-3.1-Pro as the MLLM, FLUX.2 [dev] scores 53.8 on puzzle (above Nano-Banana-Pro's 42.3) but only 89.1 on depth, below the no-tools baseline of 91.7. Qwen-Image-Edit-2511 struggles on path tracing (76.0% vs. 89.0%). Both are competitive on simpler inpainting-style operations: occlusion counting of 7.0% (FLUX.2) and 6.8% (Qwen) versus 7.1% for Nano-Banana-Pro.

• Generated images are faithful most of the time, and faithfulness predicts success. In a manual audit of 120 generated images (roughly 20 per task), 75% of generations were judged faithful, and the final answer was correct 87% of the time when the image was faithful versus 43% when it was not.

• Two distinct failure categories. Generation failures — the image model produces a wrong visualization, such as a floorplan that misplaces objects — scale with generator capability. MLLM failures — the visualization is faithful but the model misreads or mis-reasons about it — scale with MLLM capability instead.

• Automatically discovered strategies largely match human ones. With Gemini-3.1-Pro as both proposal model and solver and Nano-Banana-Pro as generator, automatic strategy improves over No Strategy on all five evaluated tasks and recovers a substantial fraction of handcrafted gains (e.g., collision 64.0 vs. 60.0 no-strategy and 69.7 handcrafted; occlusion counting 8.0 vs. 12.1 no-strategy and 7.1 handcrafted). With Qwen-3.5-27B plus Nano-Banana-2, automatic strategy improves on most tasks and effectively ties on occlusion counting and depth.

• Discovered strategies resemble human designs but can overshoot. The optimized depth prompt learns to generate a depth map; the collision prompt learns to draw an arrow from the front of the moving object. For path tracing, the handcrafted strategy makes dashed lines solid, while the discovered strategy additionally colors them — and because Nano-Banana-Pro rarely fails to make lines solid but more often fails to color correctly, this richer strategy slightly degrades performance. The composite strategy used for MMSI was not recovered; instead a panorama-like view request was discovered.

• Discovered strategies transfer across MLLMs. Applying Gemini-3.1-Pro-discovered prompts unchanged to Qwen-3.5-27B yields gains over No Tools on all five tasks and beats Visual Sketchpad and No Strategy on collision, spatial reasoning, and path tracing. Performance degrades on occlusion counting and does not improve on depth, indicating a strategy tuned for a stronger model does not always transfer.

• Cost. ReImaGin and Sketchpad are matched on MLLM reasoning cost and latency ($0.04 per instance), with ReImaGin adding $0.12 per instance for image generation. Strategy discovery costs approximately $214 per task as a one-time expense.

Methodology in Plain English

The agent is built on the ReAct pattern. On each turn, an MLLM sees a prompt containing tool descriptions, in-context examples showing how to reason visually about the task, and the running context (the question, the input images, and everything that happened before). It writes a chain-of-thought rationale, then emits a Python program. That program can call ordinary image utilities like crop, overlay, and subtract, general-purpose code like numpy, or the central generate_image tool, which takes a text prompt and optionally reference images and returns a new synthesized image. Outputs are appended back into the context as a new user message, and the loop repeats until the agent writes "Terminate," at which point the answer is parsed from the preceding text.

Because image generation is stochastic, ReImaGin optionally samples multiple candidate images for a single tool call and uses an MLLM selector to pick the one that most faithfully matches the prompt — the agent itself still sees only one returned image. This puts extra test-time compute into image quality rather than into sampling many text reasoning traces.

For strategy discovery, the authors treat "which visualization to make and how to use it" as a prompt-optimization problem. Starting from a baseline prompt with tool definitions but no task-specific visual policy, each round evaluates candidate prompts on a training and development split, then a proposal agent sees retained prompts plus a small set of correct and incorrect trajectories and proposes new prompts. The reasoning agent and image generator stay fixed throughout; only the prompt changes. The search runs for 4 rounds with 5 candidate proposals per round, using at most 50 training and 50 development samples (fewer on MMSI due to dataset size), implemented with Opik.

Evaluation covers six tasks: BLINK for relative depth, MIRA for puzzle completion with a missing region and five candidate pieces, CAPTURE for counting occluded objects, Spatial457 for collision prediction from overhead views, MMSI for spatial reasoning across two partially overlapping views of an indoor scene, and a new path tracing task where four numbers connected to four letters by dashed lines must be matched. All tasks use multiple-choice accuracy except occlusion counting, which uses symmetric mean absolute percentage error (sMAPE, lower is better); metrics are averaged over 3 seeds with standard error reported. The provided paper content does not report the total dataset sizes for the individual benchmarks.

Why This Matters

Impact on research. The paper reframes image generation as a reasoning primitive rather than an output modality. It argues that generative tools subsume many specialist vision tools while remaining extensible, and that their usefulness scales directly with progress in image generation — as generators improve, the space of visual reasoning operations a single agent can perform expands without new engineering. It also provides evidence that MLLMs carry useful priors about which visual transformations help, since automatically discovered strategies often resemble human-designed ones.

Real-world applications.

• Spatial assistants that fuse multiple views of a home or warehouse into a single floorplan to answer questions no single camera view can answer.
• Robotics and autonomous driving, where drawing a projected trajectory and checking what it intersects supports collision reasoning and motion planning.
• Counting and inspection in cluttered environments, such as retail shelves or manufacturing lines, where occluding elements can be visually removed before counting.
• Educational and diagnostic tools that redraw hard-to-parse diagrams — dashed connections, ambiguous figures — into clearer forms before answering questions about them.

Industry relevance. The approach is training-free and modular, so it can be layered onto existing multimodal agent stacks as a tool rather than requiring a fine-tuned model. The cost profile is transparent ($0.04 per instance for MLLM reasoning, $0.12 for image generation) and the comparison with open-weights generators (FLUX.2 [dev], Qwen-Image-Edit-2511) shows where commodity models already suffice — local editing and inpainting — and where a stronger proprietary generator is still needed for spatially precise, globally consistent visualizations.

Future Directions

• Improve reliability of generative visual reasoning. The authors identify stronger image generation models, better selection and verification of intermediate visualizations, and more scalable strategy discovery as the three levers, noting that 25% of audited generations were unfaithful and that success drops sharply when they are.
• Close the strategy-discovery gap. Discovered strategies recover most but not all handcrafted gains, and the composite MMSI strategy was not recovered at all; more capable proposal models or richer search could narrow this.
• Make strategies transfer more robustly. Transferred prompts helped on five tasks but degraded occlusion counting and failed to improve depth, raising the question of how to adapt a strategy discovered on a strong model to a weaker one.
• Replace handcrafted specialists where they still win. A dedicated depth estimator beat ReImaGin on BLINK with Gemini-3.1-Pro, so the boundary between when a specialist module and when a generalist generator is preferable remains open.

Target Audience

Researchers and practitioners working on multimodal agents, tool-augmented LLMs, and visual reasoning benchmarks; engineers building agentic systems who need to decide between fixed perception modules and generative tools; and anyone studying how to automate the design of reasoning policies, since the prompt-optimization loop for discovering visual strategies is independently transferable.

Authors’ abstract

Chain-of-thought reasoning has revolutionized natural language processing by enabling large language models (LLMs) to decompose problems into intermediate steps before answering. Yet confining reasoning to the textual domain presents limitations for tasks requiring direct manipulation of visual representations. Recent efforts augment multimodal LLMs with external visual expert tools such as depth estimation or object detection modules, but these remain fundamentally limited by their reliance on narrow, rigid operations that cannot flexibly generate or transform visual content. We propose ReImaGin, which leverages image generation models as a flexible visual reasoning mechanism for multimodal LLMs: unlike fixed-function tools, they accept natural language commands and can perform open-ended visual operations, like removing an occlusion or generating a floorplan from multiple disjoint views of a room. Across six diverse visual reasoning tasks including multi-view spatial reasoning and collision prediction, ReImaGin consistently outperforms both text-only reasoning and specialist vision-tool baselines, with gains of up to 25\%, demonstrating the advantage of flexible, generative visual reasoning.