Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Models

AI summary

Overview

Research area: Computer Vision — evaluation and improvement of text-to-image (T2I) generative models, specifically their spatial understanding.

Technical level: Intermediate. The paper is readable without deep generative-model expertise, but familiarity with diffusion models, multimodal LLMs, and standard T2I benchmarks (e.g., GenEval, DPG-Bench) helps.

One-sentence scope: The paper introduces SpatialGenEval, a benchmark of 1,230 long, information-dense prompts paired with 12,300 multiple-choice questions that systematically probes whether T2I models can render where objects are, how they are arranged, and why they interact — plus SpatialT2I, a companion fine-tuning dataset that measurably improves those abilities.

What This Paper Is About

Text-to-image models have become very good at drawing the right objects with the right attributes — a red apple, a wooden chair, a fluffy cat. They are much worse at getting the spatial relationships right: placing things in the correct position, orienting them, layering them with occlusion, comparing their sizes, or depicting cause and effect. Existing benchmarks largely miss this failure because they use short, simple prompts and coarse yes/no or detection-based scoring, which only verifies that objects exist rather than that the scene is spatially coherent. This paper builds a harder test to expose those weaknesses and a companion dataset to help fix them.

Key Contributions

1. SpatialGenEval benchmark. A hierarchical framework splitting spatial intelligence into 4 domains and 10 sub-domains, instantiated as 1,230 long (46–97 words) information-dense prompts across 25 real-world scenes, each paired with 10 omni-dimensional multiple-choice questions (12,300 total). Each prompt deliberately packs all 10 spatial constraints into a single coherent scene.
2. A large-scale diagnostic evaluation of 23 state-of-the-art T2I models. The authors benchmark diffusion, autoregressive, unified, and closed-source models to identify where spatial intelligence breaks down, pinpointing spatial reasoning (comparison, proximity, occlusion) as the universal bottleneck.
3. SpatialT2I, a fine-tuning dataset. 15,400 text-image pairs built by filtering high-quality generated images and using an MLLM to rewrite prompts so they better match the images while preserving information density.
4. Empirical demonstration of a data-centric path forward. Fine-tuning Stable Diffusion-XL, UniWorld-V1, and OmniGen2 on SpatialT2I yields consistent gains of +4.2%, +5.7%, and +4.4% overall, with the largest improvements concentrated in the previously weakest sub-domains.

Main Findings

• Spatial reasoning is the primary bottleneck. Across all 23 models, comparison and occlusion tasks score below roughly 30%, barely above the 20% random-selection baseline. Models can draw objects; they cannot bind object properties to the structural logic of a scene (relative size, physical layering). Error rates for relational reasoning typically exceed 35%, the highest of any error category.
• Foundation is strong, higher-order skills are weak. Top models exceed 70% on object and attribute generation, but performance drops steadily through perception, reasoning, and interaction. The gap between drawing objects and organizing them per rules is the central finding.
• Open-source is closing the gap but nobody passes. Qwen-Image (60.6%) nearly matches the best closed-source model, Seed Dream 4.0 (62.7%). However, the top score hovers around the 60-point mark, indicating even the best models have only a rudimentary grasp of complex spatial intelligence.
• Text encoder capability is a key determinant. Models using strong LLM text encoders (Qwen-Image) or T5 (FLUX.1, SD-3) substantially outperform older CLIP-only models (SD-1.5 at 28.5%). Understanding dense prompts appears prerequisite to generating spatially faithful scenes.
• Scale and architecture offer two improvement paths. Larger diffusion models generally score higher (20B Qwen-Image > 8B SD-3.5-L), but unified understanding-plus-generation models are more parameter-efficient — 7B Bagel (57.0%) is comparable to 12B FLUX.1-krea (58.5%).
• Benchmark rankings agree with existing benchmarks. Model orderings on SpatialGenEval align closely with meta-rankings from GenEval, DPG-Bench, Wise, and TIIF-Bench, validating it as a reliable general indicator of generative capability.
• Judge choice does not drive results. GPT-4o and Qwen2.5-VL-72B produce nearly identical scores and identical rankings. Against human annotators on 200 sampled images, all MLLM judges align well (Gemini-2.5-Pro 84.2%, Qwen2.5-VL-72B 80.4%, GPT-4o 78.8% balanced accuracy).
• Data quality and quantity both matter for fine-tuning. Higher-scoring model subsets contribute more to downstream gains, and performance improves monotonically as training data scales from 0% to 100%.

Methodology in Plain English

The authors first define what "spatial intelligence" means by breaking it into a ladder of four domains and ten sub-domains:

• Spatial Foundation: Object Category (S1), Object Attribution (S2) — drawing the right things with the right properties.
• Spatial Perception: Spatial Position (S3), Spatial Orientation (S4), Spatial Layout (S5) — placing, rotating, and arranging objects on the canvas.
• Spatial Reasoning: Spatial Comparison (S6), Spatial Proximity (S7), Spatial Occlusion (S8) — relative size/quantity, precise distances like "touching," and depth layering.
• Spatial Interaction: Spatial Motion (S9), Spatial Causal (S10) — mid-action poses and cause-effect relationships.

They then use a strong multimodal model (Gemini 2.5 Pro) to generate prompts: give it one of 25 real-world scenes plus the ten sub-domain definitions, and ask it to write a single fluent ~60-word prompt embedding all ten spatial constraints. Human reviewers then fix awkward phrasing, remove logical contradictions (e.g., circular "left of" chains), and replace unusual vocabulary so the test measures spatial reasoning rather than lexical knowledge. The same model generates ten multiple-choice questions per prompt, one per sub-domain, and humans verify that no question leaks its own answer.

Evaluation uses another multimodal LLM (Qwen2.5-VL-72B) as a judge: shown only the generated image and the questions — never the original prompt — it picks the best of four plausible options plus an "E: None" escape option so it can decline rather than guess when the image is simply wrong. Each answer is decided by majority vote over five rounds (correct only if chosen at least four times), which reduces randomness. The whole benchmark runs in about 40 minutes on 8 GPUs at roughly 1.8 seconds per image.

For the fine-tuning dataset, the authors repeat the prompt-generation process, generate images with 14 top-performing open-source models, filter out low-quality "Design" scenes, evaluate each image with an MLLM, and have Gemini 2.5 Pro mildly rewrite each prompt so it more accurately describes the accompanying image while preserving information density — yielding 15,400 image-text pairs used to fine-tune three existing models.

Why This Matters

Impact on research: The paper reframes T2I evaluation from "did you draw the object?" to "did you build a coherent spatial scene?" It provides a fine-grained diagnostic tool that separates ten distinct failure modes, showing the community exactly which capability is broken rather than reporting a single aggregate score. This makes it easier to target architectural or training interventions.

Real-world applications:

• Automated content and advertising creation — marketing images and product mockups require accurate item placement, occlusion, and relative scale to look credible; models that misplace objects produce unusable assets.
• Architectural and interior design visualization — layouts, spatial grouping, and views through occluded spaces are the core of design renderings, where "approximately right" placement fails the task.
• Robotics and embodied AI simulation — synthetic training data for navigation or manipulation requires physically plausible scenes where objects sit, stack, and interact correctly.
• Storyboarding, comics, and educational illustration — narrative images depend on cause-and-effect depiction (a hammer striking a nail, a rock producing ripples), a dimension current models score near random on.

Industry relevance: The finding that strong LLM text encoders correlate with spatial competence has direct implications for model architecture decisions. The SpatialT2I result also shows that targeted, information-dense training data yields measurable gains across three different model families, suggesting a cheaper route to improvement than scaling parameters alone. For teams shipping image-generation products, the benchmark offers a concrete regression test suite covering failure modes that standard benchmarks do not surface.

Future Directions

• Scaling SpatialT2I further. The paper observes a monotonic improvement trend from 0% to 100% of training data but does not establish where returns diminish. Larger and more diverse spatial datasets are a natural next test.
• Why spatial reasoning specifically lags. The authors show comparison and occlusion are near-random but do not isolate the cause — whether it is a text-encoder limitation, a training-data limitation, or an architectural inability to maintain 3D-consistent representations. Ablations targeting each would clarify which intervention matters most.
• Expanding beyond static single-image scenes. Motion and causal interaction are currently evaluated as snapshots. Extending the framework to video, multi-frame sequences, or interactive editing would test temporal spatial consistency, which the current benchmark cannot capture.
• Closing the human-judge gap on hard sub-domains. MLLM evaluators align with humans near 80% but drop noticeably on spatial reasoning questions — precisely the hardest and most important dimension. Improving automated judges for relational logic is a prerequisite for scaling evaluation.

Target Audience

Researchers and engineers working on text-to-image generation who need a rigorous way to measure and improve spatial fidelity; benchmark designers interested in information-dense prompt construction and MLLM-as-judge protocols with human validation; practitioners building production image-generation pipelines who want a diagnostic suite for locating specific spatial failure modes; and students entering multimodal generative modeling who want a clear, well-organized taxonomy of what spatial intelligence in image generation actually consists of.

Authors’ abstract

Text-to-image (T2I) models have achieved remarkable success in generating high-fidelity images, but they often fail in handling complex spatial relationships, e.g., spatial perception, reasoning, or interaction. These critical aspects are largely overlooked by current benchmarks due to their short or information-sparse prompt design. In this paper, we introduce SpatialGenEval, a new benchmark designed to systematically evaluate the spatial intelligence of T2I models, covering two key aspects: (1) SpatialGenEval involves 1,230 long, information-dense prompts across 25 real-world scenes. Each prompt integrates 10 spatial sub-domains and corresponding 10 multi-choice question-answer pairs, ranging from object position and layout to occlusion and causality. Our extensive evaluation of 21 state-of-the-art models reveals that higher-order spatial reasoning remains a primary bottleneck. (2) To demonstrate that the utility of our information-dense design goes beyond simple evaluation, we also construct the SpatialT2I dataset. It contains 15,400 text-image pairs with rewritten prompts to ensure image consistency while preserving information density. Fine-tuned results on current foundation models (i.e., Stable Diffusion-XL, Uniworld-V1, OmniGen2) yield consistent performance gains (+4.2%, +5.7%, +4.4%) and more realistic effects in spatial relations, highlighting a data-centric paradigm to achieve spatial intelligence in T2I models.