TerraVis: Towards Evaluation of World-Grounded Visual Consistency in Text-to-Image Generation via MLLM Workflows

AI summary

Overview

Research area: Evaluation of text-to-image (T2I) generation, specifically a new evaluation dimension the authors call world-grounded visual consistency, plus multimodal large language model (MLLM) based evaluation workflows.

Technical level: Intermediate. The framework itself is described procedurally and is easy to follow, but interpreting the paper's results requires familiarity with T2I evaluation metrics (FID, CLIPScore, ImageReward, HPS, MUSIQ, etc.), correlation statistics (Spearman's ρ, Kendall's τ, Pearson's r), and MLLM prompting.

Scope: The paper proposes TerraVis, a taxonomy-guided, multi-stage MLLM workflow that detects and scores violations of real-world plausibility (malformed structures, impossible anatomy, implausible interactions, inconsistent spatial relations) in generated images, and benchmarks it against six representative T2I evaluation metrics on two prompt benchmarks and five T2I models.

What This Paper Is About

Existing T2I metrics measure photorealism, aesthetics, text-image alignment, human preference, or distributional similarity, but none directly asks whether the depicted world is physically and structurally plausible. An image can score highly on all of those dimensions while showing, for example, a cat physically embedded in or supported by a shoe. TerraVis addresses this gap by defining world consistency as an explicit evaluation target and building a structured detection-and-scoring framework for it, then testing whether that framework agrees better with human judgments than existing metrics do.

Key Contributions

1. A new evaluation dimension. The paper introduces world-grounded visual consistency (shortened to "world consistency"), defined as the extent to which a depiction of a macroscopic object or scene visually conforms to real-world constraints such as plausible object structure and anatomy, valid text or symbolic content, physically plausible interactions and environmental effects, spatially plausible scale, depth ordering and continuity, and commonsense compatibility between objects, actions, events, and scene context.

2. A structured violation taxonomy. Violations are organized into three hierarchical levels comprising 18 types total: 7 object-level, 8 interaction-level, and 3 scene-level. Detailed definitions and criteria are given in the paper's Appendix A.

3. TerraVis, a four-step scoring framework. The framework performs (1) eligibility checking, (2) taxonomy-guided violation detection, (3) severity classification into major vs. minor based on whether a semantically central element is affected, and (4) aggregation into a score via S = exp[−λ(N_maj + α N_min)], using λ = 1 and α = 0.5. The framework is model-agnostic: steps 1–3 can be done by human annotators or by an MLLM, while the score itself is computed deterministically.

4. Benchmarking and human correlation study. TerraVis is compared against six representative metrics plus a holistic MLLM baseline (TerraVis-H) and a GPTScore baseline, using over 13K human annotations collected on Amazon Mechanical Turk (5-point Likert scale plus an N/A option, three independent annotators per image, 50% randomly selected subset of generated images).

Main Findings

• Existing metrics track world consistency poorly and inconsistently. On COCO-T2I, pooled correlations range from ρ = 0.06 (CLIPScore) and ρ = 0.06 (MUSIQ) up to ρ = 0.29 (RAHF); on GenAI-Bench, MUSIQ reaches ρ = −0.02, CLIPScore ρ = −0.16, and ImageReward ρ = −0.08, while RAHF reaches ρ = 0.37. Correlations are strongly model-dependent, with negative or near-zero values such as TIFA on SD 3.5-Large and HPSv3 on Nano Banana Pro.

• TerraVis has the strongest agreement with human judgments among compared metrics. Pooled results on COCO-T2I are ρ = 0.44 / τ = 0.37 for TerraVis, versus ρ = 0.28 / τ = 0.24 for TerraVis-H; on GenAI-Bench, ρ = 0.43 / τ = 0.35 for TerraVis versus ρ = 0.28 / τ = 0.23 for TerraVis-H. Model-wise, TerraVis reaches ρ = 0.55 on Qwen-Image (COCO-T2I) and ρ = 0.48 on GPT Image 1.5 (GenAI-Bench).

• Decomposition beats holistic prompting. TerraVis-H uses the same taxonomy descriptions and instructions in a single prompt and consistently scores lower. The paper attributes this to TerraVis-H having high precision but low recall, while the staged workflow improves recall while maintaining high precision.

• GPTScore, the holistic MLLM baseline without the taxonomy, is near-useless. It shows near-zero correlations and assigns almost uniformly high scores; on COCO-T2I it produced constant predictions, making rank correlations undefined ("–").

• TerraVis works on stylized images too. Using SigLIP-so400m to split evaluations into photorealistic and stylized subsets, TerraVis achieves higher Spearman correlation on stylized images (ρ = 0.539) than on photorealistic ones (ρ = 0.413).

• Results are robust across MLLM evaluators, with a caveat. Replacing GPT-5.5 with four open-source MLLMs yields comparable correlations: on COCO-T2I, Gemma 4 31B reaches 0.42 / 0.32 versus GPT-5.5 at 0.44 / 0.37; on GenAI-Bench, Gemma 4 31B reaches 0.51 / 0.38 versus GPT-5.5 at 0.43 / 0.35. Mean pairwise Kendall's τ across all five evaluators is 0.62 (COCO-T2I) and 0.64 (GenAI-Bench), but agreement within groups is higher than between them: GPT-5.5 and Gemma 4 31B agree at τ = 0.80 on both benchmarks, while the remaining three agree at mean pairwise τ = 0.87. The paper recommends Gemma 4 31B as an effective open-source alternative to GPT-5.5.

• Strong conventional performance does not imply good world consistency. Across technical quality, aesthetics, alignment, preference, and world consistency, GPT Image 1.5 ranks first in 9 of 16 comparisons across both benchmarks, including 8 of 12 on conventional metrics, and achieves the best mean rank on COCO-T2I across all eight metrics (1.38), followed by Nano Banana Pro (2.75), FLUX.2-dev (3.00), Qwen-Image (3.88), and SD 3.5-Large (4.00). However, FLUX.2-dev ranks first in world consistency on GenAI-Bench according to both TerraVis and human judgments, ahead of both proprietary models — the authors suggest qualitatively that FLUX.2-dev tends to generate simpler scenes, reducing violation risk.

• Every taxonomy level contributes. Per-level ablations (aggregated across both benchmarks) give standalone ρ of 0.384 (object, 7 types), 0.253 (interaction, 8 types), and 0.174 (scene, 3 types); leave-one-out changes in ρ are −0.149, −0.033, and −0.016 respectively, with the full 18-type framework reaching ρ = 0.428.

• Severity weighting and exponential aggregation matter separately. Comparing aggregation rules, Pearson's r improves from 0.3634 (uniform pass-rate) and 0.3630 (severity pass-rate) to 0.4212 (uniform exponential) and 0.4229 (TerraVis exponential). Spearman's ρ improves from 0.4264 (both pass-rate and uniform exponential) to 0.4281 (both severity-weighted variants), and τ from 0.3471 to 0.3477 — meaning exponential aggregation drives the Pearson gain while severity weighting drives the small rank-correlation gain.

• Four disagreement patterns characterize the remaining errors. Case studies identify human oversight (TerraVis catching symbolic/textual violations, e.g. 0.135 versus human 1.0 on SD 3.5-Large / GenAI-Bench-0347), false positives (0.050 versus human 1.0 on Nano Banana Pro / GenAI-Bench-0296), missed violations (1.000 versus human 0.0 on FLUX.2-dev / GenAI-Bench-0354, with anatomical distortions in children's hands and arms), and rater disagreement (0.011 versus human ratings of 3, 5, and 1 on Nano Banana Pro / GenAI-Bench-0584).

Methodology in Plain English

The authors treat world-consistency evaluation as a sequence of narrow judgments rather than one overall opinion. Given an image, an MLLM first asks whether the image is even eligible — if it is purely abstract, symbolic, informational, interface-based, layout-based, or microscopic, it is scored N/A. Eligible images are then checked against all 18 taxonomy violation types independently, and each detected violation is labeled major (affecting a primary subject or central object) or minor (affecting a peripheral or background element). The final score is deterministic: S = exp[−λ(N_maj + α N_min)], so each additional violation reduces the score, with the steepest drop at the first few violations and diminishing marginal decreases thereafter. λ controls overall penalty strength and α controls how much minor violations count relative to major ones; the paper fixes λ = 1 and α = 0.5.

To test whether this structured process is actually better, they also built TerraVis-H, which gives the MLLM the same taxonomy but in one holistic prompt, and a GPTScore baseline that simply asks an MLLM for an integer 1–5 world-consistency rating. Evaluation uses prompts from COCO-T2I (200 prompts) and GenAI-Bench (1,600 prompts), with images generated at 1024×1024 from SD 3.5-Large, Qwen-Image, FLUX.2-dev, GPT Image 1.5, and Nano Banana Pro using each model's default configuration. On average, less than 1% of GenAI-Bench samples are out of scope. Baselines include MUSIQ (KonIQ checkpoint), LAION-Aesthetic (Aesthetic Predictor V2.5), CLIPScore (SigLIP2), TIFA (Gemma 4 31B for QA generation and answering), VQAScore (Qwen-3.5), ImageReward, RAHF, and HPSv3. All TerraVis experiments use the GPT-5.5 API with the Batch API for parallel stage-wise calls. Agreement with humans is measured with Spearman's ρ, Kendall's τ, and Pearson's r against the AMT annotations.

Why This Matters

Impact on research. The paper argues that world consistency is a complementary evaluation dimension that existing fidelity, aesthetics, preference, and alignment metrics do not capture, and that these metrics can assign high scores to implausible images. It also shows that decomposing a subjective evaluation into taxonomy-guided atomic questions, plus severity-aware exponential aggregation, is measurably closer to human judgment than holistic MLLM scoring — a transferable design pattern for other qualitative evaluation tasks. The paper also connects to prior work (Fu et al.) showing better FID or MUSIQ scores do not necessarily mean fewer counting hallucinations, positioning world consistency as a broader version of that observation.

Real-world applications:

• Trust and safety review of generated media, where implausible anatomy or physics can signal unreliable or deceptive content.
• Dataset curation for T2I training and fine-tuning, filtering outputs that look stylistically strong but violate world constraints.
• Reward modeling and post-training signals, where TerraVis scores could act as an objective beyond aesthetics, alignment, or human preference.
• Illustration, advertising, and editorial imagery workflows, where the depicted content is expected to be a physically plausible, trustworthy representation of the world.

Industry relevance. The finding that models ranking highly on conventional metrics can still exhibit substantial world-consistency failures — and that a smaller open-weight model (FLUX.2-dev) can lead on world consistency while trailing on conventional metrics — gives model developers a reason to track this dimension separately. The paper's demonstration that an open-source evaluator (Gemma 4 31B) is a viable substitute for GPT-5.5 makes such an evaluation pipeline practical to run at scale, though the multi-stage workflow is less efficient than single-pass metrics.

Future Directions

• Improving violation detection via in-context learning. The authors report preliminary results suggesting that providing representative violation examples in context improves detection performance, and name data-centric refinement as a promising direction. This targets the recall gap and the false-positive and missed-violation cases documented in Figure 4.

• Refining the taxonomy as T2I models evolve. The paper acknowledges the taxonomy was developed through iterative refinement combining principled design and data-driven validation, but may not cover all forms of world-consistency violations in practice, so continued refinement of the taxonomy and its prompts may be needed.

• Reducing annotation ambiguity. Because human judgments of ambiguous cases and severity vary across annotators — in one case spanning the full 1–5 scale for a single image — uncertainty is injected into the reference against which automatic metrics are measured. Handling this disagreement is an open measurement problem.

• Closing the efficiency gap. The multi-stage workflow is less efficient than single-pass scoring metrics, and the paper flags this as a limitation; making the staged evaluation cheaper without losing correlation is an obvious engineering next step.

• Extending the empirical scope. The paper validates TerraVis with five T2I models and two benchmarks, and explicitly calls for establishing world consistency as a core evaluation dimension for future text-to-image models more broadly.

Target Audience

Researchers and engineers working on text-to-image generation and generative-model evaluation will get the most from this paper, particularly those building benchmarks, reward models, or post-training objectives who need an evaluation signal beyond aesthetics, alignment, and preference. It is also relevant to practitioners deploying generated imagery in contexts where physical plausibility matters, to MLLM researchers interested in decomposing subjective judgments into staged prompted workflows, and to annotation and dataset teams studying how human reliability limits automatic evaluation metrics. Readers need some grounding in T2I evaluation metrics and rank-correlation statistics to interpret the result tables, but the taxonomy and scoring procedure are described concretely enough to reimplement.

Authors’ abstract

Recent text-to-image models have made substantial progress in photorealism, aesthetics, and text-image alignment. Yet visually appealing images can still violate real-world plausibility, exhibiting malformed object structures, impossible anatomy, physically implausible interactions, or inconsistent spatial relationships. Such failures are not well captured by existing fidelity, aesthetics, preference, or alignment metrics. To address this gap, we introduce TerraVis, a framework for evaluating world-grounded visual consistency in generated images. TerraVis defines a structured taxonomy of world-consistency violations spanning object-, interaction-, and scene-level failures, and employs a multi-stage evaluation framework to identify and quantify them. Given an image, TerraVis first uses an MLLM to assess its eligibility for evaluation, then detects violations across 18 taxonomy-defined types and classifies them as minor or major to derive an overall world-consistency score. Across diverse open-source and proprietary text-to-image models on two widely used benchmarks, TerraVis achieves the strongest correlation with human judgments of world consistency among existing metrics. Our benchmark results further show that models that achieve strong performance on conventional metrics can still exhibit substantial world-consistency failures. These findings highlight world consistency as a complementary evaluation dimension and demonstrate that TerraVis enables systematic quantification, diagnosis, and comparison of such failures. Our code is publicly available at https://github.com/ShyFoo/TerraVis.