MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation

AI summary

Overview

Research area: Computer Vision — evaluation of multi-reference image generation (image generation conditioned on several reference images plus a text instruction).

Technical level: Intermediate. Readers need some familiarity with diffusion/flow-based image generators, vision-language model (VLM) judges, and standard evaluation metrics, but the paper's core ideas are conceptually straightforward.

Scope (one sentence): The paper introduces MDMR-Bench, a 2,390-case benchmark organized around five capability families of multi-reference image generation, paired with a checklist-based evaluation protocol of global-level and atomic-level yes/no questions scored across five diagnostic dimensions.

What This Paper Is About

Image generators increasingly accept several reference images at once so users can fuse entities, transfer attributes, or edit structured designs — but the benchmarks used to measure this skill are small, narrow in task coverage, and rely on coarse holistic scores. The authors argue that existing benchmarks collapse the task into a single axis (number of references, or coarse task category) and contain visually simple, unnatural reference images that do not reflect real usage. MDMR-Bench is built to fix both problems: broader dedicated task coverage across five capability families, and a fine-grained checklist protocol that breaks generation quality into verifiable questions rather than one scalar score.

Key Contributions

1. MDMR-Bench itself — a 2,390-case multi-reference generation benchmark drawn from visually realistic sources (real-world images plus high-quality synthetic images), explicitly prioritizing capability diversity over raw case count. The paper notes it is smaller than MultiBanana (3,769) and VTEdit-Bench (24,220) by sample count but claims broader dedicated task coverage than MultiBanana.

2. Five capability settings with dedicated coverage — entity fusion and attribute transfer (with 2 to 4 reference images), multi-element extraction from a single reference image, infographic manipulation involving artistic text and logos, and visual reasoning-intensive generation requiring implicit relationships to be inferred across references.

3. First checklist-based evaluation protocol for multi-reference generation — each output is judged with global-level questions (overall prompt requirements) and atomic-level questions (object- and attribute-level details), scored on Following, Consistency, Composition, Minimality, and Quality. The paper reports that this protocol achieves substantially higher inter-judge agreement than score-based evaluation (Figure 2, measured with Krippendorff's alpha).

4. A benchmarked leaderboard and failure-mode analysis — eleven models (six proprietary, five open-source) evaluated with a default Gemini-3-Pro judge, plus cross-judge robustness checks, human correlation studies, and diagnostics on reference content type and number of references.

Main Findings

• Proprietary models lead across every dimension. Under the default Gemini-3-Pro judge, GPT-Image-2 ranks first with an Overall of 88.8, followed by Nano Banana Pro at 85.1 and Nano Banana 2 at 81.2. FLUX.2 is the strongest open-source model at 51.3, ahead of Qwen-Image-Edit (49.9), BAGEL (41.9), DreamOmni2 (41.9), and OmniGen2 (28.6). The paper states that proprietary models outperform open-source models on all five evaluation dimensions.

• Different model families fail in different ways. Proprietary models score high on Following and Quality but drop on Consistency, Composition, and especially Minimality — meaning they sometimes copy the wrong details, compose references imperfectly, or make unrequested changes. Open-source models often score higher on Quality than on Following and Consistency, indicating visually plausible outputs that miss precise multi-reference constraints.

• Entity fusion is the easiest family; infographic manipulation is the hardest. In task-level results, GPT-Image-2 scores 91.5 on Entity Fusion versus 87.6 on Infographic Manipulation; the paper notes that except for Nano Banana Pro, all models perform noticeably worse on Infographic Manipulation. Seedream 4.0 scores 77.9 on Entity Fusion versus 67.9 on Infographic Manipulation.

• Multi-element extraction is often easier than multi-reference fusion. The authors observe that extracting several elements from one image is frequently easier than combining multiple separate reference images, hypothesizing that consolidating elements in a single image reduces effective multi-reference complexity, whereas more reference images mean longer visual token sequences and harder composition.

• More reference images means lower scores. On the Attribute Transfer subsets, going from two to four references lowers average Overall by roughly 0.106 and Composition by roughly 0.122 on the plotted 0–1 scale.

• Human-related attribute transfer is a weak spot. Models perform substantially worse on facial expression, makeup, outfit, pose, and hairstyle transfer, which the authors attribute to the difficulty of precise human appearance and identity manipulation.

• Trade-offs hidden by aggregate scores. GPT-Image-1.5 shows strong Following across content types but a much lower Minimality score, meaning it satisfies the transfer request while altering content that should have stayed untouched.

• Reasoning performance tracks visual understanding. Models with stronger vision-language understanding do better on reasoning-intensive generation; BAGEL attains relatively strong reasoning performance despite weaker overall image quality, which the authors connect to its unified mixture-of-transformers architecture modeling understanding and generation jointly.

• The checklist protocol is more stable across judges than score-based scoring. Figure 2 shows consistently higher inter-judge agreement for checklist-based evaluation. For reference, Gemini score-based evaluation reaches Krippendorff's alpha = 0.5896 against human scores.

• VLMs are reasonable human proxies. On 5,924 checklist decisions from 200 cases and 800 outputs, Gemini-3-Pro reaches an agreement ratio of 0.853, Pearson 0.643, Spearman 0.862, and Cohen's kappa 0.642 against human majority votes. GPT-5.4 reaches 0.854 / 0.621 / 0.834 / 0.618; Doubao-1.8 reaches 0.849 / 0.617 / 0.784 / 0.617; Qwen3-VL-32B reaches 0.800 / 0.475 / 0.725 / 0.472. Human inter-annotator agreement is Krippendorff's alpha = 0.88.

• Rankings largely survive a different judge. With GPT-5.4 as judge, GPT-Image-2 (86.37), Nano Banana Pro (81.18), and Nano Banana 2 (78.41) remain the top three, while Seedream 4.0 (75.54), GPT-Image-1.5 (75.15), and Seedream 5.0 (74.73) reorder.

• Checklist items are mostly valid. A human audit of 2,962 items from 100 cases found 2,925 valid and 37 invalid, a validity rate of 98.75% and an error rate of 1.25%.

• Synthetic references do not materially inflate scores. In a paired experiment on 100 cases, Overall rises by 0.43 points for Nano Banana Pro (87.01 real to 87.44 synthetic) and 0.23 for Seedream 5.0 (82.93 to 83.16), with the gap between the two models shifting from 4.08 to 4.28 points.

• Scores are reasonably reproducible. Three independent runs on the 101 Multi-Element Extraction cases give means of 86.2 (std 1.37) for Nano Banana Pro, 85.0 (std 1.12) for Nano Banana 2, 83.2 (std 0.05) for Seedream 4.0, and 82.7 (std 1.51) for Seedream 5.0.

Methodology in Plain English

The authors build the benchmark with a three-stage automated pipeline. First they assemble an image gallery of roughly 500K images, mostly from LAION at resolutions above 1024, supplemented by images synthesized with Nano Banana Pro and GPT-Image-1.5 from captions of real images so the content stays similar but the image is cleaner. Each image is annotated by Gemini-3-Pro with semantic tags — content categories like people, animals, objects, scenes, posters; fine-grained attributes like hairstyle, texture, facial expression, filter, painting style; and auxiliary metadata like number of people, age, body framing, and facing direction. Second, they define "task buckets" specifying a task type plus the reference-sampling rules (for example, a three-image fusion bucket requiring three references sampled under predefined entity and attribute constraints), split into multiple buckets by image category and composition, each filled with 10 to 50 samples using entity tags (person, animal, object, scene, text, outfit, decoration) and attribute tags (artistic style, filter, era, texture, color, hair, expression, makeup, pose). Third, Gemini-3-Pro checks that sampled combinations are valid, writes instructions in both English and Chinese, and humans filter out low-quality or ambiguous cases.

The resulting composition is 936 entity fusion cases, 982 attribute transfer cases (each with 2 to 4 references), 101 multi-element extraction cases, 278 infographic cases (155 Chinese, 123 English) covering addition, replacement, color modification, font change, texture/material editing, and style transfer, and visual reasoning cases spanning temporal, stylistic, state, spatial, and layout transformations split into analogical and logical reasoning.

For evaluation, Gemini-3-Pro auto-generates yes/no checklist questions per case from the instruction and references, grouped into global-level and atomic-level queries under each of five dimensions. A judge model answers "Y" or "N" with a short reason, and each dimension's score is its yes rate. The item-level Overall weights dimensions at 0.20 (Following), 0.25 (Consistency), 0.25 (Composition), 0.20 (Minimality), and 0.10 (Quality). To stop large subsets from dominating, Entity Fusion, Visual Reasoning, Attribute Transfer, and Infographic Manipulation are first averaged within each subtask and then across subtasks, while Multi-Element Extraction is averaged directly; the final score is the equal-weight mean over the five task families using the default judge alone.

Why This Matters

Impact on research. The paper shifts multi-reference evaluation from a single scalar score toward decomposed, auditable questions, and shows empirically that this improves agreement between judges. It also argues that benchmark difficulty should come from capability diversity rather than case volume, and provides a diagnostic map of which capabilities (infographic editing, human attribute transfer, minimality) remain unsolved — information a single leaderboard number would hide.

Real-world applications:

• E-commerce and product photography, where a product from one image must be placed into a scene or model from another (entity fusion).
• Virtual try-on and personal styling, mapping outfit, hairstyle, or pose from references onto a person (the attribute transfer cases the paper finds hardest).
• Design and marketing asset production, editing logos, artistic text, fonts, and materials in infographics.
• Media and character consistency workflows, keeping an identity or style fixed across many generated scenes.
• Reasoning-driven editing tools where the user's instruction is underspecified and the intent must be inferred from references.

Industry relevance. The benchmark directly compares commercial systems (GPT-Image-2, Nano Banana Pro and Nano Banana 2, Seedream 4.0 and 5.0-Lite, GPT-Image-1.5) against open-source ones (FLUX.2, Qwen-Image-Edit, BAGEL, DreamOmni2, OmniGen2) under one protocol, which gives teams a shared yardstick for procurement and for targeting the specific dimensions where open models trail. The Minimality dimension is particularly industry-relevant because unintended edits are a common blocker for production deployment.

Future Directions

• Scale up reference counts. Except for Entity Fusion, Attribute Transfer, and Analogical Reasoning, most subsets use only two reference images by design, focusing on atomic capabilities under controlled settings. Extending to more references across all families is a natural next step.

• Reduce reliance on manual filtering. The authors state that current multimodal models still err when verifying subtle visual differences, distinguishing source-specific details across references, or judging fine-grained attributes, which limits pipeline scalability. Better automated validation would allow larger and harder reasoning and infographic subsets, which the paper flags as having limited coverage compared with entity fusion and attribute transfer.

• Close the open-source gap on Consistency, Composition, and Minimality. Open-source models can produce plausible images but frequently fail the precise multi-reference constraints; these three dimensions are where improvement is most needed.

• Move beyond deterministic yes/no questions. The paper acknowledges that artistic preference, creativity, and cultural nuance remain difficult to evaluate with checklists alone, raising the question of how to combine structured checklists with judgment-based assessment.

Target Audience

Researchers and engineers working on image generation and editing, especially those building or evaluating multi-image conditioning systems; benchmark and evaluation researchers interested in VLM-as-judge reliability and checklist protocols; and practitioners in design, e-commerce, and content production who need to choose between proprietary and open-source multi-reference models. Graduate students entering multimodal generation will find the task taxonomy and failure-mode analysis a useful orientation, though some background in diffusion models and VLM evaluation is assumed.

Authors’ abstract

Image generation models have rapidly advanced, with many models now supporting generation conditioned on multiple reference images and textual instructions. Such models enable a wide range of applications, including entity fusion and attribute transfer across images. However, existing benchmarks for multi-reference image generation rely on limited evaluation dimensions and fail to capture the diverse challenges of the task. To address this limitation, we propose MDMR-Bench, a Multi-Dimensional benchmark for Multi-Reference image generation. MDMR-Bench is organized around representative capability settings and consists of multiple evaluation subsets spanning diverse dimensions of multi-reference image generation, including (1) common entity fusion and attribute transfer, (2) multi-element extraction from a single image, (3) infographic manipulation, and (4) visual reasoning-intensive generation. This design enables systematic analysis of model capabilities and failure modes across different challenge dimensions. Furthermore, to achieve more reliable and interpretable evaluation, we introduce a checklist-based evaluation protocol that decomposes generation quality into fine-grained evaluation aspects. Instead of relying solely on holistic scoring, our framework uses structured checklists composed of both global-level and atomic-level queries, enabling more diagnostic assessment of model performance. By benchmarking both open-source and proprietary models, we demonstrate that MDMR-Bench provides fine-grained insights into model performance across different dimensions. The dataset and code are publicly available.