YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction

AI summary

Overview

• Research area: Computer Vision / multimodal foundation models — specifically datasets and benchmarks for unified multimodal models (UMMs) that handle image understanding, generation, and editing in one system.
• Technical level: Intermediate. The paper is a dataset-and-benchmark contribution; it is readable with a basic grasp of multimodal models and image editing, though the evaluation protocol (VLM-as-a-Judge, multiple metrics) takes some care.
• Scope: The paper introduces YANAMI, a 402,639-dialogue training corpus (YANAMI-400K) plus a 400-dialogue benchmark (YANAMI-Bench), designed to force models to track, maintain, update, and visually realize a dialogue state across multiple turns.

What This Paper Is About

Existing multimodal datasets and benchmarks are dominated by single-turn tasks, and even recent multi-turn datasets often contain turns that can be solved independently without carrying information forward. The authors argue this leaves models without supervision or evaluation for "multimodal stateful interaction" — the ability to understand, maintain, update, and faithfully render an evolving dialogue state across turns. YANAMI is built to supply both the training data and the evaluation protocol for that capability.

Key Contributions

1. YANAMI-400K training corpus. A large-scale multimodal dialogue dataset for stateful interaction spanning understanding, generation, and editing — 400K multi-turn dialogues (402,639 in Table 1), more than 2M images (2,047,236), and 2.6M dialogue turns (2,643,291), with an average of 6.56 turns and a maximum of 43 turns per dialogue.
2. A six-capability design for statefulness. Every dialogue is constructed to exercise Reference Resolution, Context Retrieval, Consistency Maintenance, Spatial and Physical Consistency, Constraint Keeping, and Correction-Based State Updating, instantiated across four scenarios: Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning.
3. YANAMI-Bench. A 400-dialogue benchmark (100 per scenario) that jointly scores Dialogue State Understanding (DSU) and Visual Realization (VR), explicitly characterizing the gap between a model understanding the state and actually rendering it.
4. Empirical validation via fine-tuning. Fine-tuning BAGEL on YANAMI-400K improves image editing, image generation, and multimodal stateful interaction by 10.3%, 2.0%, and 13.23% respectively (per the abstract), while preserving general visual understanding.

Main Findings

• Stateful interaction is still hard for current models. Across 15 evaluated models (2 closed-source, 4 open-source image-editing, 9 unified multimodal models), the best average scores on YANAMI-Bench were 67.41 for UMMs (Emu3.5) and 67.30 for open-source image-editing models (FLUX.2 [dev]).
• Closed-source models lead clearly. GPT-Image-2 scored 83.05 overall and Nano Banana 2 scored 80.93, versus the 67.41/67.30 open-source bests, indicating a noticeable open–closed gap.
• Visual Reasoning is the weakest scenario. Most evaluated models scored lowest there; for example GPT-Image-2 scored 73.46 on Reasoning versus 86.78 on Fusion.
• Dialogue-state understanding itself is weak. The authors report that average DSU scores of most UMMs remain below 82. Show-o2 averaged 67.94, BAGEL 81.44, and BAGEL fine-tuned on YANAMI-400K 95.69.
• An understanding–realization gap exists. Even when a model correctly understands a currently valid constraint or a corrected state, it does not always realize that information in the generated image (Fig. 6a).
• Performance degrades as dialogue progresses. Comparing the first and last image-generation turns under the GT-History setting, performance dropped by 2% (BAGEL), 6% (FLUX.2), and 6% (InternVL-U).
• Self-generated history compounds errors. Under a Self-History setting, performance dropped 5.16% (BAGEL) and 3.77% (Qwen-Edit) relative to GT-History.
• Fine-tuning on YANAMI-400K yields broad gains. BAGEL improved on YANAMI-Bench from 65.02 to 78.25 overall; on GEdit-Bench-EN-full its average rose from 6.52 to 7.19 (a 10.28% improvement as stated in the text); on GenEval overall it rose from 0.73 to 0.75.
• General visual understanding is largely preserved. After fine-tuning, MMBench rose from 85.0 to 89.1 (+4.1%), POPE rose from 86.5 to 87.4 (+0.9%), and MME-P fell slightly from 1647.9 to 1616.9.
• The VLM judge is reliable. In a human study with four expert annotators over 100 sampled instances from each of three models (BAGEL+YANAMI, Emu3.5, Nano Banana 2), Pearson correlations between Qwen-3.8-Flash-Next scores and human ratings consistently exceeded 0.80, and agreement with GPT-5.6-sol was also high.

Methodology in Plain English

The authors did not collect new photographs. They took raw images from four existing public datasets and ran them through a four-stage pipeline: data preparation, dialogue construction, automatic verification, and human verification.

• Filtering. Large multimodal models perform a deliberately permissive quality screen — an image is rejected only for severe corruption, generation failure, invalid content, or extremely poor visibility. If more than half of an image group is rejected, the whole group is discarded; otherwise only the rejected images are repaired and re-filtered.
• Dialogue construction. Scenario-specific image pools are combined with scenario- and capability-oriented prompts, and models including GPT-5.6 Sol, Qwen3.7-Plus, Gemini 3.1 Pro, GPT-Image-2, and Nano Banana 2 generate the multi-turn dialogues and their images.
• Verification. Dialogues are automatically checked for logical coherence and image–text consistency, with failures iteratively corrected and regenerated; the benchmark additionally receives exhaustive human annotation by graduate-level STEM annotators.
• Leakage control. The benchmark is strictly isolated from training, and the authors compare SHA-256 hashes of all images across YANAMI-Bench and YANAMI-400K to confirm no exact duplicates.
• Evaluation. A VLM-as-a-Judge framework scores DSU through capability-oriented visual question answering on four dimensions, and VR through Visual Quality and State Consistency using fine-grained atomic criteria. To separate a model's ability to use context from error propagation, the main setting is GT-History, where ground-truth prior images are supplied; three image-placement configurations (yes-first, yes-front, concatenate) accommodate different model input formats.

Why This Matters

Impact on research. The paper reframes multi-turn multimodal evaluation: instead of a sequence of independent edits, it demands that state be established, retained, invalidated, and revised. The explicit separation of Dialogue State Understanding from Visual Realization gives the field a diagnostic that a single aggregate score would hide — a model can understand a constraint and still fail to draw it. The dataset, benchmark, and prompt templates are slated for release on Hugging Face and GitHub.

Real-world applications:

• Iterative creative tools. Real image creation is a back-and-forth of alternatives, corrections, and accumulated constraints rather than one-shot prompts; this data targets exactly that workflow.
• Design and marketing asset production. Multi-image fusion and progressive editing with persistent style and content constraints map onto assembling product or campaign visuals from multiple source images.
• Simulation and embodied planning. The World Modeling scenario — predicting and rendering scene states under specified viewpoint changes or actions — is directly relevant to training agents that reason about visual outcomes.
• Step-by-step visual reasoning assistants. The Visual Reasoning scenario, which generates visual outcomes for hypotheses, supports tutoring, analysis, and scientific diagram workflows.

Industry relevance. The measured gap between closed-source (83.05, 80.93) and open-source (67.41, 67.30) systems marks a concrete target for open multimodal model development, and the demonstrated 13.23% gain from fine-tuning a 7B model (BAGEL) on YANAMI-400K shows that stateful interaction can be improved with data rather than only scale.

Future Directions

• Closing the understanding–realization gap. The paper measures the gap but does not explain its mechanism; work is needed on architectures or training objectives that force understood constraints into the generated pixels.
• Reducing error accumulation in realistic rollouts. The 5.16% and 3.77% Self-History drops show that models trained on clean history degrade when conditioning on their own outputs; self-history training or correction mechanisms are an obvious next step.
• Longer-horizon interaction. Dialogues reach a maximum of 43 turns and performance declines across generation turns; extending supervision and evaluation to substantially longer horizons remains open.
• Coverage beyond the four scenarios. Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning are the stated scope; whether the six-capability framing transfers to video, audio, or embodied interaction is untested here.

Target Audience

Researchers and engineers working on unified multimodal models, multimodal dialogue systems, and image generation/editing; dataset and benchmark builders interested in how statefulness is operationalized and verified; and practitioners building iterative creative or simulation tools who need to know where current models break down over multi-turn use. Readers wanting a quick orientation need only the abstract and findings; those reproducing or extending the work will need the appendices on data construction, evaluation templates, and scoring.

Authors’ abstract

Unified Multimodal Models (UMMs) integrate visual understanding, generation, and editing within a single unified framework. However, existing unified multimodal datasets and benchmarks remain dominated by single-turn tasks. Even in recent multi-turn datasets, many turns can be solved independently without tracking information across turns. Consequently, they provide limited supervision and evaluation for multimodal stateful interaction, which requires models to understand, maintain, update, and faithfully realize evolving dialogue states across turns. To address this gap, we introduce YANAMI, the first large-scale multimodal dialogue dataset and benchmark for stateful interaction, comprising YANAMI-400K and YANAMI-Bench. YANAMI-400K contains 400K multi-turn multimodal dialogues, with more than 2M images and 2.6M dialogue turns. The dialogues span four representative scenarios: Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning, and require models to continuously track, preserve, and revise evolving multimodal dialogue states across turns. YANAMI-Bench comprises 400 multi-turn dialogues for systematically evaluating multimodal stateful interaction. It jointly evaluates Dialogue State Understanding and Visual Realization, characterizing the gap between models' ability to understand evolving dialogue states and their ability to faithfully realize them in visual outputs. Experiments show that fine-tuning BAGEL on YANAMI-400K improves image editing, generation, and multimodal stateful interaction by 10.3%, 2.0%, and 13.23%, respectively, while preserving its general visual understanding capabilities. Evaluations on YANAMI-Bench further reveal that existing UMMs still struggle to maintain, update, and visually realize evolving multimodal dialogue states.