IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation

AI summary

Overview

Research area: Computer Vision / multimodal generative modeling — specifically, unified multimodal models (UMMs) that generate text and images, and the evaluation of "interleaved" text-image generation.

Technical level: Advanced. The paper assumes familiarity with unified multimodal models, supervised fine-tuning, reinforcement learning for diffusion/image generation (Flow-GRPO), vision-language model judges, and image quality assessment models.

Scope (one sentence): The paper introduces IntReal — a 100,000-sample real-world interleaved text-image dataset (538,452 images) plus a seven-metric, expert-model-based evaluation suite — and uses them to benchmark four UMM families and to test post-training strategies for interleaved generation.

What This Paper Is About

Unified multimodal models can generate text and images, but generating them interleaved (like a Wikipedia article or an illustrated recipe, where text and pictures alternate and depend on each other) is treated as a side effect of model unification rather than a task in its own right. The authors argue that progress is blocked by two gaps: no well-defined task formulation with generation-oriented training data built around real-world scenarios, and no fine-grained evaluation that can diagnose which capability is failing. Their goal is to supply both a taxonomy-guided dataset and a coarse-to-fine benchmark, then use them to study how training affects interleaved generation.

Key Contributions

1. A real-world dataset (IntReal-100K). 100,000 high-quality interleaved samples containing 538,452 images and 146.6M text tokens, curated from 7 real-world sources and organized under a four-category taxonomy (Planning, Knowledge, Tutorial, Story), with both understanding-oriented and generation-oriented filtering.
2. A fine-grained, data-driven benchmark (IntReal Evaluation Suite). A 600-sample evaluation set (150 per category) evaluated in two settings — holistic full-sequence generation and next-image generation conditioned on interleaved context — across seven metrics, using vision-language models plus specialized expert models (e.g., image quality assessment) rather than a single holistic VLM judge.
3. Validation of the evaluation suite. Three studies report low inter-metric redundancy (maximum pairwise absolute Spearman's ρ ≤ 0.42), 83.3% agreement with human expert judgments, and cross-evaluator consistency (Spearman's ρ > 0.9 when swapping the judging VLM).
4. A training finding about modality-specific optimization. Post-training experiments show that training only the image-generation side (via Flow-GRPO with the authors' metrics as rewards) improves overall interleaved generation, suggesting interleaved generation does not require joint text-image training at every stage.

Main Findings

• No model dominates across dimensions. Among open-source models, Emu3.5 scores highest on Text Logicality (75.17), while U1-8B-MoT is far stronger on Quantity Compliance (95.33) and Context Fidelity (87.29). Emu3.5's mean score is 58.95 versus 71.88 for U1-8B-MoT.
• The closed-source system is not uniformly better. NanoBanana2 leads on Text Richness (90.80), Text Logicality (97.49) and Image Aesthetic (82.32), but falls below the best open-source result on Image Diversity (−10.45% relative) and Context Fidelity (−12.13% relative). Its mean is 84.84.
• The open-source gap is concentrated in text. The largest open-source versus closed-source disparity lies in the text modality, whereas the image-modality gap is substantially smaller — consistent with the authors' framing that language is more parameter-hungry while vision is more data-hungry.
• Different categories stress different capabilities. Planning and Knowledge are more demanding on Text Richness and Text Logicality; Tutorial is hardest under fine-grained evaluation (lower Context Fidelity and Subject Consistency); Story, with longer sequences, struggles most on Quantity Compliance and Image Diversity.
• Supervised fine-tuning on IntReal-100K helps text but hurts some image metrics. For U1-8B-MoT, SFT raises Text Richness by 29.04%, Text Logicality by 6.68%, and Subject Consistency by 6.53%, but drops Image Aesthetic by 5.65% and Context Fidelity by 4.89% — a text/image trade-off.
• RL with the proposed metrics as rewards recovers the degradation. Using Flow-GRPO on 10K rubric-annotated samples with IQA, DINO and rubric-based rewards, U1-8B-MoT with SFT+RL improves Image Diversity by 5.60% and Subject Consistency by 5.89% over the base model, while Image Aesthetic and Context Fidelity degrade only 1.34% and 1.97%. Its mean of 73.81 is the highest among the open-source models evaluated.
• Bagel shows the same pattern. RL after SFT improves its Image Aesthetic (81.94 vs 76.82) and Context Fidelity (76.04 vs 74.19) while its already-high Image Diversity stays essentially stable (91.35 → 90.31).
• The metrics are complementary, human-aligned, and evaluator-robust. Maximum pairwise absolute Spearman correlation is ≤ 0.42; automatic evaluation matches expert judgments on 83.3% of 750 expert-annotated comparisons; rankings correlate above 0.9 between Gemma4 and Qwen3.6-35B-A3B judges.

Methodology in Plain English

Building the data. The team first looked at naturally interleaved content on the web and in video and sorted it into four functional patterns: Planning (itineraries where text gives constraints and images show the places), Knowledge (encyclopedia articles pairing explanation with illustration), Tutorial (recipes where steps are paired with intermediate states), and Story (narratives with character and scene illustrations). They collected a million-scale raw pool from seven sources matching these patterns.

Filtering. They then filtered aggressively in four steps: metadata checks (aspect ratios between 1:2 and 2:1, no GIF/WebP, minimum shorter-side resolution of 512 pixels for the four web sources and 384 pixels for the three animated story sources, to accommodate older low-resolution content); specialist visual models (a RALI image-quality model requiring each image above 3.0 and sample average above 3.5; fine-tuned SigLIP2 classifiers for watermarks and faces, trained on 5,000 annotated Xiachufang samples with F1 of 0.82; DINOv3 to drop near-duplicate adjacent images with similarity above 0.85); a vision-language model (Qwen3.5-397B-A17B) to caption images and polish surrounding text while preserving original facts; and an LLM pass to write prompts and reasoning traces so each sample becomes trainable supervision. Only about 10% of collected samples survived, since one bad segment removes a whole sequence. Category-balanced sampling yields IntReal-100K.

Building the benchmark. From those 100K samples they semi-automatically selected 600 (150 per category) using automated scores for text quality, image quality, and text-image alignment (a "self-verification" step where the model writes visual rubrics from preceding context without seeing the target image, then checks the image against them), followed by manual removal and stratified sampling. They kept the set compact because their protocol already requires generating over 6,000 interleaved images per model.

Two evaluation modes and seven metrics. Holistic evaluation scores a complete generated sequence on Text Richness, Text Logicality, Image Aesthetic, Image Diversity (DINO similarity between adjacent generated images to catch visual repetition), and Quantity Compliance (matching the real sample's image count). Fine-grained evaluation fixes the real preceding context and scores only the next image on Context Fidelity (VLM checking against the curated rubrics) and Subject Consistency (DINO similarity to the reference image, computed only for Tutorial and Story since Planning and Knowledge don't require consistent subjects). All scores are scaled to [0, 100].

Experiments. Four UMM families were benchmarked: Bagel (fine-tuning released; base checkpoint does not natively do interleaved generation, so only post-SFT results are reported), Emu3.5 (native interleaved inference, no fine-tuning released), SenseNova-U1 in 8B-MoT and A3B-MoT variants (both fine-tuning and interleaved inference; the 8B-MoT interleaved checkpoint was used, and fine-tuned results reported only for 8B-MoT), and the closed-source NanoBanana2 (Gemini3.1-Flash-Image). Evaluation used Gemma4 as judge. Post-training used each model's official SFT framework and Flow-Factory for RL, with training data drawn from the 99.4K samples left after removing the 600 evaluation samples.

Why This Matters

Impact on research. The paper reframes interleaved generation as a first-class task with its own data curation principles and diagnostic metrics, rather than a byproduct of unifying modalities. The evidence that unimodal (image-only) optimization transfers to interleaved generation is a concrete challenge to the assumption that joint text-image training is always required, and it opens an optimization space beyond joint SFT. The reported failure modes — text gains trading against image aesthetic and context fidelity — give the field a vocabulary and measurement for a trade-off that was previously anecdotal.

Real-world applications:

• Illustrated how-to and recipe content, where steps must be paired with correct intermediate visual states.
• Travel and itinerary planning documents that combine constraints in text with maps and photographs.
• Educational and encyclopedia material that grounds abstract explanations in images.
• Illustrated storytelling and children's content, where character consistency across a sequence is essential.

Industry relevance. Teams building unified multimodal products now have a way to diagnose which dimension regresses after a training run instead of relying on a single holistic score, and a demonstration that RL with automated metric rewards can recover image quality lost during SFT. The finding that open-source models lag mainly on text suggests where compute should go when productizing interleaved generation.

Future Directions

• Mixed-data post-training. The authors deliberately avoided mixing unimodal generation data with interleaved data to keep the experimental setting clean, and explicitly leave mixed-data post-training to future work — a natural next test of whether the observed trade-offs can be removed rather than traded.
• Synthetic data augmentation. Given the low retention rate of real-world curation (about 10%), the authors flag synthetic augmentation as a way to scale beyond what aggressive filtering of web and video content can supply.
• Scaling model capacity. The asymmetry between a large text gap and a small image gap versus the closed-source system implies that scaling open-source UMM capacity may be the highest-leverage step — an open question the paper raises but does not test.
• Broadening the metric and evaluator set. Metrics like Subject Consistency are only computed for two of four categories (Tutorial and Story), and category-conditional metrics for Planning and Knowledge are left unaddressed; extending rubric-based rewards and expert evaluators to more capability dimensions is an open direction.

Target Audience

Researchers and engineers working on unified multimodal models, interleaved text-image generation, and multimodal data curation; benchmark designers who need fine-grained diagnostic evaluation rather than holistic VLM scoring; and practitioners planning post-training (SFT or RL) for generative multimodal systems. Readers new to the area will find the four-category taxonomy and the two-setting evaluation design conceptually useful, but the training and evaluation details assume intermediate-to-advanced background.

Authors’ abstract

Humans routinely consume multimodal content, where textual and visual information is naturally interleaved. Although Unified Multimodal Models (UMMs) can jointly generate text and images, multimodal interleaved generation remains largely underexplored. Progress in this area is hindered by two key limitations: (1) the lack of well-defined interleaved generation tasks and corresponding training data, and (2) the absence of fine-grained evaluation protocols that can reliably diagnose model performance across different capability dimensions. To this end, we present IntReal, comprising two complementary components: IntReal-100K, a real-world dataset of multimodal interleaved samples spanning four major categories, and the IntReal Evaluation Suite, which evaluates model performance across seven complementary dimensions using specialized expert models, enabling coarse-to-fine diagnosis. Extensive experiments validate our evaluation suite through low inter-metric redundancy (max pairwise absolute Spearman's $ρ\leq 0.42$ ), high agreement with human judgments (83.3% accuracy), and strong consistency across different expert evaluators (Spearman's $ρ> 0.9$ ), supporting the complementarity and reliability of our metrics. By benchmarking four families of UMMs, we show that IntReal Evaluation Suite reveals distinct strengths and weaknesses across models. Post-training experiments demonstrate the utility of IntReal-100K for improving interleaved generation and further show that training unimodal generation can also benefit interleaved generation. These findings suggest that interleaved generation can benefit from modality-specific optimization, rather than requiring joint text-image training at every stage.