Can Visual Generation Expand What Unified Multimodal Models Know?

AI summary

Overview

Research area: Multimodal machine learning — specifically unified multimodal models (UMMs) that perform both image understanding and image generation, and the question of whether generation training transfers knowledge into language.

Technical level: Intermediate. The protocol and metrics are simple to state, but the paper assumes familiarity with text-to-image diffusion/autoregressive training, vision-language judging, and multimodal alignment data.

One-sentence scope: The paper introduces Knowledge Transfer Probing (KTP), a controlled protocol that teaches fictional facts only through image-generation training and then measures whether three different UMMs can answer text-only questions about those facts.

What This Paper Is About

Unified multimodal models are trained to both generate and understand images, and image-generation supervision exposes them to visual details that captions never spell out — the shape of a badge, the number on a jersey, the arrangement of stripes on a flag. The paper asks whether learning to generate such images actually expands what the model knows in the sense of being able to answer questions about it in plain text. To test this cleanly, the authors build a synthetic dataset of invented entities and facts, supply the facts only through training images, and then probe the models with text-only questions — no images at inference.

Key Contributions

1. Knowledge Transfer Probing (KTP): A controlled protocol using fictional entity–attribute–value triplets in an invented world ("MagicWorld") so that target knowledge cannot be confused with facts the model already learned during pretraining. The protocol separates the source of knowledge (training images) from the test of knowledge (text-only QA).

2. A measurement of direct transfer from generation to language: Across seven knowledge categories and three UMMs (BAGEL, SenseNova-U1, and an in-house GLM-Omni), the authors show that generation training alone makes some knowledge textually answerable, but the transfer is partial and varies widely by model and category.

3. Two alignment strategies compared: Textual Alignment (extra text-only QA examples for a subset of entities) and Interleaved Image–Text Alignment (a single response that generates an image, interprets it, and answers). Both are applied only to a held-out "alignment entity set," and evaluation is on disjoint "test entities" whose answers appear only in images.

4. Diagnostics beyond accuracy: An image-loss ablation on GLM-Omni, a Visual CoT inference procedure, and a conditional-probability analysis of whether knowledge reproduced in images is more likely to be answerable in text.

Main Findings

• Direct transfer is real but weak and uneven. With only text-to-image supervision, average textual accessibility was 8.86% for BAGEL, 14.57% for GLM-Omni, and 20.00% for SenseNova-U1, against a random-guessing average of 15.96%. Only SenseNova-U1 clearly exceeded guessing baselines in several categories.

• The same knowledge is often far more accessible visually than textually. Average visual accessibility was 31.43% (BAGEL), 54.00% (GLM-Omni), and 61.14% (SenseNova-U1). SenseNova-U1 reached 88% visual accessibility on number while scoring 0% on text-only QA for that category.

• Generation sometimes fails too. The gap is not universal visual success: spatial was 0.0% visual accessibility for both BAGEL and SenseNova-U1, so low QA scores in those settings reflect genuine failure to learn the association in either modality.

• Textual alignment helps some models and hurts others. Mean textual accessibility moved from 8.86% to 15.43% (BAGEL) and from 20.00% to 28.00% (SenseNova-U1), but dropped from 14.57% to 12.57% for GLM-Omni. Gains concentrated on shape and size; all three models stayed at 0% on number and text for test entities even though alignment entities reached 72% and 76–96% respectively.

• Interleaved image–text alignment beats textual alignment on average for all three models. Averages were 17.14% (BAGEL), 26.86% (GLM-Omni), and 28.57% (SenseNova-U1). GLM-Omni gained most, rising from 12% to 64% on shape and from 0% to 40% on spatial, though some individual tasks declined.

• The image loss inside interleaved examples matters. Masking only the image-portion loss in GLM-Omni's interleaved data dropped average test-entity accuracy from 26.86% to 13.71%, with the decline concentrated in color, shape, and spatial.

• Visual CoT at inference adds further gains. After interleaved training, generating an image and reasoning with it raised averages to 20.57% (BAGEL), 41.14% (GLM-Omni), and 34.86% (SenseNova-U1). On number, all three models scored 0% with direct text answering; with Visual CoT they reached 4%, 36%, and 20% respectively.

• Visual correctness correlates with textual correctness, but not enough. The association gap ranged from +11.51 to +41.26 percentage points under textual alignment, and from +8.09 to +36.00 under interleaved alignment; GLM-Omni flipped from −11.28 to +8.09. Even with interleaved alignment, the probability of a correct textual answer given a correct generated image stayed below 50% for every model.

Methodology in Plain English

The authors build a small fictional universe and invent facts inside it: a character named Arvenbrock has jersey number 10, a badge has a triangular shape, a flag has a particular stripe arrangement. Because these entities and values do not exist in pretraining data, any correct answer must come from the experiment itself.

They create 50 such knowledge units for each of seven categories — color, count, number, shape, size, spatial, and text — and render each unit in roughly 25 different scenes or viewpoints, giving 8,552 training images in total (per-category counts range from 1,195 to 1,248). Captions describe the entity and the scene but never state the target attribute value; only the image does. A language model checks that captions do not leak or hint at the answer, entity names are entirely invented, and attribute values are roughly balanced.

Each model is trained under three conditions. In direct transfer, only image-generation data plus 10,000 unrelated text-only QA examples are used — the unrelated examples exist purely to teach the response format and contain none of the target entities. In textual alignment, half the entities (the alignment set) also get 10 text QA examples each. In interleaved alignment, those same entities instead get responses in which the model generates an image mid-answer, describes what it sees, and then answers. Evaluation is always on the untouched test entities, always in text only.

Two metrics are computed. Textual accessibility is the share of questions whose final answer matches the ground truth. Visual accessibility is the share of generated images that a vision-language model judges to depict the correct entity–attribute–value, sampled from 350 prompts across the full entity set, with edit-distance tolerances for rendered names and titles. The difference between the two is reported as the "accessibility gap," and a separate analysis conditions textual accuracy on whether the corresponding image was generated correctly.

Why This Matters

Impact on research. The paper turns a widely assumed but rarely tested claim — that generation training enriches understanding — into a measurable quantity, and it separates "the model learned the fact" from "the model can say the fact." The finding that interleaved supervision, including its image loss, outperforms text-only alignment gives the field a concrete data-design lever rather than an architectural one.

Real-world applications:

• Training data curation for multimodal assistants, where much visual detail in product photos, diagrams, or scanned documents is never written down in captions.
• Industrial inspection and quality control, where a model could learn part geometry from images of assemblies and later answer textual queries about them.
• Cultural heritage and archival digitization, where photographs of artifacts carry structure and ornamentation that catalog captions omit.
• Accessibility tooling that converts visually acquired information into spoken or written descriptions for users who cannot see the source images.

Industry relevance. Building UMMs is expensive, and the paper suggests that the marginal value of an image-generation objective depends heavily on how that data is organized during training. The model-to-model differences (GLM-Omni gaining most from interleaved data; SenseNova-U1 strongest in direct transfer) indicate that data recipe and architecture must be tuned together.

Future Directions

• Move from synthetic to natural visual data. The authors explicitly call for extending the study beyond the constructed MagicWorld dataset, which uses invented entities and deliberately out-of-distribution facts.
• Isolate architectural effects. The three models differ in visual representation (semantic encoder plus VAE for BAGEL, pixel-level NEO-unify for SenseNova-U1, discrete tokens based on X-Omni for GLM-Omni), but the paper does not disentangle which design choices drive the differing transfer rates.
• Explain the persistent gap. Correct visual reproduction still yields below-50% textual accuracy even under interleaved alignment, and number and text remain near zero without Visual CoT; the mechanism behind this residual failure is unresolved.
• Refine alignment data and inference strategies. The image-loss ablation and Visual CoT results suggest headroom in both how interleaved examples are constructed and how models are queried at test time.

Target Audience

Researchers and engineers working on unified multimodal models, multimodal training-data design, or cross-modal knowledge transfer. It is also useful for practitioners deciding whether to invest in image-generation objectives or interleaved training data for a multimodal product, and for graduate students looking for a clean, reproducible experimental protocol — the dataset and evaluation code are released at the project's GitHub repository.

Authors’ abstract

Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text. Can learning to generate images expand what pretrained models know? We introduce Knowledge Transfer Probing (KTP), a controlled protocol that supplies new knowledge through image-generation training and tests its accessibility through text-only question answering. Across seven knowledge categories and three UMMs, we observe direct but uneven transfer from visual supervision to language. Motivated by these observations, we successively introduce Textual Alignment and Interleaved Image--Text Alignment, which connect generation and answering through shared knowledge and interleaved responses, respectively. Guidance on selected entities can improve knowledge access on other entities without QA supervision, with additional gains from interleaved alignment. Further analysis shows that knowledge correctly reproduced in images is often more accessible through text, although substantial gaps remain between the two forms of access. These findings support visual generation as a source of knowledge beyond textual supervision, and motivate joint investigation of training data design and model architecture to make this knowledge more broadly accessible.