Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

AI summary

Overview

Research area: Computer vision and multimodal machine learning — specifically unified multimodal models (UMMs) that perform both visual understanding and image generation inside one model.

Technical level: Advanced. The paper assumes familiarity with transformers, autoregressive language modeling, flow matching, mixture-of-transformers routing, and probing techniques such as linear CKA and layer-wise feature probes.

Scope: A controlled, three-level (representation, task, system) empirical study of when visual understanding and image generation reinforce each other versus when they interfere, conducted in a "native" pixel-in, pixel-out setting without a pretrained vision encoder or a generative VAE.

Authors and affiliations: Penghao Wu, Haiwen Diao, Weichen Fan, and Ziwei Liu (S-Lab, Nanyang Technological University); Lewei Lu and Dahua Lin (SenseTime Research). Posted as arXiv:2609.01607v1 [cs.CV], 01 Sep 2026, under a CC BY 4.0 license.

What This Paper Is About

Unified multimodal models can both read images and create them, but being able to do both through one interface does not prove the two abilities actually help each other. Their training objectives might reinforce each other, might compete for the same model capacity, or might simply coexist without any real transfer. The paper sets up a deliberately controlled "native" testbed — raw image patches in, raw pixels out, no pretrained vision encoder, no image tokenizer or VAE — and asks a single question at three levels of scope: when and why do visual understanding and generation help each other, and when does unification instead create interference?

Key Contributions

1. A controlled and comprehensive study of understanding–generation interaction at the representation, task, and system levels under a native multimodal setting.
2. The finding that understanding and generation can provide useful learning signals to each other, but whether those signals become mutual gains is architecture-dependent.
3. Task-level synergy demonstrated through three case studies — geometry, SVG, and 3D spatial intelligence — showing that transfer emerges when the paired tasks rely on shared knowledge or capabilities.
4. A system-level advantage for unification: an end-to-end UMM outperforms a matched planner–executor pipeline on reasoning-intensive image editing, a task that requires tightly coupled understanding, reasoning, and generation.

Main Findings

• Generation enriches understanding features (dense routing): With all tokens routed through the same pretrained LLM decoder, joint training (Und+Gen) beats understanding-only (Und) on understanding — General 68.69 vs 67.82, OCR 72.12 vs 71.96, Vision-centric & SI 61.77 vs 60.72 — with the largest gain on the vision-centric/spatial category. Frozen-backbone probes (ImageNet classification, ADE20K semantic segmentation, NYU-Depth V2 monocular depth) and PCA feature visualizations show joint training strengthens the learned visual representations, most clearly in early and middle layers.

• But dense joint training costs generation: Against the generation-only model, Und+Gen degrades on every generation metric — GenEval2 51.03 vs 51.16, DPGBench 78.12 vs 79.11, HPSv3 5.92 vs 7.05, Aesthetic 4.94 vs 5.12. The paper hypothesizes the pretrained LLM acts as a strong semantic anchor that lets understanding dominate the shared feature space, leaving generation as little more than an auxiliary regularizer.

• Modality-decoupled MoT flips the asymmetry: When text stays in the pretrained LLM branch and all visual tokens go through a scratch-trained branch, generation improves strongly under joint training (GenEval2 63.47 vs 57.55, DPGBench 83.09 vs 82.06, HPSv3 7.97 vs 7.72, Aesthetic 5.48 vs 5.37) while understanding degrades (General 61.58 vs 64.20, OCR 62.11 vs 66.84, V-Centric & SI 57.49 vs 59.54). Here generation dominates and understanding becomes the auxiliary signal. The modality-decoupled Und model also trails its dense counterpart, especially on General and OCR.

• Understanding improves vision–language alignment for generation: Layer-wise linear CKA between pooled text features from the language branch and generative image features from the visual branch — computed on 5000 text-image pairs from the COCO validation set — is consistently higher under joint training than under generation-only training.

• A task-decoupled design avoids both failure modes: Keeping understanding visual tokens language-anchored while assigning generation visual tokens to a specialized branch yields General 69.02, OCR 72.09, V-Centric & SI 62.38, GenEval2 63.96, DPGBench 82.31, HPSv3 7.90, Aesthetic 5.50 — better than either the dense or modality-decoupled joint models on understanding and generation. Soft alignment between branches comes from shared attention to the same textual features and, for editing/interleaved samples, from the context image conditioning the generation branch. An additional 20% of the training mixture is allocated to an image-reconstruction task formulated as image editing, where 50% of the visual tokens in each context image are randomly masked.

• Task-level transfer in geometry: Joint training on 965K geometry problem-solving examples plus general math data and generation data from MathCanvas-Imagen and MathCanvas-Edit improves all four benchmarks over understanding-only: Geometry3K 65.39 vs 59.90, PGPS9K 73.40 vs 72.40, MathVerse testmini 63.15 vs 60.13, MathVista testmini 74.30 vs 71.80. A control baseline (Und + Textual-Desc) that converts the same generation data into image-description and edit-description tasks scores 63.56, 73.90, 59.11, and 73.60 — a smaller and less consistent benefit, showing the gain is not merely extra geometry data.

• Task-level transfer in SVG: Using 359K SVG code-image pairs from UniSVG, joint training beats single-direction training in both directions — Image→SVG 81.70 vs 80.64 and SVG→Image 86.52 vs 80.21, under UniSVG's weighted metric combining SSIM, LPIPS, and CLIP similarity. Two diagnostics support the interpretation: joint training scores higher on a purpose-built code-conditioned multiple-choice "blind VQA" benchmark that requires mentally executing SVG operations, and its single-step clean-image prediction recovers recognizable contours and layout earlier than the generation-only model.

• Task-level transfer in 3D spatial intelligence: Joint training improves eight of nine 3D SI understanding benchmarks and raises the average from 57.15 to 59.01 (VSI-Bench 55.49 vs 53.98, SPAR-Bench 41.13 vs 38.80, CV-Bench 78.77 vs 77.67, DA-2K 73.89 vs 70.69, MindCube-tiny 77.90 vs 75.71, ViewSpatial 58.07 vs 58.75, 3DSRBench 60.51 vs 59.84, MMSI 39.20 vs 36.40, ALL-Angles 46.11 vs 42.50). On generation it lowers Spatial-Edit Viewpoint Error (0.3766 vs 0.4062) and Framing Error (0.6297 vs 0.6564), and improves all three View Sequence Completion metrics (PSNR 19.15 vs 16.76, SSIM 0.6674 vs 0.6365, LPIPS 0.3078 vs 0.3828). ViewSpatial is the one understanding benchmark that goes down.

• System-level advantage of end-to-end unification: On reasoning-intensive image editing, under the same base model and data supervision, the end-to-end UMM beats a planner–executor pipeline on both RISEBench (Temporal 22.35 vs 18.82, Causal 27.77 vs 22.22, Spatial 16.00 vs 14.00, Logical 9.41 vs 11.76, Overall 18.88 vs 16.66) and KRIS-Bench (Factual 64.73 vs 61.40, Conceptual 72.32 vs 71.17, Procedural 66.16 vs 64.58, Overall 68.33 vs 66.48). The end-to-end model is worse only on the RISEBench Logical sub-score.

• Three headline claims: (1) understanding and generation can benefit each other, but mutual gains depend on architecture — selectively specializing conflicting visual pathways while preserving semantic interaction lets both improve; (2) shared underlying knowledge or capabilities make understanding and generation tasks mutually reinforcing; (3) end-to-end unification can outperform modular composition for tasks explicitly requiring both.

Methodology in Plain English

The authors build a deliberately minimal testbed so that any measured effect can be attributed to the interaction between understanding and generation rather than to a borrowed visual prior. Images enter as raw patches through two convolutional layers with patch size 32, and images leave as pixels — no pretrained vision encoder, no image tokenizer, no VAE. The model starts from a pretrained Qwen3-1.7B language model augmented with a stack of randomly initialized pre-buffer layers, text is modeled autoregressively, and the visual tokens of each generated image are modeled bidirectionally with flow matching. Positions are encoded with MRoPE-I to preserve 2D image structure and the language model's positional priors; attention is causal across the overall multimodal sequence but bidirectional within an image.

Against that fixed backbone they compare three routing architectures. Dense sharing sends text, clean understanding tokens, and noised generation tokens all through the same LLM layers. Modality-decoupled MoT keeps text in the pretrained branch and routes all visual tokens through a parallel scratch-initialized branch. Task-decoupled MoT keeps understanding visual tokens language-anchored and sends only the noised generation tokens to a specialized branch, with soft alignment between the two branches through shared text attention and the context image.

The comparison is controlled at each level. At the representation level, matched variants (understanding-only, generation-only, joint) are trained under each architecture with single-stage training for 210k optimization steps at a constant learning rate of 1×10⁻⁴, using understanding and generation data drawn from SenseNova-U1. Understanding is measured as category averages over General (MME, MMBench, MMStar, SEED Bench, MMMU), OCR (DocVQA, ChartQA, InfoVQA, OCRBench, AI2D), and Vision-centric & SI (PerceptionBench, P2GB, BLINK, MME-Realworld, DA2K, CV-Bench, MindCube, 3DSR, ViewSpatial) benchmarks; generation is measured with GenEval2 (atom-level score), DPGBench, HPSv3, and Aesthetic Score. To see why joint training helps, the authors freeze the backbones and run layer-wise probes (ImageNet classification, ADE20K segmentation, NYU-Depth V2 depth) plus PCA visualizations, and separately measure layer-wise linear CKA between language-branch text features and visual-branch image features.

At the task level, each of the three case studies holds 50% of its training mixture fixed to the same general-purpose data used at the representation stage (with task-related examples removed), so only the task-specific understanding and generation data vary. At the system level, all models are initialized from the same task-decoupled MoT checkpoint and the identical set of reasoning-editing examples is reformatted three ways — full end-to-end, planning-only, and execution-only — so the agentic pipeline and the unified model differ only in whether the process is learned as one model or composed from two.

Why This Matters

Impact on research. The paper reframes a question the field has largely answered by demonstration rather than by controlled experiment: UMM papers show that both capabilities can be supported at scale, but offer limited and sometimes conflicting evidence about whether joint learning actually helps either one. By isolating the effect in a setting without pretrained visual priors, the work separates two things that are usually entangled — the benefit of the interface (one model that does both) from the benefit of the learning signal (each objective improving the other). Its practical message is a design rule rather than a single architecture: specialize computation where objectives conflict, share it where knowledge is shared, and unify training and inference when the two capabilities must interact throughout a task.

Real-world applications:

• Reasoning-intensive image editing, where a user gives an implicit request such as "make this scene match that mood" — the system must read the source image, infer the concrete edit, and produce the result. The paper shows this is better learned end-to-end than split into a separate planner and executor.
• Geometric and mathematical diagram tools, such as educational systems that solve geometry problems and also generate or edit the diagrams that go with them, where the same structural knowledge underlies both directions.
• Vector graphics workflows, where a model both converts rendered icons to SVG code and renders SVG code back to images — a naturally bidirectional domain the paper tests with 359K paired examples.
• 3D spatial assistants and embodied agents, for tasks like ego-motion prediction, multi-view reconstruction, view-sequence completion, and layout-to-image generation, which sit alongside spatial VQA in the same capability family.

Industry relevance. The results give concrete guidance to teams deciding how to allocate parameters and training compute across two objectives in one model. Dense sharing of every token type is the simplest design and it measurably harms generation; a fully separate visual branch is the opposite extreme and it measurably harms understanding. The task-decoupled compromise — language-anchored understanding tokens, a specialized generation branch, and a lightweight reconstruction objective to keep the branches aligned — is the configuration the paper reports as avoiding both trade-offs. The system-level result is also directly actionable for product teams choosing between a single unified model and an agentic pipeline of two specialized models: for tightly coupled tasks under matched base model and data, the paper finds the single model wins.

Future Directions

1. Validate generality across modeling formulations. The study uses a hybrid formulation pairing autoregressive text modeling with continuous flow matching for images. Whether the same conclusions hold under fully autoregressive multimodal modeling or fully discrete denoising objectives is left to future work, though the authors expect conclusions about parameter sharing, task transfer, and system design to extend beyond this particular generative objective.
2. Search the architecture design space more broadly. The experiments mainly use one task-decoupled architecture to show that the two capabilities can mutually benefit. The paper explicitly does not claim that task-decoupled routing is the unique optimal solution and points to conditional architectures such as mixtures-of-experts with task- or token-dependent routing as a promising alternative for balancing specialization and sharing.
3. Extend task-level synergy to broader shared-knowledge settings. The paper suggests world modeling and policy/action learning as a natural extension, since understanding the current state, predicting future observations, and generating actions all depend on a common model of the physical world and its dynamics.
4. Internalize broader agentic generation processes — such as multimodal search plus generation and iterative visual refinement — as native UMM capabilities, so that increasingly complex tasks can be learned and completed end-to-end within a single unified model.

Target Audience

This paper is best suited to researchers and engineers working on unified multimodal models, multimodal representation learning, or image generation architectures who need evidence about how to structure a model that must do both. It will also interest teams making build-versus-compose decisions — whether to ship one end-to-end model or an agentic pipeline of specialized models — since the system-level comparison speaks directly to that trade-off. Readers need a working knowledge of transformer architectures, flow matching, and representation probing to follow the methodology; the conclusions themselves are stated plainly enough for a technical product or research-strategy audience. Those looking purely for a new state-of-the-art architecture or benchmark leaderboard should note that the paper's contribution is a controlled analysis of when synergy occurs, not a claim of best-in-class performance.

Authors’ abstract

While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.