Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer

AI summary

Overview

Research area: Computer vision and multimodal machine learning — specifically unified visual tokenization and autoregressive vision-language modeling.

Technical level: Intermediate. Readers benefit from familiarity with autoregressive sequence models, visual tokenizers (VAE, CLIP, VQ-based tokenizers), and vision-language benchmarks, though the core argument is stated in plain terms.

Scope: This paper introduces a continuous visual tokenizer (MingTok) and a unified autoregressive multimodal model (Ming-UniVision) that handles image understanding, text-to-image generation, and multi-round in-context image editing within a single token space.

What This Paper Is About

Unifying image understanding and image generation in one autoregressive model is difficult because the two tasks want opposite things from visual representations: understanding favors high-dimensional semantic features, while generation favors compact low-level latent codes. Existing systems either keep two separate tokenizers (adding architectural complexity) or force both tasks into one discrete token space, where quantization error hurts semantic expressiveness. The paper's goal is a single continuous tokenizer that serves both tasks well, and a unified model built on top of it.

Key Contributions

1. MingTok, a continuous visual tokenizer. A three-stage architecture — a low-level encoder, a semantic decoder, and a pixel decoder — that avoids vector quantization entirely while producing both compact latents for generation and rich semantic features for understanding.

2. Ming-UniVision, a unified autoregressive framework. Diverse vision-language tasks are cast as next-token prediction over a shared continuous space, removing the need for task-specific visual representations and the dual-encoder designs used by prior unified models.

3. Efficient multi-round in-context editing. Because semantic features persist in latent space, edits can be chained without pixel-space re-encoding. Ming-UniVision requires up to 66% fewer input visual tokens than hybrid AR-diffusion models and 50% fewer than existing unified autoregressive models.

4. Released artifacts. Inference code and model weights are released (GitHub inclusionAI/Ming-UniVision, Hugging Face and ModelScope inclusionAI).

Main Findings

• Tokenizer architecture splits the job. MingTok uses a low-level encoder for compact latents, a semantic decoder with causal attention to expand them into high-dimensional semantic features, and a pixel decoder to reconstruct images. The semantic decoder's causal attention is what enables per-token autoregressive generation.

• Reconstruction quality. MingTok encodes 512x512 images into 256 continuous latent tokens at a 32x compression ratio, reaching rFID 0.54, PSNR 30.77 dB, SSIM 0.62, and LPIPS 0.14 on the ImageNet validation set. After the semantic decoder is jointly trained during Ming-UniVision pre-training, rFID drops to 0.38, PSNR rises to 31.09, SSIM to 0.64, and LPIPS to 0.12.

• Understanding results. Using Ling-lite with 2.8 billion activated parameters, Ming-UniVision-16B-A3B scores MMB 78.5, MMStar 63.7, MMMU 40.3, MathVista 66.6, HallusionBench 47.8, AI2D 82.8, MM-Vet 64.2, OCRBench 724, and MME 2023. The paper reports competitive results on MMStar, HallusionBench, AI2D, and MM-Vet but an acknowledged gap on OCRBench and MMMU, attributed to the compressed latent space and the causal semantic decoder.

• Generation results. On GenEval, Ming-UniVision reaches 1.00 Single Obj., 0.93 Two Obj., 0.59 Counting, 0.93 Colors, 0.92 Position, 0.70 Color Attribute, and 0.85 Overall, plus 82.12 on DPG-Bench. The paper states this is state-of-the-art on overall GenEval, with the largest leads on Position, Colors, and Color Attribute subtasks.

• Editing results. On GEdit-Bench (EN), scored 0–10 by GPT-4.1, the model gets G_SC 6.04 / G_PQ 6.86 / G_O 5.54 for single-round editing and 6.60 / 6.25 / 5.78 for multi-round. The paper notes the overall score lags prior work, attributing the gap to the absence of large-scale multimodal sequence pre-training and the high per-token detail density of the continuous tokenizer. It also reports that a base model without the mixed-resolution strategy is used, because consistent resolution across understanding and generation proved critical for editing.

• Unified representation helps both tasks. An ablation with a smaller dense model (Qwen-2.5-3B) and reduced pre-training data compares four tokenizer combinations. Using MingTok as both understanding and generation tokenizer gives the best understanding average (53.57) and the best GenEval (0.4654), versus CLIP+VAE (51.92 / 0.3591), CLIP+MingTok (52.54 / 0.4600), and MingTok+VAE (52.03 / 0.3950).

• Speed claim. Figure 1 reports over 3.5x acceleration for text-to-image generation compared with SD-VAE.

Methodology in Plain English

The tokenizer is the centerpiece. Instead of turning an image into a grid of discrete vocabulary tokens — which introduces rounding error — MingTok keeps everything continuous and splits the work across three transformer-based stages. The low-level encoder squashes an image into a small number of compact latent vectors (channel dimension 16 or 32), keeping tokens few for fast autoregressive generation. The semantic decoder then expands those compact vectors into high-dimensional features that carry text-aligned meaning, using causal attention so each token can be produced one at a time. The pixel decoder turns those semantic features back into pixels, using a pixel unshuffle layer to recover fine texture and edge detail.

Training uses masked image modeling with three objectives at once. Masked tokens go into the low-level encoder and are asked to predict features from a pre-trained vision foundation model such as DINOv2. The semantic decoder performs the same masked feature prediction against features from a text-aligned backbone such as CLIP. The pixel decoder is trained on both masked and unmasked inputs so it stays robust when latents are missing or noisy — mimicking the sequential generation process.

The multimodal model attaches MingTok to a language model. For understanding, a real image is encoded and all semantic tokens are computed in parallel and fed to the LLM. For generation, the LLM's vision head produces compact latents one token at a time; each latent is immediately expanded by the semantic decoder and fed back as context for the next prediction. Text is predicted with a standard language model head. The vision head replaces a diffusion denoising head with a rectified flow prediction objective, and uses a SwiGLU-based feed-forward network instead of a standard MLP.

Training proceeds in phases. Pre-training stage 1 warms up the MLP bridge and rectified flow head with MingTok and the LLM frozen, using roughly 30% understanding and 70% autoregressive generation data. Pre-training stage 2 unlocks the LLM and only the semantic decoder of MingTok, with mixed resolution (1024x1024 for understanding, 512x512 for generation) and a data mix of about 25% understanding, 70% text-to-image, and 5% general NLP. Supervised fine-tuning stage 1 freezes MingTok and trains the rest with about 30% understanding, 10% NLP, and 60% text-to-image. SFT stage 2 shifts the mix to 15% understanding, 5% NLP, 35% standard text-to-image, and 55% single- or multi-round editing.

Why This Matters

Impact on research. The paper argues that the standard assumption — that understanding and generation need fundamentally different visual representations — may be avoidable. If a single continuous tokenizer can serve both, it removes a major source of architectural complexity in unified multimodal systems and shifts the field's attention from discrete-token design toward continuous latent design. It also frames continuous tokenization as a way to eliminate quantization error, which prior work identified as a cause of degraded understanding performance.

Real-world applications:

• Interactive photo editing, where a user chains instructions such as upsampling then colorizing, or segmenting then editing, without waiting for repeated decoding and re-encoding.
• Conversational image creation tools where a user iterates on an image across several rounds while the model retains the full visual context.
• Assistive or low-latency deployment scenarios, where the reported 66% fewer input tokens and 50% fewer tokens versus comparable unified architectures translate into lower memory overhead and faster iteration.
• Multimodal assistants that need to reason about an image and then produce or modify one within the same session.

Industry relevance. The model is released with weights and inference code, and the authors frame unified latent interaction as a step toward interactive, human-in-the-loop vision systems where generation, editing, and understanding are interwoven steps rather than separate pipelines. The efficiency claims matter for deployment cost, since fewer tokens per round reduces both latency and memory pressure in iterative workflows.

Future Directions

1. Detail-preserving latents and semantics. The authors explicitly leave this for future work, after observing gaps on OCRBench and MMMU where character-level and fine-grained recognition matter.

2. Closing the editing quality gap. The paper attributes the GEdit-Bench overall-score gap to the absence of large-scale multimodal sequence pre-training and to the per-token detail density of the continuous tokenizer, and states intent to address both.

3. Fine-grained detail generation. Section 4.2 notes that further improvements in fine-grained detail generation remain an avenue for future work.

4. Scalable multi-round editing. Section 5.2 (partially beyond the truncated content) discusses multi-round understanding, generation, and editing, suggesting further study of instruction chains and in-context manipulation training.

Target Audience

Researchers and engineers working on unified multimodal models, visual tokenization, or autoregressive image generation; practitioners building interactive image editing and generation products who care about token efficiency and multi-round workflows; and readers tracking the debate over discrete versus continuous visual representations in vision-language models.

Authors’ abstract

Visual tokenization remains a core challenge in unifying visual understanding and generation within the autoregressive paradigm. Existing methods typically employ tokenizers in discrete latent spaces to align with the tokens from large language models, where the quantization errors can limit semantic expressiveness and degrade the capability of vision-language understanding. To address this, we introduce MingTok, a new family of visual tokenizers with a continuous latent space, for unified autoregressive generation and understanding. While understanding tasks favor discriminative high-dimensional features, generation tasks prefer compact low-level codes. Thus, to reconcile these competing demands, MingTok adopts a three-stage sequential architecture involving low-level encoding, semantic expansion, and visual reconstruction. Built on top of it, Ming-UniVision eliminates the need for task-specific visual representations, and unifies diverse vision-language tasks under a single autoregrsssive prediction paradigm. By formulating both understanding and generation as next-token prediction in a shared continuous space, it seamlessly supports multi-round, in-context tasks such as iterative understanding, generation and editing. Empirically, we find that using a unified continuous visual representation reconciles the competing requirements on the tokenizers by the understanding and generation tasks, thereby leading to state-of-the-art level performance across both domains. We hope our findings will facilitate unified visual tokenization in the continuous domain. Inference code and model weights are released to benefit community.