Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

AI summary

Overview

Research area: Computer vision and multimodal machine learning — specifically image tokenization and unified autoregressive (AR) models that handle both text and images.

Technical level: Intermediate. The core idea is accessible, but full appreciation requires some familiarity with vector-quantized image tokenizers, autoregressive language modeling, and loss-scaling analysis.

Scope: A controlled empirical study that treats image tokenizers as defining a "visual language," using task-specific validation losses during joint multimodal training to reveal how tokenizer design choices affect generation, understanding, and even text modeling.

What This Paper Is About

Image tokenizers turn pixels into discrete symbols that a language model can predict alongside text, effectively defining the visual vocabulary a unified multimodal model must learn. Prior work evaluates tokenizers in isolation—via reconstruction metrics like rFID or through generation-only or understanding-only pipelines—which cannot show how visual tokens behave when trained jointly with text. This paper builds a controlled pure-AR testbed where image and text tokens are modeled under a single objective, then uses task-specific validation losses (text, image, T2I, I2T) as a lens to study tokenizer effects on joint multimodal learning.

Key Contributions

1. A controlled pure-AR testbed for tokenizer study. The authors extend pretrained Qwen3 language models (0.6B, 1.7B, 4B, with an 8B recipe validation) by expanding the vocabulary with discrete image tokens, then continually pretrain on mixed-modal data and finetune on instruction data—giving a reproducible setting where tokenizer effects can be isolated.

2. A task-specific loss framework for unified multimodal training. Instead of one aggregate loss, they track four separate validation losses (text, unconditional image, T2I, I2T) and characterize how each scales with data and model size, showing that averaging obscures meaningful differences.

3. Evidence on how losses relate to downstream benchmarks. They show that the loss–performance relationship depends on the predicted token space: T2I loss shifts across tokenizers while I2T loss (computed over a shared text vocabulary) is more consistent, and both relate to post-SFT generation and general VQA performance.

4. Case studies on three tokenizer design axes. They revisit the discriminator (PatchGAN vs. DINO), semantic supervision, and vocabulary size, showing that reconstruction fidelity can diverge from multimodal learnability and that tokenizer choice can affect text modeling itself.

Main Findings

• Losses must be interpreted per task. All four task losses follow roughly power-law scaling, but they exhibit qualitatively different behavior and rank tokenizers differently across tasks. UniTok has the highest text loss yet the lowest T2I loss, while GigaTok shows the reverse pattern. No single tokenizer ranking holds across tasks, so an averaged loss is too coarse.

• The loss–performance relationship depends on the token space. Within a fixed tokenizer, T2I loss strongly tracks generation quality, and I2T loss also correlates (more weakly) with generation quality. Across tokenizers, T2I loss–performance shifts with the image-token space, whereas I2T loss over a shared text vocabulary remains a more consistent cross-tokenizer signal.

• Vocabulary normalization partially corrects the T2I shift. Dividing the image loss by log₂(B), the entropy of a uniform distribution over B tokens, produces a more consistent T2I loss–performance relation across IBQ variants (B = 1024, 8192, 16384). The residual gap at a fixed vocabulary size is associated with reconstruction fidelity, with lower-rFID tokenizers reaching the same generation quality at higher T2I loss.

• I2T loss predicts post-SFT performance. I2T loss measured before supervised finetuning correlates with both post-SFT generation quality and general VQA scores (VQAv2, GQA) across tokenizers, making it a useful diagnostic. The relationship is weaker for specialized benchmarks like TextVQA that require OCR capability.

• Better reconstruction does not imply better multimodal learning. Replacing GigaTok's PatchGAN discriminator with a DINO-based one improves rFID from 0.81 to 0.51, but validation losses do not improve, GenAI-Bench is unchanged, and VQAv2 drops from 52.25 to 51.31. Conversely, adding semantic supervision to UniTok worsens rFID (1.86 to 2.23) yet improves all three validation losses and both downstream metrics (GenAI 0.670 to 0.690; VQAv2 57.21 to 61.28).

• Semantic supervision encourages object-level token–word associations. The UniTok-sem variant improves multimodal learnability despite worse reconstruction, indicating that pushing tokens toward high-level semantics helps joint modeling even when pixel fidelity suffers.

• The image token space can affect text modeling. Under joint training, UniTok-sem achieves lower text loss than UniTok, and IBQ-1024 achieves lower text loss than IBQ-8192, despite identical text data, text tokenizer, and language backbone. When the image-generation objective is ablated (training on text + I2T only), the text-loss gaps largely disappear—showing the interference stems from image-token prediction rather than captioning.

• Vocabulary size has a non-monotonic relationship with learnability. Larger vocabularies do not monotonically improve joint modeling, though they can still benefit downstream performance, likely through higher reconstruction fidelity.

Methodology in Plain English

The researchers wanted to see how a tokenizer's design choices ripple through a unified model that predicts both text and images. They started with pretrained Qwen3 language models and gave each model a new set of learnable embeddings—one per image token in the tokenizer's codebook—so the model could produce image tokens the same way it produces words. They added special markers to denote the start and end of image sequences.

Training had two stages. In continual pretraining, the models saw 60 million samples: about 6.6 million pure-text samples and 53.3 million image–text pairs from LAION-Aesthetics, JourneyDB, and BLIP3o. Image–text samples were formatted either as T2I (text prompt followed by image tokens) or I2T (image tokens followed by a caption), with 80/20 weighting and 10 percent of T2I samples made unconditional to enable classifier-free guidance later. The loss was computed only on the tokens being predicted—prompts and conditioning images provided context but were not scored. A second stage finetuned the models on 4.9 million instruction-following samples (text instructions, multimodal instructions, and T2I samples).

The key measurement was validation loss on held-out data, split into four tasks: pure text, unconditional image, T2I, and I2T. The authors held out 50,000 text and 50,000 image–text samples. They then plotted these losses against training compute and compared them to benchmark scores—GenAI-Bench and MJHQ-30K for generation, VQAv2 and GQA for understanding—both before and after supervised finetuning. To isolate tokenizer effects, they compared pairs of tokenizers differing in exactly one design axis: PatchGAN vs. DINO discriminator (GigaTok pair), with vs. without semantic supervision (UniTok pair), and vocabulary size 1024 vs. 8192 vs. 16384 (IBQ family).

Why This Matters

Impact on research. The paper reframes image tokenizers as a "visual language" that must be learned jointly with text, rather than as a preprocessing step judged by reconstruction quality. It supplies concrete evidence that rFID and similar isolated metrics are insufficient for selecting tokenizers in unified models, and it introduces I2T loss as a practical, cross-tokenizer diagnostic that correlates with both generation and understanding performance. It also documents a cross-modal effect—image tokenizer choice changing text modeling difficulty—that single-axis evaluations cannot detect.

Real-world applications.

• Unified multimodal assistants. Systems that both generate images and answer questions about them can use I2T validation loss as an early model-selection signal, avoiding expensive full finetuning cycles for every candidate tokenizer.
• Text-to-image generation pipelines. Understanding that vocabulary normalization makes T2I loss comparable across tokenizers helps teams choose tokenizers more rationally, rather than defaulting to the lowest rFID.
• Efficient model training. The finding that some tokenizers reduce text-modeling interference under joint training could inform architectures that train faster or need less capacity for a given quality target.
• Visual compression and representation. Because the token space shape affects cross-modal alignment, the results are relevant to any system that compresses images into discrete codes for downstream language-model consumption.

Industry relevance. Any organization building or deploying unified multimodal models—image generation, captioning, visual question answering—faces tokenizer selection decisions. This paper argues those decisions should be made inside the joint training loop, not from reconstruction benchmarks alone. The vocabulary-normalization trick (dividing by log₂B) is a low-cost adjustment that makes loss comparisons across tokenizer families more meaningful.

Future Directions

• Extend the study to other compression axes. The authors deliberately fixed the number of image tokens (16×16) and input resolution (256×256), deferring varying resolution (e.g., any-resolution tokenizers) and varying token counts (e.g., highly compressed 1D tokenizers) to future work, since these change visual detail and sequence length.

• Move beyond single-codebook tokenizers. Multi-codebook designs that represent each spatial position with several indices were excluded because they introduce extra choices in sequence organization; studying them would require new controlled comparisons.

• Apply the framework to continuous tokenizers. The testbed is pure-AR with discrete tokens, but many unified models use continuous or hybrid tokenization; extending the loss-based lens to those settings is an open question.

• Close the loop with end-to-end tokenizer tuning. The authors freeze tokenizers to compare fixed visual token spaces, explicitly contrasting with work that tunes tokenizers jointly with the downstream model. Whether a learnability-aware objective could be used to train better tokenizers directly is unresolved.

• Clarify why semantic supervision helps. The observed benefit of semantic supervision on joint learnability, despite worse reconstruction, suggests object-level token–word associations matter—but the mechanism and its limits across tasks and scales remain to be characterized.

Target Audience

This paper is most useful to researchers and engineers working on unified multimodal models, particularly those involved in tokenizer design, model selection, or scaling analysis. It also benefits practitioners who train autoregressive image-and-text models and need diagnostic signals that predict downstream quality before committing to full finetuning. Readers interested in the theory of how discrete visual representations interact with language modeling—and in why reconstruction metrics can mislead—will find the framing valuable. Some background in tokenization and AR modeling is assumed.

Authors’ abstract

Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. We examine how these losses scale and relate to downstream performance, then use them to study multimodal learnability---how well image and text tokens are jointly modeled---and tokenizer design. We find that (1) losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently. (2) The loss--performance relationship depends on the predicted token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers, the T2I loss--performance relationship shifts with the image-token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal. I2T loss also correlates with both generation and visual understanding performance after supervised finetuning. Using losses as a lens, we show that (3) better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that (4) image tokenizer choice can affect text modeling under joint optimization. As case studies, we revisit three tokenizer design axes---the discriminator, semantic supervision, and vocabulary size---to examine their effects on joint modeling and downstream performance. Together, our testbed offers a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training.