Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

AI summary

Overview

Research area: Computer vision and multimodal foundation models — specifically discrete visual tokenization and native multimodal large language models.

Technical level: Advanced. The paper assumes familiarity with vector quantization, codebooks, vision transformers, and vision-language model training pipelines.

Scope: A study of what governs the semantic fidelity of discrete visual tokens, an analysis of bit-budget scaling limits in existing quantizers, and a proposed quantizer (Factorized Vector Quantization, FVQ) validated with an industrial-scale 8B multimodal model.

What This Paper Is About

Multimodal models normally feed continuous visual features from a vision encoder into a language model, but a growing line of work instead converts images into discrete tokens so that vision and language share one next-token-prediction interface. The standard objection is that quantization into a finite codebook necessarily destroys semantic information, making continuous features an intrinsic upper bound. This paper tests that assumption, argues that quantization error is the wrong yardstick, and proposes a quantizer design that keeps improving as the bit budget grows until discrete representations match continuous ones on multimodal understanding.

Key Contributions

1. Reframing semantic fidelity. The authors show empirically that feature-reconstruction error does not predict downstream multimodal understanding performance, and argue from first principles (digital images are themselves finite-bit signals) that a sufficiently large effective bit budget is what determines semantic capacity.

2. Factorized Vector Quantization (FVQ). A multi-codebook quantizer that projects the full visual feature into complementary, fixed-dimensional views via dense tight-frame projections, quantizes them with symmetric parallel codebooks, and reconstructs through tight-frame synthesis. It avoids the saturation seen in residual vector quantization (RVQ) and product quantization (PQ).

3. Industrial-scale validation. A model pairing Qwen3-8B with an any-resolution tokenizer built from 16-codebook FVQ on Qwen3 ViT, trained on large-scale multimodal data, achieves state-of-the-art results among discrete-representation models — 22.8% relative improvement over UniAR and 59.2% over Emu3-Chat on a nine-metric average — and is reported as the first discrete-representation model competitive with strong continuous models such as Qwen3-VL, including fine-grained OCR.

4. A practical recipe and stated design principles. The paper distills the conditions that make discrete representations competitive: fixed-dimensional views derived from the full feature, collectively information-preserving projections, and symmetric codebook roles without a residual hierarchy.

Main Findings

• Quantization error is not a reliable proxy for semantic fidelity. With codebook size 16,384, single-codebook VQ has MSE 0.027 and an eight-benchmark average of 51.2; the RVQ "odd" variant (4 codebooks) has roughly an order-of-magnitude larger MSE of 0.246 yet scores 60.7, while the "even" variant with MSE 0.028 scores 59.3. Full 8-codebook RVQ reaches MSE 0.016 and average 64.0.

• Bit budget drives performance. The discrete budget is B = L·log₂C, linear in the number of codebooks L but only logarithmic in codebook size C, which makes multi-codebook scaling the efficient route. Scaling the budget produces steady gains in multimodal understanding.

• Existing quantizers saturate. Both RVQ and PQ plateau at high budgets: going from 8 to 16 codebooks at C = 16,384 doubles the nominal budget (112 to 224 bits) but adds little. RVQ stays at 64.0 average, PQ moves from 64.1 to 64.3. The authors attribute this to diminishing information per codebook — weaker residuals in RVQ, narrower channel groups in PQ.

• FVQ keeps improving and matches the continuous baseline. At 16 codebooks / 224 bits, FVQ reaches 65.3 average versus 64.6 for the continuous QwenViT representation, after scoring 62.4 at 56 bits and 63.9 at 112 bits. Higher budgets also progressively recover fine-grained text needed for correct OCR predictions.

• Strong results on STEM and general benchmarks. The 8B model scores 64.3 on MMMU, 40.8 on MMMU-Pro, 77.4 on MathVista, 72.0 on RealWorldQA, and 68.5 on MMStar — matching Qwen3-VL-8B on MathVista (77.2 reported for Qwen3-VL) and taking the best RealWorldQA result in the table, well ahead of discrete baselines such as UniAR-8B (44.3 MMMU, 50.8 MathVista) and Emu3-Chat-8B (31.6 MMMU, 47.6 MathVista).

• The OCR gap narrows substantially. The model scores 867 on OCRBench, 58.7/59.5 on OCRBenchV2 (en/zh), 83.9 on ChartQA, 86.1/49.3 on CharXiv (DQ/RQ), 95.4 on DocVQA, 77.7 on InfoVQA, and 0.179/0.249 on OmniDocBench (en/zh). It is comparable to Qwen3-VL-8B on OmniDocBench and DocVQA and surpasses the continuous models listed on both CharXiv tasks.

• The effect generalizes across encoders. On MoonViT-v2, moving from 14-bit single-codebook VQ to 112-bit eight-codebook FVQ raises the average from 38.7 to 61.1; on SigLIP2, from 44.7 to 53.9.

• Gains survive without industrial-scale pretraining. Under LLaVA-OneVision SFT only, FVQ scores 53.1 MMMU, 60.2 MathVista, 65.2 RealWorldQA, 58.5 MMStar, 791 OCRBench, 81.6 ChartQA, and 90.3 DocVQA — the best discrete result in that comparison, ahead of ViQ (49.1, 54.2, 711, 72.8, 88.9).

• Qualitative evidence on hard fine-grained cases. The model handles a table at 21,253 × 3,912 pixels, irregular handwritten selections on a scanned exam sheet, and a dense page requiring more than 5,000 output characters.

Methodology in Plain English

Step 1 — Test the assumption. Instead of trusting reconstruction error, the authors train small vision-language models on discrete features from different quantizers and compare them on eight understanding benchmarks. They deliberately build "partial" RVQ variants (keeping alternating codebook groups) that trade off error against informativeness, to see whether error predicts performance.

Step 2 — Reason about bits. They treat raw pixels, continuous features, and discrete indices all as finite-bit encodings and compute per-patch bit budgets for each. Since images are highly redundant, the semantic content should be capturable with far fewer bits than raw pixels require — so the question becomes whether a discrete code can be given enough useful bits.

Step 3 — Diagnose why scaling stalls. They scale codebook counts up for RVQ and PQ and observe both flattening. The diagnosis: later codebooks get less informative inputs (small residuals, or narrow slices of the feature).

Step 4 — Design FVQ. Each codebook gets its own dense projection of the entire feature into a fixed-dimensional view, so adding codebooks adds new views rather than shrinking inputs. The stacked projection matrix is constrained to be a tight frame (AᵀA = ρI with ρ = Ld/D), meaning every feature direction is scaled equally and none is lost. Concretely, they build it by taking a fixed Gaussian matrix and applying reduced QR decomposition, scaled by √ρ, then keep it frozen during training. Codebooks quantize their views independently and in parallel; reconstruction uses the closed-form synthesis operator (1/ρ)Aᵀ, which exactly inverts the projection before quantization, so the factorization itself introduces no error — all distortion comes from nearest-codeword replacement.

Step 5 — Train at scale. The quantizer is trained to reconstruct frozen vision-foundation-model features (no pixel supervision), using reconstruction, codebook, and commitment losses. Training proceeds in four stages: quantizer training (100B tokens), alignment with tokenizer and LLM frozen (10B tokens), pretraining (60B tokens), and supervised fine-tuning (30B tokens), with context lengths of 8K and 16K and maximum image resolutions of 1024px and 2048px. During VLM training the quantizer is frozen and only the discrete indices are mapped to newly initialized token embeddings; standard 1D RoPE is used and DeepStack connections from Qwen3 ViT are omitted, keeping the model close to a native next-token setup.

Step 6 — Ablate. They check other vision encoders (MoonViT-v2, SigLIP2) and a reduced LLaVA-OneVision-only training recipe to separate the FVQ effect from data scale.

Why This Matters

If discrete visual tokens can match continuous features on understanding, then vision can be folded into the same token interface, architecture, and scaling recipes as language — a step toward genuinely unified multimodal foundation models rather than encoder-plus-LLM assemblies. The paper also supplies a concrete, diagnosable failure mode (codebook saturation from uninformative inputs) and a design principle for fixing it, which is more useful to the field than a single benchmark win. Its claim that weak OCR in discrete models stems from an information bottleneck rather than from discretization itself directly challenges a common assumption.

Real-world applications implied by the evaluated capabilities:

• Document and form processing — the paper demonstrates extraction from dense scanned exam sheets with irregular handwriting, and parsing across diverse PDF document types via OmniDocBench.
• High-resolution document and table reasoning — the 21,253 × 3,912-pixel table case points to uses in financial statements, invoices, and technical reports.
• Chart and infographic understanding — evaluated on ChartQA, CharXiv, AI2D, and InfoVQA.
• Unified multimodal generation pipelines — the introduction notes discrete visual representations can serve as visual chains of thought or intermediate representations for image generation, spatial understanding, and depth estimation within unified models.

Industry relevance: The work comes from an industrial lab context (Meituan LongCat, with authors from Huazhong University of Science and Technology, Shanghai Jiao Tong University, Peking University, Tsinghua University, The Chinese University of Hong Kong, and Mohamed bin Zayed University of Artificial Intelligence), is trained on industrial-scale multimodal data, and ships with a public code repository and project page — signalling an intent for the tokenizer to be reused as a component in production-scale multimodal systems.

Future Directions

• Pushing the bit budget further. FVQ was tested up to 16 codebooks / 224 bits. Whether the gains continue at substantially larger budgets, and where a new saturation point lies, is left open.

• Studying "effective" versus nominal capacity directly. The paper argues that nominal bits only help when added codes remain informative, but the reported diagnostics are downstream benchmark averages, not an intrinsic measure of effective semantic capacity.

• Extending beyond understanding. The introduction motivates discrete tokens as visual chains of thought and as intermediate representations for generation, spatial understanding, and depth estimation, but the experiments here evaluate multimodal understanding only.

• Scaling the recipe. The competitive results come from a specific four-stage schedule and a Qwen3-8B backbone with a frozen quantizer; how the design principles transfer to much larger LLMs, other fusion architectures, or encoder-free designs is not reported.

Target Audience

Researchers and engineers working on multimodal foundation models, visual tokenization, and vector quantization — especially those building discrete-token or "native" multimodal architectures who need to know where the real information bottleneck lies. It is also useful for practitioners deploying document, chart, and OCR-heavy vision-language systems, and for readers interested in the broader argument that vision-language unification need not sacrifice fine-grained visual fidelity. Readers without background in quantization and VLM training will find the empirical comparisons accessible but the FVQ derivation demanding.

Authors’ abstract

Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and positioning continuous visual representations as an intrinsic upper bound. Using multimodal understanding as a proxy for semantic fidelity, we revisit this assumption and find quantization error unreliable, as discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information. As one approach to alleviating this saturation, we introduce Factorized Vector Quantization, which quantizes complementary, fixed-dimensional views of the visual feature with symmetric parallel codebooks. It continues to improve at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding. Trained on industrial-scale multimodal data, our model achieves state-of-the-art performance among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2%, respectively, on a nine-metric average. To our knowledge, it is also the first discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks. We further identify the key principles and a practical recipe enabling this competitiveness. These results and insights support the potential of discrete visual representations to underpin large-scale native multimodal foundation models. Code and models are available at https://github.com/hustvl/FVQ