A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models

AI summary

Overview

Research area: Efficient inference for discrete diffusion-based multimodal large language models (dMLLMs), specifically visual token redundancy and visual token pruning.

Technical level: Advanced. The paper combines empirical benchmarking of multimodal models with attention/layer-level analysis and a floating-point-operation (FLOPs) complexity analysis of iterative denoising.

Scope: A systematic study of whether visual token redundancy exists in dMLLMs, how pruning visual tokens changes accuracy and inference time, and which pruning strategies fit which dMLLM architecture.

What This Paper Is About

Discrete diffusion-based MLLMs generate text by repeatedly denoising a whole answer sequence in parallel, which means every denoising step runs full-sequence bidirectional attention and becomes expensive. Existing acceleration work (key-value cache optimization, efficient sampling, variable-length generation) treats the problem as modality-agnostic and ignores whether visual tokens are redundant. This paper asks two questions: does visual token redundancy exist in prevalent dMLLMs, and how does pruning those tokens affect inference speed and accuracy? It answers both by testing representative pruning methods on two dMLLM backbones across twelve benchmarks, then explaining the results through attention and logit dynamics.

Key Contributions

1. The first comprehensive study of visual token redundancy in dMLLMs, covering two representative backbones (LLaDA-V, a from-scratch dMLLM, and LaViDa-Dream, an AR-to-diffusion dMLLM) and six token pruning methods taken from the autoregressive MLLM literature.
2. A demonstration that visual token pruning causes much greater information loss in dMLLMs than in MLLMs, and that only from-scratch dMLLMs can progressively recover that loss during late denoising steps, which explains why their performance drops stay minimal on long-answer tasks.
3. Practical guidance on matching acceleration strategy to architecture: layer-skipping is promising for AR-to-diffusion dMLLMs, while progressive or late-step pruning works better for from-scratch dMLLMs, with attention scores and step-wise output logits both serving as reliable pruning indicators.
4. A computational cost formulation for dMLLM decoding showing that each denoising step performs a prefilling-style computation with bidirectional attention, so the linear and quadratic terms on the number of active tokens make reducing redundant visual tokens especially valuable.

Main Findings

• Redundancy is architecture- and task-dependent. Visual redundancy emerges only in from-scratch dMLLMs (LLaDA-V) and only while handling long-answer tasks; in the AR-to-diffusion dMLLM (LaViDa-Dream) it remains largely absent.
• Initial-step pruning hurts both backbones. Applying visual token pruning at the initial denoising step causes substantial performance drops in both LLaDA-V and LaViDa-Dream, indicating visual tokens are essential at that stage.
• Short-answer tasks show no usable redundancy. For both backbones, pruning at retention ratios of 50%, 25% and 10% gives either slight degradation for marginal efficiency gains or minimal acceleration at the cost of large performance degradation. For LaViDa-Dream the trade-off is worse still, and FastV and SparseVLM are even slower than the baseline due to incompatibility with efficient attention operators.
• Long-answer pruning diverges sharply between backbones. On LLaDA-V, DivPrune prunes 75% of visual tokens in the middle decoding steps, giving a 1.44 times speed-up on InfoVQA and DocVQA (30.5% and 30.7% time reduction) with almost no performance degradation (65.0 vs. 66.2; 83.1 vs. 83.9). On LaViDa-Dream, compression at any portion of the denoising steps yields consistent but suboptimal results, with performance drops of nearly 50% alongside reduced efficiency.
• dMLLMs rely on visual tokens far more than MLLMs. Answer tokens in MLLMs show near-zero attention to visual tokens across the whole decoding process, whereas dMLLMs maintain a much stronger attention ratio toward visual tokens. In from-scratch dMLLMs this ratio stays high across layers; in AR-to-diffusion dMLLMs it spikes sharply at a specific layer.
• Self-attention among answer tokens drives restoration. Self-attention intensity among answer tokens increases from MLLMs to LaViDa-Dream to LLaDA-V, giving dMLLMs (especially LLaDA-V) a greater capacity to recover lost information through bidirectional contextual refinement.
• Logit surges follow self-attention surges. In LLaDA-V on sentence-level long-answer tasks, a rise in logits follows a preceding surge in answer-token self-attention (for example, the logit increase at Step 8 coincides with the rise in self-attention at Step 7), suggesting stronger self-attention facilitates information restoration during denoising.
• Restoration needs both attention strength and enough steps. Short-answer tasks have very few denoising steps and the answer is typically determined within the first one or two steps, so early pruning leaves no room to recover while late pruning saves little time. This is why redundancy appears only in from-scratch dMLLMs on long-answer tasks.
• Redundancy has a different meaning in dMLLMs. In MLLMs, redundancy is dispensable visual tokens that cost computation but contribute little. In dMLLMs, redundancy instead arises from the model's ability to recover missing information through iterative bidirectional refinement.

Methodology in Plain English

The researchers took two representative dMLLM backbones: LLaDA-V, trained from scratch with a diffusion objective, and LaViDa-Dream, produced by adapting an autoregressive model to diffusion. They selected six visual token pruning methods that work well on autoregressive MLLMs, covering three underlying mechanisms: similarity-based pruning (ToMe, DivPrune), attention-based pruning (FastV, VTW) and query-based pruning (SparseVLM, TRIM). Transformation-based pruning was excluded because it is a built-in, fixed-rate design that cannot be adjusted at inference time.

They then ran a grid of experiments: prune at different retention ratios (50%, 25%, 10%, and 0% for VTW), start pruning at different denoising steps or apply it to different fractions of the total denoising steps (for example 25%, 50% or 75%), and evaluate on twelve widely adopted benchmarks, ten for image understanding and two for video understanding. All experiments ran on 8 x A800 GPUs. For reference, Figure 1 also applies three pruning methods to the autoregressive MLLM LLaVA-NeXT alongside the two dMLLMs.

To explain the results, they measured two things. First, the fraction of attention from answer tokens to each token type across layers, comparing MLLMs, LaViDa-Dream and LLaDA-V on short-answer, sentence-level long-answer and paragraph-level long-answer tasks. Second, logit dynamics across denoising steps, to see whether rising confidence follows rising answer-to-answer self-attention. They also derived FLOPs expressions for prefilling, autoregressive decoding and dMLLM decoding, where the dMLLM cost sums over denoising steps and depends quadratically on the number of tokens actively involved in each step.

Note that the provided paper content is truncated at Section 5.3, so later sections are not covered here.

Why This Matters

Impact on research. Prior dMLLM acceleration work was modality-agnostic. This study shows that visual token redundancy in dMLLMs is not a simple extension of the MLLM picture: whether redundancy exists depends on the training paradigm (from-scratch diffusion versus AR-to-diffusion adaptation) and on the task type (short-answer versus long-answer). That reframes visual token pruning for diffusion models as a problem of whether the model can restore pruned information through iterative bidirectional refinement, not merely whether tokens are dispensable.

Real-world applications.

• Document and information-heavy visual QA, where page images produce large visual token counts and long answers.
• Video description and captioning, where generation is long and redundancy is reported to emerge most clearly.
• Multimodal assistants that need lower latency per query, since dMLLM inference cost grows with the number of denoising steps and active tokens.
• Deployment on constrained hardware, where reducing token count directly reduces attention cost.

Industry relevance. Inference cost is the main practical obstacle to serving dMLLMs, whose reference runtimes in this study are reported in minutes (for example 79:44 on the LLaDA-V baseline and 90:05 on LaViDa-Dream across the benchmark suite). Clear rules for when pruning helps and which strategy to use per architecture let teams cut GPU time without retraining, and the finding that some pruning methods are slower than the baseline because they break efficient attention kernels is directly actionable for serving stacks.

Future Directions

• Turning the diagnostic findings into concrete acceleration methods: progressive or late-step pruning schedules for from-scratch dMLLMs, and layer-skipping designs for AR-to-diffusion dMLLMs.
• Testing whether the attention-score and step-wise logit signals identified as pruning indicators can drive adaptive, per-step pruning policies rather than fixed schedules.
• Investigating whether AR-to-diffusion dMLLMs can be trained or adapted to gain the restoration ability that from-scratch dMLLMs show, for example

Authors’ abstract

Discrete diffusion-based multimodal large language models (dMLLMs) have emerged as a promising alternative to autoregressive MLLMs thanks to their advantages in parallel decoding and bidirectional context modeling, but most existing dMLLMs incur significant computational overhead during inference due to the full-sequence attention computation in each denoising step. Pioneering studies attempt to resolve this issue from a modality-agnostic perspective via key-value cache optimization or efficient sampling but most of them overlook modality-specific visual token redundancy. In this work, we conduct a comprehensive study on how visual token redundancy evolves with different dMLLM architectures and tasks and how visual token pruning affects dMLLM responses and efficiency. Specifically, our study reveals that visual redundancy emerges only in from-scratch dMLLMs while handling long-answer tasks. In addition, we validate that visual token pruning introduces non-negligible information loss in dMLLMs and only from-scratch dMLLMs can recover the lost information progressively during late denoising steps. Furthermore, our study shows that layer-skipping is promising for accelerating AR-to-diffusion dMLLMs, whereas progressive or late-step pruning is more effective for from-scratch dMLLMs. Overall, this work offers a new perspective on efficiency optimization for dMLLMs, greatly advancing their applicability across various multimodal understanding tasks. The code is available at: https://github.com/Yrdal3910/dMLLM-Visual-Token-Redundancy-Analysis.