Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspective

AI summary

Overview

Research area: Efficient inference and interpretability for diffusion-based Multimodal Large Language Models (dMLLMs), sitting at the intersection of computer vision, multimodal language modeling, and model-mechanism analysis.

Technical level: Advanced. The paper assumes familiarity with transformer attention, KV caching, masked/iterative denoising decoding, and information-theoretic quantities such as entropy.

Scope: The paper diagnoses a text-repetition side effect introduced by caching in diffusion-based multimodal LLMs, traces it to disrupted information flow through "context tokens," and proposes a training-free mitigation method called CoTA.

What This Paper Is About

Diffusion-based Multimodal Large Language Models are slow at inference because they must recompute attention over the entire sequence at each denoising step, so practitioners apply caching strategies to speed them up. The authors show that these caches cause responses to degenerate into repeated tokens, a phenomenon they name the Repeat Curse. The paper's goal is to explain the mechanism behind this failure, using information flow and entropy analysis, and to design a plug-and-play fix that removes the repetition without retraining the model.

Key Contributions

1. Identification of the Repeat Curse. The authors document a repetition phenomenon that appears when caching is applied to dMLLMs, and quantify it with four complementary metrics (Adjacent Repetition Rate, Sample Repetition Rate, Maximum Repetition Length, Average Repetition Length, plus a 95th-percentile repetition length).
2. A mechanistic explanation via information flow. Through attention-map and cross-layer entropy visualization, they show that context tokens act as anchors that aggregate semantic information and absorb attention, that their entropy converges in deeper layers under normal decoding, and that repetition coincides with disrupted information flow and non-convergent deep-layer entropy.
3. The CoTA method. A training-free, plug-and-play approach with two components: Context Tokens Attention Enhancement (CTAE), which applies a Gaussian distance-decay term to attention, and Context Tokens Entropy-Guided Voting (CTEV), which adds aggregated deep-layer context-token entropy (layers 26–30) as a penalty term to the confidence score used for token selection.
4. Extensive validation. Experiments on the repeat-curse metrics, eight multimodal benchmarks, a generality study, and hyperparameter ablations, including application to a second dMLLM (MMaDA) and comparison against a prefix-KV cache.

Main Findings

• Context tokens act as anchors. Attention maps from LLaDA-V 8B (generation length 128, 32 decoding steps) show context tokens receiving persistently high attention that concentrates progressively from shallow to deeper layers, guiding the final prediction. A similar pattern appears in MMaDA, including vertical bands of concentrated attention at Layer 22, Step 46.
• Entropy converges under normal decoding. For context tokens (defined as the target token plus its two nearest neighbors in relative position), entropy is high in shallow layers and converges in deeper layers, reflecting growing prediction certainty.
• Repetition correlates with broken information flow. With cache applied, the attention distribution becomes randomized and the entropy of repeating context tokens fails to converge in deeper layers.
• Cache design details matter. Ablations show the prompt-token recompute interval has no effect on repetition (SRR of 0 at intervals 1, 5, 15 and 25), while the output-token recompute interval and similarity threshold matter greatly: SRR rises from 0 at interval 1 to 79.9, 87.4 and 89.7 at intervals 3, 5 and 7; and threshold values of 0, 0.25, 0.5, 0.75 and 1 produce SRR of 89.7, 75.0, 69.8, 29.7 and 0. A prefix-KV cache produced SRR of 0, while dLLM-Cache produced 75.0.
• The Repeat Curse is not architecture-specific. On MMaDA, enabling dLLM-Cache raised ARR by +3.7 (from 0.7 to 4.4) and SRR by +49 (from 6.0 to 55.0). Applying prefix-KV cache to LLaDA-V and LaViDa did not trigger repetition (ARR 0, SRR 0).
• CoTA substantially reduces repetition. On 500 randomly sampled COCO2014 images for captioning, at a maximum generation length of 512, dLLM-Cache raised ARR from 0.2 to 14.3 and SRR from 6.9 to 82.3; the full method reduced these to ARR 1.2 and SRR 6.3 (a reduction of 13.1 in ARR, described as up to 92% lower adjacent repetition). At length 64, dLLM-Cache raised ARR from 0.1 to 7.1 and SRR from 3.3 to 65.6; the full method brought them to 1.0 and 3.0. Each component alone also mitigated repetition (CTEV alone: ARR 3.2 / 2.5; CTAE alone: ARR 2.9 / 1.8).
• Benchmark performance recovers. On the six benchmark columns of the comparison table, LLaDA-V scored DocVQA 83.9, ChartQA 78.3, MMStar 60.1, MME_p 1507, Seed_I 74.8, MMBench 82.9; adding dLLM-Cache gave 82.1, 78.1, 58.3, 1410, 72.1, 83.0; the proposed method gave 84.1, 78.4, 59.3, 1523, 73.9, 83.1.
• Generality and overhead. In the generality table, cached LLaDA-V scored 63.2 with ARR 7.3 on one setting and 54.9 with ARR 6.9 on another, while the method scored 69.9 with ARR 1.4 and 59.7 with ARR 1.3 respectively. The paper states gains of +6.7 in Score and +5.9 in ARR on MathVerse and +1.8 in ACC and +5.6 in ARR on MathVista, and reports only a 2.8 TPS reduction and a 1.9 FLOPs increase over the cached baseline.
• Hyperparameters. Best configuration on MathVerse: α = 0.75, γ_min = 0.5, and 3 context tokens, giving ARR 1.2 and ACC 23.1; the temperature factor τ is fixed at 5; using 5 context tokens performed worse. Using layers 26–30 for entropy accumulation was best (ARR 2.9 / SRR 10% at length 512), outperforming ranges such as 1–10, 11–20, 21–25, 26–31 and 26–32.
• Repetition is linguistically skewed. Over 200 samples exhibiting repetition, the most frequently repeated words were "the" (98%), "of" (76%) and "a" (56%).

Methodology in Plain English

The authors first pick a concrete baseline: LLaDA-V (with an LLaDA-8B-Instruct language tower, a siglip2-so400m-patch14-384 vision tower, and a two-layer MLP projector) accelerated by dLLM-Cache, with cache hyperparameters fixed at α = 25%, E_p = 25 and E_s = 7. They measure repetition with four metrics and confirm the problem visually and statistically.

To find out why, they look inside the model rather than treating it as a black box. They plot attention matrices to see which tokens receive attention at which layers, and they compute per-token entropy across layers to see how confident the model is. Comparing runs with and without the cache reveals that caching scrambles attention and prevents deeper layers from settling, so they build a fix directly targeting those two symptoms.

The fix has two parts. The first, CTAE, multiplies the attention matrix by a decay term that is large for nearby key/query positions and smaller further away, so attention flows back toward nearby context tokens. The decay is a Gaussian on the relative distance, with a temperature τ of 5 and a lower bound γ_min that stops it from going to zero. The second, CTEV, computes the normalized entropy of each token's prediction in layers 26 to 30, sums it across the token and its two nearest neighbors, and adds that sum, scaled by α, to the confidence score the model already uses to choose which tokens to unmask. Higher uncertainty around a position therefore makes it less likely to be committed early. Both parts are training-free and operate at inference time.

Why This Matters

The paper reframes a practical speed-versus-quality tradeoff in diffusion-based multimodal LLMs as an interpretability problem: the same caching that makes these models deployable quietly corrupts their internal attention and entropy dynamics, and knowing the mechanism makes it fixable without retraining. This matters both for practitioners who want caching's latency gains without the quality loss, and for researchers studying information flow in non-autoregressive multimodal models, where the "anchor token" concept is now shown to carry over from autoregressive architectures.

Real-world applications:

• Document and chart question answering, where caching is used to serve VQA-style requests (DocVQA, ChartQA) at scale and repeated tokens are directly visible in user-facing answers.
• Image captioning and accessibility tools, the exact setting used here (500 MSCOCO images), where repetition makes generated descriptions unreadable for users relying on them.
• Mathematical and reasoning assistants (MathVista, MathVerse), where cached decoding degrades scores and the method recovers part of that loss.
• Latency-sensitive multimodal deployment, such as on-device or high-throughput serving, where the reported overhead of a 2.8 TPS reduction and 1.9 FLOPs increase is small relative to the quality recovery.

Industry relevance: caching is a standard production technique for serving large multimodal models, and repetition is a visible quality bug. A training-free, plug-and-play intervention that can be layered on top of existing cache strategies and then removed with limited cost is directly compatible with existing serving infrastructure, and the diagnosis gives engineers concrete knobs (output-token recompute interval, similarity threshold, cache policy) that the ablations show dominate repetition behavior.

Future Directions

• Broader dMLLM coverage. The authors state that CoTA has not yet been validated across more open-source dMLLMs or base models

Authors’ abstract

Recent diffusion-based Multimodal Large Language Models (dMLLMs) suffer from high inference latency and therefore rely on caching techniques to accelerate decoding. However, the application of cache mechanisms often introduces undesirable repetitive text generation, a phenomenon we term the \textbf{Repeat Curse}. To better investigate underlying mechanism behind this issue, we analyze repetition generation through the lens of information flow. Our work reveals three key findings: (1) context tokens aggregate semantic information as anchors and guide the final predictions; (2) as information propagates across layers, the entropy of context tokens converges in deeper layers, reflecting the model's growing prediction certainty; (3) Repetition is typically linked to disruptions in the information flow of context tokens and to the inability of their entropy to converge in deeper layers. Based on these insights, we present \textbf{CoTA}, a plug-and-play method for mitigating repetition. CoTA enhances the attention of context tokens to preserve intrinsic information flow patterns, while introducing a penalty term to the confidence score during decoding to avoid outputs driven by uncertain context tokens. With extensive experiments, CoTA demonstrates significant effectiveness in alleviating repetition and achieves consistent performance improvements on general tasks. Code is available at https://github.com/ErikZ719/CoTA