Diffusion Language Models Change Speed Across Tasks and Languages

AI summary

Overview

Research area: Natural Language Processing — inference efficiency and decoding behavior of diffusion language models (with a secondary focus on multilingual fairness and reasoning-model reliability).

Technical level: Intermediate. The paper is readable without deep math, but it assumes familiarity with terms like "forward pass," "tokenization," "majority voting," and "reasoning traces."

Scope: An empirical measurement of how many forward passes two diffusion language models spend per output token across 10 tasks, 25 target languages, and different difficulty levels, plus an analysis of which output properties predict that cost.

What This Paper Is About

Diffusion language models can emit several tokens per forward pass, and they decide at generation time how many passes to spend — so two answers of identical length can cost very different amounts of compute. The paper asks what drives that variation: which tasks, output properties, languages, and even which answers (correct vs. incorrect) make decoding cheap or expensive. It also tests whether the cost signal itself is useful, by using decoding speed to pick among sampled answers.

Key Contributions

1. A normalized cost metric and a 10-task measurement. The authors define cost as forward passes per 100 output tokens, cost(y) = 100 · P/T, where an autoregressive model is fixed at 100. They measure it on 2,468 items per model across seven tasks in three groups (reproducing, chained, open-ended), plus three rewriting tasks (translation, clinical note writing, abstractive summarization) from Section 3.1 onward.

2. Two interpretable predictors of decoding cost. They introduce novelty (how many copied fragments versus new words an output contains) and surprisal (average word rarity within the task), and combine them into new information (NI = novelty × surprisal), which correlates with cost better than either alone.

3. A within-generation analysis with thinking on. They show that a final answer behaves like a "reproducing" task whose input is the model's own reasoning trace: answers that copy the trace are cheap, answers that add new content are not. They extend this to problem difficulty and to correctness.

4. Two practical downstream findings. A "double multilingual tax" (languages needing more tokens also need more passes per token), and a cost-free answer-selection method that weights each sampled answer's vote by how quickly the answer was decoded, beating plain majority voting.

Main Findings

• Task dominates decoding cost. With thinking off and fixed decoding settings, DiffusionGemma averages 4.0 forward passes per 100 tokens on reproducing tasks, 5.8 on chained tasks, and 9.8 on open-ended tasks. Grammar correction is cheapest at 3.72 and story generation most expensive at 11.43 — story tokens cost 3.1× as much.

• The ranking replicates across architectures. DreamReasoner requires 22.1, 56.2, and 87.0 passes per 100 tokens for the same three groups (a 4.3× spread). Its absolute costs are far higher because it writes blocks of 32 tokens versus DiffusionGemma's 256, but the ordering of tasks is identical.

• New information is the best single predictor. For DiffusionGemma, novelty alone correlates with cost at r = +0.72 and NI at +0.76 (p = 0.011); for DreamReasoner, +0.79 and +0.85 (p = 0.002). Surprisal alone does not correlate significantly (r = +0.18, p = 0.623 for DiffusionGemma; r = −0.09, p = 0.800 for DreamReasoner).

• Novelty and surprisal capture different things. Math and code generation are nearly as novel as abstractive summarization for DiffusionGemma (83%, 85%, 83%) but far less surprising in their wording (7.0 and 7.9 bits vs. 9.2), and they cost much less (6.2 and 6.2 vs. 9.5).

• Three tasks break the fit. Translation has NI comparable to story generation (8.90 vs. 8.86 for DiffusionGemma) yet costs 6.0 versus 11.4, because novelty only sees surface copying, not cross-language reuse of content. Story writing and abstractive summarization are unpredictable at the discourse level, which word-level surprisal cannot capture.

• Final answers copy from their own thinking. In one worked example (AIME 2025 problem 2, answered correctly), the model thinks for 10,339 tokens across 40 blocks plus one shared block, then writes a 1,241-token answer in 5 more blocks. Thinking blocks take 4 to 22 forward passes and cost 3.81 on average; answer blocks are even and cost 2.88. The answer has 87% novelty against the question but only 15% against the thinking.

• The copying pattern holds across tasks. With thinking on, the less novel the final answer is relative to the thinking, the cheaper it is compared with the thinking (r = +0.89, p = 0.001). Grammar correction answers have 1% novelty and cost 3.52 vs. 5.32 for the thinking; extractive summaries have 3% novelty and cost 2.07 vs. 3.50. On WildBench and story writing, where the answer is 68% and 76% novel relative to the thinking, the answer costs more (+0.60 and +2.32). Within-task correlations between novelty and the gap are positive in all 10 tasks (+0.06 to +0.90) and significant in 7.

• Harder problems cost more, but the rise comes from wrong answers. Across 160 competition-math problems from 5 contests with 5 seeds each, accuracy falls from 79% on the easy third to 57% on the middle third and 44% on the hard third. Median thinking length grows from 4,416 to 9,710 tokens while answer length barely changes; thinking cost rises from 5.00 to 5.89 and answer cost from 4.15 to 5.21. Correct answers stay flat (3.88 → 4.02) while wrong answers climb (5.16 → 6.13). Controlling for contest, position, and token counts, incorrect answers need +1.20 more forward passes per 100 tokens; holding problem and length fixed across seeds, +0.83.

• Wrong answers "keep working." Qualitatively, wrong final answers stop to self-correct ("wait," "let me re-check") roughly five times as often as right ones.

• Multilingual cost compounds. Translating the same 200 English FLORES sentences into 25 languages, Armenian needs 16.9 forward passes per sentence versus 5.3 for Spanish (3.2×). Part is token count (75 vs. 36 tokens, 2.1×) and part is passes per token (24.7 vs. 15.7 per 100 tokens). Across the 20 space-separated languages, tokens per word correlates with cost per token (r = +0.65, p = 0.002), and heavier tokenization also means worse translation quality (r = −0.81 with chrF++).

• Decoding speed is a usable certainty signal. On 255 competition-math problems with 5 runs each, weighting each answer's vote by answer tokens per forward pass beats plain majority voting at every sample count: 4.2 points with two answers and 2.3 with five. With five answers, accuracy goes from 71.4% to 73.7% (95% CI of the gain: [0.3, 4.5] points over 10,000 bootstrap resamples). Weighting by 1/(thinking tokens) instead gives 71.0%, close to the plain vote, so the gain is not just a preference for short reasoning.

Methodology in Plain English

The authors pick two diffusion models that both decide adaptively how many tokens to "commit" at each step, and run them with their published settings only — no tuning. DiffusionGemma-26B-A4B-it writes blocks of 256 tokens and commits the lowest-entropy tokens first up to a total entropy of 0.1; DreamReasoner-8B is dense, writes blocks of 32, and commits every token with confidence ≥ 0.9. Both allow up to 48 steps per block.

To measure cost, they count how many times the network runs to produce an output and divide by the number of real generated tokens (up to the first <eos>), scaling to a per-100-token figure. An autoregressive model is always 100 by construction, so any number below 100 is a saving. They normalize by generated tokens rather than block size so that models with 32-token and 256-token blocks can be compared directly.

The tasks are chosen to vary how much the answer is pinned down by the input: copying tasks (grammar correction, extractive summarization), chained tasks (code editing, code generation, competition math), open-ended tasks (WildBench user queries, story writing), plus three rewriting tasks (German-to-English translation, clinical notes, abstractive chapter summaries). Prompts are either the benchmarks' official ones or written by the authors.

To explain cost, they work at the word level rather than the token level, so the measures are tokenizer-independent. Novelty counts segments: contiguous runs copied verbatim from the input count as one segment each, and every uncopied word counts as its own segment, divided by output length. Surprisal is the average −log₂ of each word's add-one-smoothed frequency within that task's generations. Multiplying the two gives new information.

For the reasoning analysis they turn thinking on and exploit the fact that DiffusionGemma decodes thinking and answer the same way, in 256-token blocks, so the passes spent on each can be counted separately. They use contest position within a round as a rough difficulty proxy and run five seeds per problem. For the multilingual study they translate the same 200 sentences into 25 languages across 16 scripts, measuring fertility and length on reference translations and quality with chrF++. Outputs that hit the token limit or fall under 5 tokens are excluded; 3,860 generations were produced in total on NVIDIA H100 GPUs.

Why This Matters

Impact on research. Speed claims for diffusion language models are usually reported as tokens per forward pass on a handful of math and code benchmarks. This paper shows that number is not a property of the model alone — it moves by 3.1× between tasks within one model, and by more once language and answer correctness are considered. That reframes diffusion's efficiency advantage as conditional rather than general, and it gives the interpretability literature a quantitative handle (NI) on why certain decoding steps are cheap.

Real-world applications:

• Inference cost forecasting and routing: because cost is predictable from task type and output properties, serving systems can budget compute per request, or route open-ended generation to autoregressive decoding and copying-heavy work to diffusion.
• Free answer selection in reasoning deployments: weighting sampled answers by decoding speed improved accuracy with zero extra overhead, which is attractive anywhere self-consistency voting is already used.
• Multilingual serving fairness: the "double multilingual tax" means users of heavily tokenized languages pay twice — in tokens and in passes per token — which matters for pricing, latency SLAs, and equitable access.
• Early stopping and verification: since wrong answers keep consuming passes and self-correct out loud, decoding effort can flag low-confidence generations for a second look.

Industry relevance. Teams deploying diffusion models for document editing, code repair, or summarization are exactly in the cheap regime, while creative writing and low-resource translation are in the expensive one. The voting result requires no model changes and no extra compute, making it immediately deployable.

Future Directions

• Effort-guided inference: use decoding effort to drive verification, extra sampling, or early stopping, rather than using it only as an after-the-fact measurement.
• Task-adaptive decoding: adjust block sizes and stopping rules to the structure of the task and to how much of the output reuses earlier text — for instance, smaller blocks for open-ended writing.
• Hybrid decoding: selectively switch between autoregressive and diffusion decoding depending on what remains to be generated, an idea the authors raise explicitly in the conclusion.
• Broadening the evidence: the study rests on two models, both very recent (DreamReasoner from June 2026, DiffusionGemma from August 2026), because few others combine adaptive decoding with enough capability for reasoning, translation, and long-form writing. Extending to more scales and training recipes is left open, as is measuring discourse-level unpredictability, which the authors note resists word-level surprisal and may need human judgment.

Target Audience

Researchers and engineers working on diffusion language models, inference efficiency, or serving infrastructure; multilingual NLP researchers interested in tokenization fairness; and anyone studying reasoning-model reliability and self-consistency methods. It is also a useful read for practitioners deciding whether diffusion decoding is the right choice for a given workload, since the per-task cost numbers in Table C.1 are directly actionable. Readers without background in LLM decoding will need to pick up the cost metric first, but the conceptual argument — that cost tracks how much of the output is already determined — is accessible without it.

Authors’ abstract

Diffusion language models can write several tokens in one forward pass, and they decide during generation how many passes to spend. Two outputs of the same token length can therefore vary a lot in cost. To understand the underlying factors of this cost variation, we analyze two diffusion models across 10 tasks. We find that the number of forward passes per token depends heavily on the task. Text copied or lightly edited from the input is cheap, while open-ended writing is costly. In DiffusionGemma, generating a token for story writing requires more than three times as many forward passes as for grammar correction. Two output properties explain much of this gap: how much of it is new relative to the input, and how rare its words are within the task. The same holds within a single response when thinking is involved. The final answer is cheap when it repeats the model's reasoning trace and costly when it adds something new. In multilingual generation, we find that highly tokenized languages also require more passes per token, adding to the cost they already pay with more tokens. Finally, cost tracks correctness: on the same math problem, wrong answers take more passes per token than correct ones, and weighting candidate answers by decoding speed beats majority voting by up to 4.2 points at no extra cost. Taken together, our results show that diffusion's computational advantage depends on how much of the output is already determined, while its decoding effort offers a practical signal for making inference more efficient and reliable.