Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers

AI summary

Overview

Research area: Efficient inference for video diffusion transformers (video generation with Diffusion Transformers, or DiTs), specifically block-sparse attention and the problem of how to allocate sparsity across layers and denoising steps.

Technical level: Intermediate. The paper assumes familiarity with transformer attention, diffusion/denoising trajectories, and block-sparse kernels, but its core argument is stated in plain terms.

Scope (one sentence): The paper proposes ALLOT, a training-free method that precomputes how much each (layer, step) pair of a video DiT can be sparsified before it damages the final video, and then solves for a density table that fits a measured wall-clock latency budget, replacing only the allocation component of existing block-sparse attention pipelines.

What This Paper Is About

Full 3D attention in video DiTs is expensive: the authors note that on a single GPU, Wan2.1-14B takes roughly one hour to generate a 720p, 81-frame video, and attention accounts for more than half of the end-to-end time. Block-sparse attention cuts this by computing only selected query–key blocks, but existing methods decide how many blocks to keep in each layer at each denoising step using hand-set rules (e.g., keep the first layer or the first 10–30% of steps dense) or local proxy metrics such as attention mass, Hellinger distance, or one-step output error. The paper's goal is to replace those guesses with an allocation derived directly from measured damage to the final video, while staying within a measured latency budget.

Key Contributions

1. A diagnostic finding about sensitivity. The authors show that final-video sensitivity varies by orders of magnitude across layers and denoising steps, that local attention-map metrics misjudge this ranking, but that the ranking is nearly stable across input prompts for a fixed model and selection strategy.
2. The ALLOT method. A training-free allocator that treats each (layer, step) pair as a "cell" and estimates its sensitivity as the product of a one-step cell probe and a calibrated trajectory gain, then jointly optimizes all cell densities under a measured-latency budget.
3. A latency-aware solver. The allocation is posed as a constrained minimization where the cost of a density is the measured attention time, not a density- or FLOP-proportional estimate, and is solved by Lagrangian relaxation with a bisection on the multiplier (Proposition 1 states each per-cell independent minimization is an exact optimum when the budget equals the table's own total cost).
4. Drop-in integration and broad evaluation. ALLOT keeps each selection strategy's ranking and kernel, replacing only the cutoff that decides how many ranked blocks or clusters are retained; experiments span six selection strategies and three model families (Wan2.1, Wan2.2, HunyuanVideo).

Main Findings

• Layer sensitivity is extreme and mispredicted by attention metrics. The first one or two layers have one-step errors up to 154× the median-layer error (Wan2.1-14B, t=0.85). Attention-mass-based coverage density and Hellinger² rank layers differently from the cell probe; their Spearman correlations with the probe are as low as −0.51 on HunyuanVideo, and allocations guided by them degrade fidelity.
• Step sensitivity is understated by one-step error. Final-video error ratios between the first and last 5-step windows reach 463× (Wan2.1-1.3B), 186× (14B), 69× (HunyuanVideo), and 109× (Wan2.2-A14B), whereas the corresponding one-step error ratios are only 6.4×, 2.4×, 31×, and 0.18×. On Wan2.2 the one-step ordering even reverses.
• Rankings transfer across prompts. Median Spearman correlations between per-input and aggregated sensitivity rankings are 0.80–1.00, which is what allows the probing and calibration to be done once offline and reused.
• ALLOT beats every compared allocation strategy on XAttention. On Wan2.1-1.3B 480p×81 with n=32, ALLOT reaches 26.39 dB PSNR versus 18.14 dB for XAttention's published default, 19.43 dB for HEART (EBC), 19.38 dB for DiTFastAttn-style, 22.75 dB for ScalingAttention (FAST), 21.91 dB for PASA, and 24.86 dB for DFSAttn-style, at 151.3 s versus 148.7 s for the default — a stated margin of 1.53–7.31 dB over compared allocation methods.
• Large gain on HunyuanVideo. The abstract reports an 11.8 dB PSNR gain with XAttention on HunyuanVideo at equal or shorter wall-clock time.
• Works as a drop-in on published pipelines. Changing only the allocation across nine settings, ALLOT wins in six (by up to 5.03 dB) and ties in three. On HunyuanVideo with SVOO it improves PSNR by 3.38 dB while raising speedup from 1.75× to 1.90×.
• Cross-model gains over fixed rules. With mean-pooled top-k and the VSA kernel, PSNR goes from 19.59 to 25.99 on Wan2.1-1.3B 480p×33 (+6.40 vs. uniform, +2.55 vs. a strengthened baseline), 24.87 to 31.25 on 14B 720p×81 (+6.38, +0.76), and 19.36 to 25.86 on Wan2.2-A14B 480p×33 (+6.50, +1.67).
• Trajectory calibration is the dominant component. Removing it (setting w(s)≡1) drops PSNR by 7.05 dB; using one density for all layers drops 1.07 dB; a parametric shape rule drops 1.76 dB; density-proportional cost changes results by only −0.12 dB after wall-clock alignment.
• The sensitivity estimate predicts well. Across 80 tables on held-out prompts, predicted and measured rankings agree with a pooled Kendall τ of 0.83, and permuting ALLOT's densities among cells lowers PSNR on every prompt.
• Density tables are not simple warm-ups. The solved tables keep the first few layers and certain deeper layers dense longest, and density decreases over several intermediate steps rather than dropping at a single step; the best hand-set rule also changes with the budget.
• Per-head splitting hurts. Splitting a cell's budget per head according to SVOO's default head densities is 1.44 dB worse than ALLOT's equal split (n=16).

Methodology in Plain English

The authors start from the observation that you cannot afford to test every (layer, step) combination by generating full videos, so they break the problem into two cheap measurements.

First, a cell probe: take a real video, add noise to it at a given noise level, sparsify exactly one layer to a given density while leaving every other layer dense, and compare that step's velocity prediction against the dense model's prediction. This costs one forward pass and tells you how much error a layer injects locally. They tabulate it over 8 densities × 9 noise levels on four OpenVid videos.

Second, a trajectory calibration: divide the 50 denoising steps into 10 windows of 5 steps, and for each window generate a video with all layers sparsified to a fixed density of 0.30 in that window only, keeping every other step dense. The pixel MSE of that video against the dense reference, divided by the summed probes of the cells inside the window, gives a gain w(s) shared by all steps in the window. This captures how an early error is amplified by all later denoising steps — the effect one-step errors miss.

Multiplying the two gives an estimated sensitivity per cell. A solver then picks, for every cell, a density from a candidate set (equally spaced sparse densities plus fully dense), minimizing total estimated sensitivity subject to a budget on measured attention time. The time cost r(b) is measured with the actual kernel on the actual shape and normalized so a dense cell costs 1, because sparse kernels only beat dense FlashAttention below a crossover density — assuming cost proportional to density would overstate the savings. The constrained problem is handled by Lagrangian relaxation: for a given penalty λ each cell independently picks the density minimizing its own estimated error plus λ times its cost, and a bisection on λ finds the table whose total cost meets the budget. Cells whose density would fall above the crossover are simply set dense, and cells with b=1 use the original dense attention.

At inference, the precomputed table is looked up per step, the same table is used for all prompts, and each sparse cell is executed by the original selection strategy and kernel at that density. Warm-up length and which layers stay dense are outputs of the solver, not settings.

Why This Matters

Impact on research. The paper reframes sparsity allocation as a measurable, output-grounded optimization rather than a tuning problem, and shows that the popular local proxies used in prior work can rank layers in the wrong order. It also isolates allocation as a separable axis: existing selection strategies and kernels can be kept, so improvements from better block selection and better allocation should compose. The observation that layer–step sensitivity is input-stable is a reusable empirical result.

Real-world applications (bullets):

• Text-to-video services that must serve many concurrent requests, where cutting attention time directly increases throughput per GPU.
• High-resolution, long-duration generation (the paper evaluates 720p and 129-frame settings) where attention dominates latency and full 3D attention is impractical.
• Creative and pre-visualization tools that need fast iterative prompting at fidelity close to the dense model.
• Content pipelines that already run block-sparse attention stacks and want a fidelity improvement without retraining or new kernels.

Industry relevance. The method requires no training, no architecture change, and no new kernel — only an offline measurement pass and a lookup table — which makes it cheap to adopt on top of shipped inference stacks. The offline cost reported is roughly 6–23 GPU-hours per model (over 30 for the two experts of Wan2.2), measured once per model and selection strategy, and changing the latency budget or step count only requires re-solving the table, which takes about 4 s on CPU.

Future Directions

• Modeling cell interactions. The estimate assumes layer perturbations superpose within a step and that downstream amplification depends only on the step, not the layer; the authors list modeling these interactions as future work, and note the estimate predicts rankings more reliably than absolute error magnitudes.
• Preserving native adaptive thresholds. The common density interface replaces per-head or per-input adaptation in threshold-based methods (SVOO, SVG2, SpargeAttn, XAttention), which the authors flag as a limitation to address.
• Reducing calibration cost and transfer. Offline measurement is required, and switching selection strategy requires new cell probes; cheaper or transferable calibration across strategies and model families is an open question. The reported 0.993 correlation in w when calibrating with SpargeAttn rather than mean-pooled top-k, with 96.0–96.5% cell-decision agreement, points in this direction.
• Extending beyond block-sparse attention. Since allocation is framed as a per-cell error-versus-latency trade-off, applying the same output-calibrated allocation to other approximate-attention families (static masks, trained sparse attention) is a natural next step.

Target Audience

Researchers and engineers working on efficient video generation, diffusion transformer inference, or sparse attention kernels will get the most from this paper. It is also useful for practitioners deploying Wan2.1, Wan2.2, or HunyuanVideo pipelines who want a fidelity gain without retraining, and for students studying how to ground system-level optimization decisions in end-to-end output measurements rather than proxy metrics. Readers need working knowledge of transformer attention and diffusion sampling; the paper's argument is otherwise self-contained.

Authors’ abstract

Full 3D attention is costly in video Diffusion Transformers. Block-sparse attention reduces this cost by computing only selected attention blocks. However, existing methods either overlook the potential to optimize sparsity allocation or rely on heuristic rules or local metrics to guide it. In this work, we observe that sparsifying attention in different layers and at different denoising steps affects final-video fidelity to substantially different degrees. Yet the identities of the most sensitive layer-step pairs remain nearly unchanged across inputs. Building on these findings, we propose Allocating Layer-step Latency by Output Tests (ALLOT), a training-free sparsity allocation method. Treating each layer-step pair as a cell, ALLOT estimates each cell's contribution to final-video error through (i) cell probing and (ii) trajectory calibration, then jointly optimizes cell densities under a measured-latency budget. It preserves existing selection strategies and kernels, replacing only the allocation component. At equal or shorter wall-clock time, ALLOT outperforms every compared allocation strategy with XAttention, with an 11.8 dB PSNR gain on HunyuanVideo. Extensive experiments across four additional published pipelines on Wan2.1, Wan2.2, and HunyuanVideo show that ALLOT outperforms or ties the default in all nine settings.