LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generation

AI summary

Overview

Research area: Efficient video generation with diffusion models — specifically replacing quadratic (softmax) self-attention with linear attention in pre-trained video diffusion transformers (DiTs).

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching models, diffusion transformers, attention complexity, and few-step distillation.

Scope: One-sentence scope: LinVideo is a data-free post-training framework that selectively converts a target number of self-attention layers in a pre-trained video diffusion model into linear attention, using a learned layer-selection mechanism and an "anytime distribution matching" objective, to cut inference latency while preserving generation quality.

What This Paper Is About

Video diffusion models produce high-quality video, but their self-attention cost grows quadratically with sequence length, and generating a 10s video often involves >50K tokens. Linear attention reduces this cost from O(n²) to O(n), but fully replacing quadratic attention has so far required costly pretraining because linear attention is less expressive and struggles with video's complex spatiotemporal dynamics. This paper asks whether an efficient post-training procedure — requiring no original training data — can replace as many quadratic attention layers as possible without degrading the video diffusion model.

Key Contributions

1. LinVideo framework: The authors introduce what they describe as the first efficient data-free post-training framework that replaces quadratic attention with linear attention in a pre-trained video diffusion model, enabling efficient video generation without compromising performance.
2. Selective transfer: A learning-based method that automatically and progressively replaces a target number of quadratic attention layers with linear attention, minimizing performance drop. Each layer is cast as a binary classification problem with a learnable scalar score in [0, 1], guided by a constraint loss toward the target count and a regularization term pushing scores toward 0/1.
3. Anytime distribution matching (ADM): An objective that aligns the sample distributions of the linearized model and the original model at any timestep along the sampling trajectory, estimating the score function with the current model itself rather than an auxiliary diffusion model.
4. Empirical results: Extensive experiments report a 1.43–1.71× latency speedup with preserved quality and better VBench results than prior post-training methods; the authors state they are the first to apply few-step distillation to a linear-attention video DM, with 4-step models attaining a 15.9–20.9× speedup.

Main Findings

• Layer choice matters greatly: Using 4 VBench dimensions on Wan 1.3B after 2K-step fine-tuning (10 adjacent layers per data point), linearizing layers 2–11 improved Subject Consistency by +2.86 and Image Quality by +6.31 versus linearizing layers 21–30. Shallow-layer linearization recovers more easily, but replacing the first layer causes significant drops that persist even after 3K additional fine-tuning steps.
• Latency and quality trade-off (Wan 1.3B, CFG=5.0, 480p, fps=16): FlashAttention2 baseline is 97.32s (1.00×); LinVideo is 68.26s (1.43×); LinVideo + DMD2 is 6.110s (15.9×). On most VBench dimensions LinVideo is at or above the baselines — for example Imaging Quality 66.07, Aesthetic Quality 59.41, Motion Smoothness 98.19, Dynamic Degree 59.67, Background Consistency 96.72, Subject Consistency 95.12, Scene Consistency 39.18, Overall Consistency 26.52.
• Wan 14B results (CFG=5.0, 720p, fps=16): FlashAttention2 is 1931s (1.00×), LinVideo 1127s (1.71×), and LinVideo + DMD2 92.56s (20.9×). LinVideo reports Imaging Quality 66.47, Aesthetic Quality 61.36, Motion Smoothness 97.24, Dynamic Degree 69.82, Background Consistency 96.34, Subject Consistency 93.68, Scene Consistency 33.72, Overall Consistency 26.16.
• Outperforms sparse-attention baselines: On the 1.3B model, LinVideo is 1.43× versus 1.09× (DFA), 1.17× (XAttn), 1.31× (SVG), and 1.15× (SVG2). On the 14B model, LinVideo is 1.71× versus 1.40× (DFA), 1.51× (XAttn), 1.61× (SVG), and 1.42× (SVG2).
• Few-step distillation impact: The 1.3B LinVideo + DMD2 model shows roughly a 1% performance drop while achieving a 15.92× end-to-end speedup.
• VBench-2.0 results: For Wan 1.3B, total scores are 56.74 (FA2), 55.81 (SVG2), 56.74 (LinVideo), and 55.51 (LinVideo + DMD2). For Wan 14B, total scores are 59.85 (FA2), 58.74 (SVG2), 59.62 (LinVideo), and 58.22 (LinVideo + DMD2). The 4-step distilled 1.3B model incurs less than 3% performance drops with higher scores than FA2 on specific metrics such as Human Identity and Multi-View Consistency.
• Target ablation: With target values 10, 12, 14, 16, 18, and 20, latency is 78.14s, 74.81s, 71.48s, 68.26s, 65.00s, and 61.68s respectively. Performance degrades slowly and stays stable until target = 18, after which a non-negligible drop occurs.
• Selective transfer ablation: LinVideo scores 66.07 Imaging Quality, 59.41 Aesthetic Quality, 98.19 Motion Smoothness, 59.67 Dynamic Degree, and 26.52 Overall Consistency. Manual layer assignment scores 62.97 / 57.21 / 92.25 / 52.87 / 20.08, and Heuristic (grid search) scores 60.74 / 54.13 / 90.36 / 50.61 / 18.94.
• Regularization matters: Removing the regularization term causes a severe collapse — scores fall to 18.62 Imaging Quality, 17.83 Aesthetic Quality, 12.59 Motion Smoothness, 7.48 Dynamic Degree, and 1.42 Overall Consistency.
• Lambda sensitivity: Performance variation across λ = 0.01, 0.1, and 0.001 is about 1%, indicating LinVideo is not sensitive to that hyperparameter.

Methodology in Plain English

The team starts from an already-trained video diffusion model (Wan 1.3B and Wan 14B) and avoids needing the original training dataset. Instead, they generate the training data from the model itself: they randomly sample initial noise, run the model's own sampling trajectory, and record the input/output pairs at each point along that trajectory.

To decide which attention layers to convert, they give every attention layer a learnable score between 0 and 1 that blends the original quadratic attention output with a linear attention output. At inference, the score is rounded so each layer becomes fully quadratic or fully linear. A constraint loss nudges the total number of linear layers toward a pre-set target, and a regularization term (borrowed in spirit from model quantization) pushes scores toward the extremes of 0 or 1, so rounding does not throw away a large attention contribution. Without this, many scores hover near 0.5 and rounding hurts accuracy.

For optimization, they reject two alternatives: direct output matching (which produces temporal artifacts like flicker and jitter) and standard few-step distillation (which only matches final sample distributions and requires an extra diffusion model to estimate the score). Their ADM objective instead matches distributions at any timestep along the trajectory, and it estimates the needed score functions using the model currently being trained. Under rectified-flow modeling, the score difference reduces to a simple expression involving the difference between the original model's velocity prediction and the linearized model's velocity prediction. Training uses a combined loss with a weighting hyperparameter λ.

Finally, they optionally apply DMD2-based few-step distillation on top of the linearized model to get 4-step generation, noting that directly distilling the original model into a few-step linear-attention generator causes catastrophic performance drops.

Why This Matters

Impact on research: The work shows that linear attention can be introduced into large pre-trained video diffusion models through post-training rather than pretraining, and that layer selection is a critical, learnable decision rather than a heuristic one. It also demonstrates few-step distillation on a linear-attention video DM, and its dense linear/quadratic design is described as orthogonal to sparse-attention methods, suggesting combinability.

Real-world applications:

• Faster text-to-video generation services where latency and compute cost dominate deployment budgets.
• Interactive or near-real-time creative tools for video content production, given the 15.9–20.9× speedups from the 4-step models.
• Deployment on resource-constrained or single-GPU inference setups — the reported latencies are measured on a single H100 GPU.
• Adaptation of existing pre-trained video models by teams that lack access to the original training data, via the data-free post-training setup.

Industry relevance: The framework targets an existing open-source model family (Wan) and reports comparisons against production-oriented sparse-attention methods, which is directly relevant to serving costs for video generation. The authors note they use no specialized kernel implementation, indicating further speedup is possible, and mention a comparison with SLA, which relies on a CUDA kernel that only supports the RTX 5090 GPU.

Future Directions

• Integrate efficient sparse attention modules into LinVideo, since the current design uses only dense linear attention or retained dense quadratic attention, which is orthogonal to sparsification.
• Develop specialized kernel implementations for the linear attention path, which the authors say would allow for an enhanced speedup ratio.
• Combine the data-free finetuning strategy with SLA's intra-layer mixed attention to achieve more efficient and higher-performing linearization for video DMs.
• Explore layer selection and target tuning more broadly, since the ablation indicates performance stays stable up to target = 18 but drops at target = 20, leaving the optimal point for other models and resolutions an open question.

Target Audience

Researchers and engineers working on efficient video generation, diffusion transformers, attention mechanism design, and model compression/distillation. It is most useful to readers already comfortable with diffusion and flow-matching models and with attention complexity analysis, and to practitioners looking to reduce inference cost for pre-trained video diffusion models without retraining from scratch.

Authors’ abstract

Video diffusion models (DMs) have enabled high-quality video synthesis. However, their computation costs scale quadratically with sequence length because self-attention has quadratic complexity. While linear attention lowers the cost, fully replacing quadratic attention requires expensive pretraining due to the limited expressiveness of linear attention and the complexity of spatiotemporal modeling in video generation. In this paper, we present LinVideo, an efficient data-free post-training framework that replaces a target number of self-attention modules with linear attention while preserving the original model's performance. First, we observe a significant disparity in the replaceability of different layers. Instead of manual or heuristic choices, we frame layer selection as a binary classification problem and propose selective transfer, which automatically and progressively converts layers to linear attention with minimal performance impact. Additionally, to overcome the ineffectiveness and inefficiency of existing objectives for this transfer process, we introduce an anytime distribution matching (ADM) objective that aligns the distributions of samples across any timestep along the sampling trajectory. This objective is efficient and recovers model performance. Extensive experiments show that our method achieves a 1.25-2.00x speedup while preserving generation quality, and our 4-step distilled model further delivers a 15.92x latency reduction with minimal visual quality drop.