Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization

AI summary

Overview

Research area: Efficient inference for auto-regressive video diffusion models — specifically KV-cache quantization and long-horizon video generation.

Technical level: Intermediate. Readers should be comfortable with transformer attention, KV-caching, and the basics of integer quantization; the paper is otherwise self-contained.

Scope: A training-free KV-cache quantization framework (Quant VideoGen, or QVG) that cuts the memory footprint of auto-regressive video generators by up to ~7x while preserving near-lossless video quality and long-horizon consistency.

What This Paper Is About

Auto-regressive video diffusion models generate long videos chunk by chunk, but they must keep a growing KV-cache of all past frames in GPU memory. That cache balloons quickly — a 5-second 480p clip from LongCat-Video needs roughly 38K tokens, or about 34 GB of KV-cache, which alone exceeds a single RTX 5090. Worse, shrinking the cache to fit on-device hardware directly degrades long-horizon consistency in identity, layout, and motion, so memory is both an efficiency ceiling and a capability ceiling. This paper introduces a training-free quantization scheme that compresses the KV-cache down to 2 bits using structure specific to video, so longer effective context fits in the same hardware.

Key Contributions

1. Diagnosis of a system–algorithm coupled bottleneck. The authors show that KV-cache memory, not raw compute, binds auto-regressive video generation, and that bounded memory budgets directly cause long-horizon drift. They show video KV-caches have far more heterogeneous token- and channel-wise numeric distributions than LLM KV-caches, which is why LLM-oriented quantization methods fail here.

2. Semantic-Aware Smoothing. A chunk-wise preprocessing step that runs k-means over latent tokens along the sequence axis, groups semantically similar tokens, and subtracts each group's centroid. This produces low-magnitude, homogeneous residuals that are far friendlier to low-bit quantization.

3. Progressive Residual Quantization. A coarse-to-fine, multi-stage scheme inspired by progressive video codecs that repeatedly re-smooths and re-quantizes residuals, with the number of stages providing a tunable quality–memory trade-off.

4. Algorithm–system co-design. A streaming centroid-caching trick that reuses the previous chunk's cluster assignments to cut k-means overhead by ~3x, a fused dequantization-plus-centroid-add kernel that keeps intermediates in registers, pre-RoPE key caching, and FP8 E4M3 per-group scaling factors.

Main Findings

• Up to ~7x KV-cache compression at near-lossless quality. QVG reaches 6.94x compression on LongCat-Video-13B (PSNR 28.7) and 7.05x on HY-WorldPlay-8B (PSNR 29.2), versus baselines that top out at 6.40x with PSNR around 20–25 under INT2.

• Existing quantization baselines collapse at 2 bits. RTN, KIVI, and QuaRot all degrade severely at INT2 — e.g., 20.3–21.6 PSNR on LongCat versus QVG's 28.7. The gap comes from video-specific spatiotemporal redundancy that LLM-oriented methods do not exploit.

• Semantic-Aware Smoothing directly reduces quantization error. Measured error reductions of roughly 6.9x for the key cache and 2.6x for the value cache, since keys are more regular than values.

• Progressive stages have diminishing returns. The first stage produces the dominant error reduction (about 5.83x MSE reduction over naive quantization); later stages still help but by progressively smaller margins (≥1.10x each).

• Long-horizon drift is substantially mitigated. On Self-Forcing, QVG tracks BF16 quality out to 1400 frames (~90 seconds), while KIVI falls from 67.57 at 350 frames to 35.85 at 1400 frames.

• Latency overhead stays under 4%. Measured at 2.1% (LongCat), 1.5% (HY-WorldPlay), and 4.3% (Self-Forcing) on H100; about 1.7–1.9% on both H100 and RTX 5090, and stable across batch sizes from 1 to 5.

• Consumer-GPU deployment becomes feasible. QVG is the first method to run HY-WorldPlay-8B on a single RTX 4090 with PSNR above 29 relative to the BF16 reference.

• Resolution scales without retuning. Moving LongCat-Video from 480p to 720p preserves the same per-chunk clustering scale and error bounds, so no additional centroids are needed — because compression operates on fixed-size KV chunks, not whole frames.

• Design choices validated by ablation. Block size 64 gives the best compression–quality trade-off; block size 16 maximizes quality. Codebook size K=256 was chosen because it fits cluster IDs naturally into a uint8 assignment vector while retaining most of the reconstruction benefit.

Methodology in Plain English

The researchers observed that in a video model's KV-cache, tokens corresponding to nearby patches in space or adjacent frames in time look numerically similar. That similarity is the lever.

Their pipeline works chunk by chunk. For each new chunk of video tokens, they cluster tokens into groups using k-means, then subtract each group's average (centroid) from every token in it. What remains is a residual tensor with a much smaller dynamic range and far fewer extreme outliers. Since quantization error scales with the largest magnitude in the tensor being quantized, shrinking that range directly shrinks the error.

They then repeat the process on the residual itself. Each additional stage clusters and subtracts again, capturing progressively finer details — analogous to how video codecs encode a coarse base layer plus refinement layers. The centroids and cluster assignments are stored (cheaply, since assignments fit in one byte), the intermediate residuals are thrown away, and the final residual is quantized to 2 or 4 bits. Reconstruction simply reverses the process, adding centroids back one stage at a time.

To keep this practical for streaming inference, they cache centroids from the previous chunk to warm-start the next clustering, fuse the dequantization and centroid-addition into a single kernel, and store keys pre-RoPE so their distribution is more quantization-friendly.

Why This Matters

Impact on research. The paper reframes the KV-cache in video diffusion as a capability constraint, not just an efficiency one — more retained history means better identity, layout, and motion preservation over minutes of generation. It also establishes that video KV-caches have distinct statistics from LLM KV-caches, opening a new sub-area of quantization research tuned to spatiotemporal structure. Because QVG is training-free, it can be dropped onto existing checkpoints without retraining.

Real-world applications:

• Live-streaming and interactive video generation, where chunk-level causality plus a small memory footprint enables real-time streaming.
• World models and 3D-consistent spatial exploration, where minute-scale rollouts on a single GPU become viable.
• Interactive content control (e.g., game environments) that need to condition on long interaction histories.
• On-device or prosumer-GPU content creation, demonstrated by running an 8B world-play model on a single RTX 4090.

Industry relevance. The result that an 8B video model can now fit on consumer hardware substantially broadens who can deploy these systems. Cloud providers serving video generation also benefit directly, since memory — not compute — has been the binding constraint, meaning the same GPUs can serve far longer videos or more concurrent users.

Future Directions

• Pushing below 2 bits. The paper evaluates INT2 and INT4; whether 1-bit or mixed-precision-per-channel schemes remain viable given the residual structure is unexplored.
• Combining with orthogonal memory-reduction techniques. Token eviction, sparse attention, and learned memory mechanisms are complementary to quantization but were not jointly evaluated.
• Extending the idea beyond the KV-cache. The authors quantize only the cache; applying similar semantic-smoothing to weights and activations (which QuaRot-style methods do jointly) could yield further savings.
• Making clustering adaptive. The number of centroids, groups, and progressive stages is fixed globally; content-adaptive allocation could improve the quality–memory Pareto frontier, and learnable codebooks may beat k-means.

Target Audience

Systems and machine learning researchers working on efficient inference, quantization, and generative video; engineers deploying video diffusion models under real GPU memory budgets; and practitioners building streaming video, world-model, or interactive-content applications who need long-horizon consistency without datacenter-class hardware.

Authors’ abstract

Despite rapid progress in autoregressive video diffusion, an emerging system algorithm bottleneck limits both deployability and generation capability: KV cache memory. In autoregressive video generation models, the KV cache grows with generation history and quickly dominates GPU memory, often exceeding 30 GB, preventing deployment on widely available hardware. More critically, constrained KV cache budgets restrict the effective working memory, directly degrading long horizon consistency in identity, layout, and motion. To address this challenge, we present Quant VideoGen (QVG), a training free KV cache quantization framework for autoregressive video diffusion models. QVG leverages video spatiotemporal redundancy through Semantic Aware Smoothing, producing low magnitude, quantization friendly residuals. It further introduces Progressive Residual Quantization, a coarse to fine multi stage scheme that reduces quantization error while enabling a smooth quality memory trade off. Across LongCat Video, HY WorldPlay, and Self Forcing benchmarks, QVG establishes a new Pareto frontier between quality and memory efficiency, reducing KV cache memory by up to 7.0 times with less than 4% end to end latency overhead while consistently outperforming existing baselines in generation quality. Code is available at: https://github.com/svg-project/Quant-VideoGen