QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for Video World Models

AI summary

Overview

Research area: Efficient inference for generative video models — specifically low-bit KV cache quantization applied to video world models and autoregressive video generation models (Computer Vision / model compression).

Technical level: Intermediate. Readers need some familiarity with Transformer attention, KV caching in autoregressive decoding, and post-training quantization, but the paper's core argument is stated in intuitive terms.

Scope: The paper diagnoses why 2-bit KV cache quantization silently degrades video world models even when benchmark scores look fine, traces the cause to Key-induced shifts in attention logits, and proposes a training-free two-part fix (QSAC + PSAC) that is evaluated on five models.

What This Paper Is About

Video world models keep a KV cache of past Key and Value representations so each newly generated video chunk can attend to earlier context, but that cache grows continuously and becomes a deployment bottleneck — the paper states LingBot-World-v2 needs more than 21 GiB of KV cache to generate a 5-second video of about 93 frames. Existing 2-bit KV cache quantization methods report nearly lossless scores on video benchmarks such as VBench, yet the authors find they still produce severe temporal flickering, blurring and artifacts on video world models. The goal is a training-free 2-bit quantization framework that explicitly protects attention behavior so visual quality and temporal consistency hold up.

Key Contributions

1. A diagnosis of an overlooked failure mode. The authors show that benchmark-level scores (e.g., VBench) do not capture the temporal flickering and visual degradation that 2-bit KV cache quantization causes in video world models, and that this degradation is dominated by Key quantization rather than Value quantization.

2. QuantiSation-sensitivity-Aware Clustering (QSAC). A centroid-selection scheme that replaces plain K-means distance with a score combining historical Query-channel sensitivity and group-wise residual dynamic range, so chosen centroids are the ones expected to perturb QKᵀ least after INT2 residual quantization.

3. Principal-Subspace Attention Compensation (PSAC). A low-rank correction that extracts the top-r eigenvectors of the accumulated second-order Query statistics and restores the remaining Key quantization error along those dominant directions, directly compensating the attention logits at inference time.

4. A training-free framework with empirical validation. QuantWM is evaluated on three world models (Matrix-Game-2, HY-World 1.5, LingBot-World-v2) and two autoregressive video generation models (LongCat-Video, Causal-Forcing), reporting up to 6.20× KV cache memory compression with limited additional overhead.

Main Findings

• Key quantization, not Value quantization, drives the damage. In the paper's sample-level comparison, K16V2 beats K2V16 on every frame-level metric: on Matrix-Game-2, PSNR 21.58 vs. 18.47, SSIM 0.708 vs. 0.547, LPIPS 0.126 vs. 0.219. On LingBot-World-v2, K16V2 gets PSNR 17.70 / SSIM 0.476 / LPIPS 0.157 versus K2V16's 15.38 / 0.355 / 0.263. K2V2 performs similarly to K2V16.

• Smaller reconstruction error does not mean smaller output error. Table 1 reports Key MSE below Value MSE in most measured layers, yet K2V16 generally produces a larger output MSE than K16V2. For example, in the last layer of HY-World 1.5, K2 MSE is 0.0090 versus V2 MSE 0.2939, but K2-Output MSE is 0.5884 versus V2-Output MSE 0.1679.

• The mechanism is token-selection shift. Because attention logits are QKᵀ/√d, any Key perturbation Δ changes the relative ordering of logits and can redirect which temporal-spatial tokens a Query selects. Visualizations confirm that 2-bit Keys cause Queries to attend to different historical frames or spatial regions than BF16.

• QuantWM substantially reduces token-selection shifts. Top-1 temporal-spatial token selection shift ratio drops from 0.5388 to 0.1019 (Matrix-Game-2), 0.5491 to 0.1600 (LingBot-v2), 0.6136 to 0.1392 (HY-World 1.5), 0.5034 to 0.2225 (LongCat-Video), and 0.3728 to 0.1420 (Causal-Forcing) — an average reduction from 51.55% to 15.31%.

• Frame-level quality improves on all five models at both resolutions. At 480p, LongCat-Video PSNR rises from 20.683 (QVG) to 25.508 and LPIPS falls from 0.0784 to 0.0332. At 720p, the same model goes from 20.988 to 25.127 PSNR. QuantWM also reports the best average rank on VBench across all five models, while the Temporal score stays close across methods — which the authors use as further evidence that this metric misses the degradation.

• Ablation shows both components matter. On HY-World 1.5, the baseline (neither) scores PSNR 16.118 / SSIM 0.6559 / LPIPS 0.2114 / token shift 0.6136. QSAC alone gives 17.990 / 0.7423 / 0.1285 / 0.3817. PSAC alone gives 17.849 / 0.7358 / 0.1312 / 0.2290. Both together give 18.449 / 0.7661 / 0.1082 / 0.1392 — best on every metric, and PSAC shows the stronger effect on attention preservation.

• Memory drops sharply. QuantWM achieves up to 6.20× KV cache memory compression. On LongCat-Video the cache falls from 21.709 GiB to 3.625 GiB for 93-frame generation.

• Latency overhead is modest and not always a cost. Against BF16, generation time increases 5.44% on LingBot-World-v2 (193.56 s to 204.09 s) and 5.87% on HY-World 1.5 (112.17 s to 118.75 s), while on LongCat-Video it actually decreases 17.78% (255.35 s to 209.94 s).

• Robustness under long-horizon generation. For 1-minute video generation, QuantWM outperforms QVG on most VBench dimensions and takes the best average rank across all five models; on HY-World 1.5 the Image score improves from 0.4756 to 0.6242.

• Query energy is concentrated in few directions. The top-8 directions capture more than 60% of total Query energy in LingBot-v2, which justifies the low-rank design; the authors set r = 8 after finding that raising the rank to 16 yields only modest gains while adding 0.1341 bits on HY-World 1.5 and 0.2569 bits on Causal-Forcing.

Methodology in Plain English

The starting point is an empirical dissection: quantize only Keys, then only Values, and compare both the numerical error and the visual output. The surprising result is that Keys have smaller numerical error but cause worse output, which points away from error magnitude and toward how Keys interact with Queries.

That interaction is the attention score QKᵀ. The authors reason that what matters visually is not how far a reconstructed Key is from the original, but whether it changes which past tokens a Query picks. They test this by visualizing which temporal-spatial patch each Query selects as its top-1 choice, and show that 2-bit Keys shuffle those choices.

Two techniques follow. QSAC keeps QVG's structure of a BF16 centroid plus an INT2-quantized residual, but changes how centroids are chosen. Instead of picking the nearest centroid, it accumulates second-order statistics of Queries seen in earlier generation steps, uses the diagonal of that matrix to weight each Key channel by how sensitive it is to Query error, and forms a Query-aware weighted distance to shortlist candidate centroids. Each candidate is then scored by combining those channel weights with the group-wise dynamic range of the residual it would produce — a proxy for how badly that residual will quantize. The centroid with the lowest expected logit perturbation wins.

PSAC handles what QSAC cannot fix. The remaining Key error after QSAC is decomposed against the eigenvectors of the same historical Query statistics; only the top r eigenvectors (r = 8) are kept as the principal subspace, and the error projected onto that subspace is stored as a low-rank correction (U_r in BF16, C = EU_r in INT8). At inference, this correction is added back so the attention logits are partially restored, at low storage cost because both matrices are low-rank.

Everything is training-free and applied stream-wise, chunk by chunk, on the model's native cached tensors. At the cold start, when no Query history exists, QSAC uses uniform channel sensitivity (w = 1) and PSAC is inactive until the first chunk's Query observations have been incorporated. Implementation uses 256 centroids, quantization groupsize 64 (QVG used 64 and KIVI 32), BF16 centroids with INT2 residuals, on NVIDIA A800 80GB GPUs, with 480p and 720p videos of 93 frames.

Why This Matters

For research: The paper challenges the assumption that benchmark parity implies quantization safety. It shows that VBench's Temporal dimension stays nearly flat even when QVG visibly flickers, and reframes KV cache quantization as a problem of preserving attention decisions rather than minimizing reconstruction error. That reframing — Key sensitivity is about logit and token-selection shifts, not MSE — is likely transferable to other autoregressive generative settings that rely on KV caches.

Real-world applications:

• On-device and edge video generation, where the stated KV cache requirement of more than 21 GiB for a 5-second clip on LingBot-World-v2 blocks deployment on memory-constrained hardware.
• Interactive world simulation, where each generated chunk must attend to prior world context under user actions and camera trajectories, making temporal consistency a functional requirement rather than a cosmetic one.
• Long-form / streaming video generation, since the paper explicitly tests 1-minute generation where quantization error accumulates across steps.
• Serving infrastructure for autoregressive video, where KV cache, not weights, is described as the dominant GPU memory bottleneck.

Industry relevance: The pitch is cost — up to 6.20× less KV cache memory for a latency change measured at +5.44%, +5.87%, and −17.78% on three models, with no retraining. For anyone serving long-context video generation, that is a direct lever on how many concurrent generations fit on a GPU and how long a video a given device can produce.

Future Directions

• Extending the attention-preservation principle beyond Keys. The paper's diagnosis centers on Key-induced logit shifts; whether the same low-rank compensation framing helps other cached representations, or other quantization bit-widths, is left open.

• A better benchmark for temporal degradation. The authors repeatedly note that VBench's Temporal score does not track the flickering they observe. A dedicated metric for quantization-induced temporal inconsistency would make this class of failure visible in standard evaluations.

• Reducing the residual overhead further. PSAC's rank trade-off (r = 8 vs. 16 vs. 32) is studied, but the additional storage of U_r and C, and the +5.44% / +5.87% latency on two models, suggest room for a cheaper correction or a learned subspace.

• Scaling and generalizing evaluation. Results cover five models at 480p/720p for 93 frames plus 1-minute generation; behavior at higher resolutions, longer horizons, or on models with different cache conventions is not established.

Target Audience

Researchers and engineers working on efficient inference for video generation and world models — particularly those doing KV cache compression, post-training quantization, or long-context autoregressive video serving. It is also relevant to anyone evaluating quantization methods against video benchmarks, because its central lesson is that benchmark parity can mask visible degradation. Readers should be comfortable with Transformer attention and basic quantization notation; the paper is not an introductory treatment of either.

Authors’ abstract

Video world models achieve long-range temporal consistency by storing KV cache during generation, but the growing cache makes KV cache memory a major deployment bottleneck, which motivates low-bit quantization study for efficiency. Existing 2-bit KV cache quantization methods can achieve nearly lossless performance on VBench, however, when applied to video world models, we find they still cause severe temporal flickering and visual degradation. Meanwhile, deeper investigates show that Key quantization produces smaller reconstruction errors than Value, but surprisingly leads to larger output degradation. We trace this discrepancy to attention in video world models: Key perturbations can change the attention logits, and shift the temporal-spatial tokens selected by Queries. These observations motivate us to preserve attention logits and temporal-spatial token selection during KV cache quantization. To address this issue, we present QuantWM, a training-free 2-bit KV cache quantization framework for video world models. QuantWM introduces two complementary techniques to mitigate the attention shifts. Firstly, quantization-sensitivity-aware clustering (QSAC) jointly considers historical Query sensitivity and residual ranges to select INT2-friendly Key centroids, which reduces quantization errors in channels that are more critical to attention. In addition, principal-subspace attention compensation (PSAC) restores the remaining Key errors along the dominant Query subspace using low-rank projections, which provides a direct and efficient correction to stabilize attention logits. Experiments on LingBot-World-v2, HY-World 1.5, Matrix-Game-2, Longcat-Video and Causal-Forcing demonstrate that QuantWM significantly improves visual quality and temporal consistency, while outperforming existing methods across benchmarks with up to 6.20 KV cache memory compression and limited additional overhead.