VAR-Q: Tuning-free KV Cache Quantization for Visual Autoregressive Image and Video Generation

AI summary

Overview

Research area: Efficient inference for visual autoregressive (AR) generative models — specifically KV-cache compression via low-bit quantization for image and video generation.

Technical level: Intermediate to Advanced. Readers should know what a transformer KV cache is, what group-wise quantization does, and roughly how RoPE (rotary position embeddings) works; the paper does not teach these from scratch.

Scope: A single paper that characterizes why KV caches in visual AR models have a structure unlike language-model caches, and turns that characterization into a training-free, calibration-free quantization scheme evaluated across nine models.

What This Paper Is About

Visual autoregressive models generate images and video by emitting whole blocks of tokens at a time — one resolution scale in next-scale prediction, or one frame in next-frame prediction — rather than one token at a time like a language model. The KV cache these models accumulate grows large enough to rival or exceed the model weights and to saturate a single GPU, but the quantization recipes developed for LLMs assume statistics that do not hold here. The paper explains what the visual-AR cache actually looks like, and proposes VAR-Q, a grouping rule that matches that structure so the cache can be stored in very few bits without retraining or calibration.

Key Contributions

1. A necessity result for cache outliers. Through direct channel ablation on four visual AR models, the authors show that zeroing the top 0.5% of channels by magnitude in LongLive collapses generation (CLIP 0.32 to 0.28), while zeroing ten times as many inlier channels leaves CLIP at the 0.32 baseline — establishing that these outliers are functionally load-bearing, not merely numerically inconvenient.
2. A two-axis structural characterization. Massive values organize along a channel axis (persistent stripes) and a block axis (bands that shift sharply at block boundaries), and they concentrate at the intersections of the two. A grouping table on Infinity-2B shows per-(channel, block) grouping reduces output error ε_O from 32.32% (per-tensor) to 10.13%, better than per-block or per-channel alone.
3. An architectural attribution of both axes. Key outliers arise from a compounding chain — heavy-tailed W_K row norms, post-projection RMSNorm gain γ_K, and RoPE pair rotation — while the block axis is traced to RoPE frequency: fast-rotating pairs swing across blocks while slow pairs stay flat. Value outliers are shown to be largely input-driven rather than architectural.
4. VAR-Q, a tuning-free quantizer with kernels. Groups are indexed by (block, channel), block boundaries come free from the generation loop, and a Triton bit-packing implementation converts nominal bit-width into realized memory savings — up to 78.5% KV reduction and up to 6× larger inference batches on a single NVIDIA A100 80GB.

Main Findings

• Ablation separates necessity from magnitude. Across Infinity, InfinityStar, SelfForcing and LongLive, and at every layer measured, outlier-channel ablation collapses generation while matched inlier ablation does not. A per-tensor decomposition further shows video models are more sensitive to K outliers and image models to V outliers.
• K outliers are architectural; V outliers are input-driven. Measured by cross-prompt persistence at N=10 and p=1%, K retains 0.53–0.62 of its outlier channels across all prompts while V retains only 0.07–0.14, a 3.8×–8.5× gap. At N=2, V already drops to 0.31–0.35 while K stays at 0.77–0.82.
• The K chain compounds multiplicatively. At p=1%, W_K alone recovers 35–42% of final K outliers (35–42× random), adding γ_K lifts this further, and the full projection chain reaches 55–63× random recall. W_V alone yields only 5–6× random recall as a control.
• Fast RoPE pairs generate the block axis. Within a pair, the two channels swap energy across blocks in anti-phase while the pair's ℓ2 norm stays approximately flat; the lowest-index (fastest-rotating) pairs dominate the block-axis outliers.
• Grouping must cover both axes. On VAR-d30 at INT4, removing the channel axis degrades FID from 1.97 to 12.07 and IS from 323.67 to 159.52; removing only the block axis raises FID to 2.83. On Self-Forcing, removing the channel axis drops Subject Consistency from 91.22 to 79.17 and Dynamic Degree from 74.00 to 27.00.
• Boundary placement matters at fixed grouping count. Shifting every group boundary by half a block lowers Self-Forcing quality from 80.91 to 80.46, worsening every VBench dimension measured.
• Next-scale image results. On VAR-d20/d24/d30 at INT4, VAR-Q matches BF16 FID (e.g., VAR-d30: 1.97 vs 2.07) and leads KIVI and FlexGen on PSNR, SSIM and LPIPS against matched-seed BF16 outputs; at INT3 it stays within 1.0 FID of BF16 while FlexGen collapses (FID 15.3/14.9/9.5). On Infinity-2B/8B at INT4 it holds GenEval and DPG Overall within 0.3 points of BF16. Infinity-8B also holds at INT3 (GenEval 83.28, Δ=0.26, versus KIVI losing 0.79) and remains usable at INT2 (GenEval 74.38, DPG 83.83).
• Next-scale video results. On InfinityStar at 720p and 480p, VAR-Q matches BF16 in VBench Total Quality at INT4 (81.66 and 81.32 vs 82.33 and 81.30). At INT3 on 720p, KIVI drops to 75.46 while VAR-Q stays at 80.59, within 1.74 points of BF16.
• Next-frame video results. At INT6, VAR-Q is indistinguishable from BF16 (Self-Forcing 83.34 vs 82.94; LongLive 80.94 vs 80.92), and Opus4.7-VL judge scores on LongLive corroborate this (3.84/3.31/3.71 vs 3.75/3.25/3.69). At INT4 the gap to BF16 stays within 2 VBench points, against 2.61/1.97 for KIVI and 7.03/4.33 for FlexGen.
• Finer sub-blocks help further, for free. Going below the default r=1 raises Self-Forcing INT4 VBench Quality from 80.99 to 81.34 at r=1/4, saturating at r=1/8; at r=1/8 gains range from +0.34 (Infinity-2B INT3) to +1.07 (InfinityStar INT3), with Infinity-8B INT2 at +0.88.
• Image and video AR sit on opposite sides of a K/V axis. Image AR is V-dominant (Infinity-2B at p=5%: ΔLPIPS 0.20 for V-only vs 0.08 for K-only), video AR is K-dominant (LongLive-30s at p=5%: ΔLPIPS 0.40 for K-only vs 0.14 for V-only). The damage ratio Δ_K/Δ_V at p=0.5% is at or below 1× for image AR, 2.5×–61× for video AR, and 320× for LLaMA-3-8B, whose WikiText-2 perplexity moves from 8.4 to 82.4 under K-only ablation and only to 11.5 under V-only.

Methodology in Plain English

The work proceeds in three stages. First, the authors ablate: they rank cache channels by maximum magnitude, zero the top few percent, and compare against zeroing an equally sized random set drawn from the remaining channels. If quality collapses only for the high-magnitude set, those channels are doing real work. Second, they attribute: they test whether outlier identities survive across prompts (architectural outliers should persist; input-driven ones should drift), and they rebuild the key projection chain piece by piece — W_K alone, then adding γ_K, then RoPE — measuring how much of the final outlier set each stage recovers. Third, they exploit: since massive values live at specific (channel, block) intersections, quantization groups are drawn at the same intersections, with block boundaries taken directly from the generation loop instead of detected or calibrated. A single knob, the sub-block ratio r, splits blocks further along the token axis. Because current CUDA toolchains have no sub-8-bit storage types, the authors add a Triton kernel that packs the low-bit values into byte-aligned containers on write and unpacks them during attention, so the nominal bit-width becomes an actual memory reduction.

Why This Matters

Research impact. Prior KV-quantization work (KIVI, FlexGen) was calibrated to language-model caches, which form a homogeneous token stream with no block structure. This paper shows that applying those recipes directly to visual AR collapses generation, and explains why: no existing rule protects more than one of the two axes. It also provides the first direct ablation evidence that visual-AR cache outliers are load-bearing, and traces them to named architectural components rather than describing them only statistically.

Real-world applications.

• Serving high-resolution text-to-image models such as Infinity-8B on a single GPU, where the cache is otherwise the binding memory constraint.
• Long-form video generation (30-second clips in LongLive) where the cache grows with horizon and would otherwise saturate device memory.
• Increasing serving throughput by admitting 6× more concurrent requests per device at the same memory budget.
• On-device or edge deployment where weight memory fits but cache memory does not.

Industry relevance. The method is tuning-free, requires no calibration dataset, and composes orthogonally with cache eviction and attention sparsification — properties that matter for production serving stacks where retraining or per-model calibration is expensive. The realized savings are measured on NVIDIA A100 80GB hardware with a shipping kernel implementation rather than being theoretical.

Future Directions

• Per-modality grouping. The appendix shows image AR is V-dominant and video AR is K-dominant, with video sitting much closer to the LLM regime. A quantizer that allocates precision asymmetrically across K and V by modality, rather than applying one unified rule, is an obvious extension the paper sets up but does not build.
• Joint use with eviction and sparsification. The authors state VAR-Q composes orthogonally with cache eviction and attention sparsification; the combined reduction from quantizing and dropping entries is not measured here.
• Pushing below INT2. The reported floor is INT2 on Infinity-8B; whether the two-axis structure supports more aggressive rates, and where the breakdown point lies for smaller backbones, is left open.
• Sub-block granularity as a tuning target. The r ablation shows consistent gains from finer sub-blocks and saturation at r=1/8 on Self-Forcing; an adaptive or per-model r schedule, and the interaction between r and block size in next-scale versus next-frame models, remain unexplored.

Target Audience

Systems and inference-efficiency researchers working on quantization, KV-cache management, and serving infrastructure; practitioners deploying visual autoregressive image or video models under tight GPU memory budgets; and anyone interested in how the statistical structure of transformer caches differs between language and visual generation — including readers who want a template for attributing cache outliers to specific architectural operations rather than treating them as a generic numerical nuisance.

Authors’ abstract

Visual autoregressive generation spans two dominant paradigms: next-scale prediction for images and video, and next-frame prediction for video. Both decode block-by-block rather than token-by-token, yielding KV caches whose statistics diverge from those of language models. At scale, KV memory exceeds the weights and limits large-scale deployment caused by a large amount of visual tokens. Our analysis of these caches reveals three findings. First, a small set of massive-value…