DeCoPrune: Efficient KV-Cache Pruning for Autoregressive Video Diffusion via Denoising Consistency

AI summary

Overview

Research area: Efficient inference for autoregressive video diffusion models, specifically compression of the key–value (KV) cache used to condition each newly generated video chunk on prior history.

Technical level: Advanced. The paper assumes familiarity with diffusion denoising trajectories, transformer attention, KV caching, and rotary position embeddings (RoPE).

Scope: The paper proposes DeCoPrune, a training-free KV-cache pruning rule driven by "denoising consistency," introduces the CMBench benchmark for long-range visual recall, and reports experiments on the LingBot World v2 backbone showing near-FullKV recall at over 85% historical-token pruning.

What This Paper Is About

Autoregressive video diffusion generates video chunk by chunk, with each new chunk attending to the KV cache of everything generated before it. That cache grows without bound, so memory and attention cost rise steadily with rollout length, yet simply truncating old history destroys the visual evidence needed to stay consistent over long videos. The paper asks how to decide which historical tokens are genuinely redundant and which carry information the current chunk cannot already predict, and answers with a pruning signal derived from the model's own denoising process.

Key Contributions

1. DeCoPrune, a training-free KV-cache pruning method that scores each current-chunk token by the discrepancy between its intermediate clean prediction and the final denoised output from the same trajectory. High-discrepancy tokens are retained in a long-term cache; low-discrepancy tokens are pruned. A shared mask physically compacts KV entries across layers rather than merely masking attention weights.
2. CMBench (Context Memory Benchmark), a reference-grounded benchmark comprising 58 approximately one-minute generated or real-world context episodes and 116 Reappear or Revisit continuation tasks that require recalling a specific previously observed object, person, or scene.
3. Supporting implementation components: prefix pruning for already-observed video contexts (which lack a denoising trajectory), RoPE re-indexing of retained history into a fixed virtual span, and an optional head-specialized variant, DeCoPrune–HS, that applies DeCoPrune to dynamic heads and a streaming policy to static heads.
4. Experiments on LingBot World v2 showing near-FullKV long-range recall while pruning over 85% of historical KV tokens and accelerating continuation generation by over 4x, substantially outperforming the evaluated compression baselines at comparable budgets.

Main Findings

• Main DeCoPrune result: 0.6701 DINO on CMBench at an 85.43% pruning ratio, 6.489 FPS, and a 4.14x continuation-generation speedup over FullKV.
• Head-specialized variant: DeCoPrune–HS reaches 0.6783 DINO at an 86.19% pruning ratio and 4.09x speedup, a 0.0020 DINO gap to FullKV's 0.6803.
• Uncompressed reference: FullKV scores 0.6803 DINO at 0.00% pruning ratio and 1.568 FPS (1.00x).
• Comparison to temporal-difference selection: At similar pruning ratio and throughput, DeCoPrune exceeds TempDiff (0.6229 DINO, 86.36% PR, 4.11x) by 0.0472 DINO.
• Aggressive-discard baselines lose recall: Streaming scores 0.4592 DINO at 94.37% PR (6.12x speedup) and DummyForcing 0.4461 DINO at 98.03% PR (4.72x); ForcingKV reaches 0.5313 DINO at 80.62% PR (3.37x).
• Selection direction matters: At nearly matched pruning ratios (85.31–85.43%), DeCoPrune achieves 0.6701 DINO versus 0.6091 for Random and 0.4512 for the reverse criterion, a 0.2189 DINO gap to reverse selection. Reverse selection retaining 81.21% of historical entries (PR = 18.79%) still reaches only 0.5621 DINO.
• RoPE re-indexing helps methods that keep distant history: It raises DINO by +0.2418 for FullKV, +0.0470 for DeCoPrune–HS, +0.0346 for TempDiff, and +0.1208 for Random; Streaming and the two head-wise policies change little (−0.0046, −0.0007, −0.0035 respectively).
• Denoising consistency as an empirical signal: Across four backbones, the with-context condition shows lower step-to-final MSE than the without-context condition, supporting the premise that predictable tokens are redundant.
• Probe robustness: Figure 6(a), on 13 independent cases, shows similar DINO across probe timesteps over a broad pruning range, with degradation only under aggressive compression.
• Quality metrics are insensitive to recall loss: VBench subject and background consistency remain nearly saturated while CMBench DINO responds to lost contextual information. The four reported VBench dimensions remain broadly comparable across policies (for example, DeCoPrune temporal 0.9424, motion 0.9650, aesthetic 0.4736, image quality 0.7067 versus FullKV 0.9499, 0.9705, 0.4633, 0.7125).
• Second backbone: On LongLive with 10-second contexts over 18 cases, DeCoPrune achieves 0.5880 DINO at 57.25% PR, beating Streaming (0.5162 at 53.57%), ForcingKV (0.5302 at 54.17%), and DummyForcing (0.5157 at 55.60%) against FullKV's 0.6106 at 0.00% PR.
• Backbone suitability: Full-context DINO on the completed subset is 0.6803 for LingBot World v2, versus 0.2627 (Causal Forcing), 0.2609 (Self-Forcing), and 0.2608 (LongLive), which the authors cite as the reason for choosing LingBot World v2 as the main backbone.
• Synthetic versus real contexts: In a single-seed, source-stratified analysis, DeCoPrune scores 0.6909 on synthetic and 0.6412 on real videos, versus FullKV at 0.6873 and 0.6621, and TempDiff at 0.6484 and 0.5212. Real videos are harder for all methods.

Methodology in Plain English

The core idea is to let the model tell you which tokens it actually needed. When the model denoises a new chunk of video, it makes an intermediate guess at the clean output partway through the denoising schedule, and then a final cleaned-up result. The authors measure, per token, the squared difference between that intermediate guess and the final result. If the intermediate guess was already close to the final output, the token was largely predictable from the retained context and its cached entries are treated as redundant. If the two differ a lot, the token carried information the context did not explain, so its KV entry is kept.

Concretely, for each chunk, per-token mean squared discrepancy values are thresholded (γ = 0.10) into a binary retention mask. The mask is shared across layers and physically gathers the selected entries, shrinking the KV sequence rather than only masking attention weights. Scoring reuses predictions from the normal denoising trajectory, so no extra model evaluations or training are needed. The initial sink chunk and a recent window (w = 1 chunk) are kept dense, and masks are stored until a chunk leaves that window and its compaction is applied.

For observed video contexts that have no denoising trajectory (for example, a provided prefix video), the authors re-noise each finalized chunk and compare its context-conditioned clean prediction against the observed clean chunk, processing chunks sequentially to build a compressed cache.

Two supporting pieces address side effects. RoPE re-indexing compresses the temporal rotary coordinates of older retained keys into a fixed virtual span (L_virt = 17 latent frames, with the newest N_r = 8 frames left at their true positions), applying a phase correction to the cached keys; spatial RoPE is untouched. Head specialization splits attention heads offline from 15 calibration videos using a recent-attention ratio threshold of 0.8, routing dynamic heads to DeCoPrune and static heads to a streaming cache over the first four and most recent four context latent frames; layer 0 uses DeCoPrune for all 40 heads, leaving 232 static and 1368 dynamic layer–heads in the 40-layer, 40-head model.

Evaluation uses CMBench, where each task pairs a continuation prompt with an annotated reference frame and target bounding box. OWL-ViT localizes the target in generated frames, SAM 2 segments reference and generated targets, and DINOv2 embeddings are compared by cosine similarity, taking the maximum over continuation frames (zero if the target is never detected). Efficiency is reported as a sequence-level pruning ratio aggregating historical KV counts over chunks, layers, and heads, plus FPS and speedup measured on continuation generation only, excluding prefix processing.

Why This Matters

The paper's main argument is that a model-intrinsic signal, the change between an intermediate and final denoising prediction, can identify which cached history is worth keeping, without training an importance predictor, building a retrieval index, or running a motion estimator. It also argues that general video-quality metrics can stay nearly unchanged while long-range recall degrades, which is why a targeted benchmark such as CMBench is needed.

Real-world applications the work points toward:

• Streaming and interactive video generation, where a user keeps prompting or controlling a world model and expects earlier objects, people, or scenes to still exist.
• Long-form generated content such as minute-scale or longer sequences, where holding the entire KV history becomes impractical.
• Evaluating and comparing memory policies for video generators, since CMBench provides reference-grounded Reappear and Revisit tasks under controlled historical-KV compression.
• Cost reduction in deployment, since the reported 4.14x continuation-generation speedup over FullKV on four NVIDIA H200 GPUs translates into fewer GPU-hours for the same rollout.

Industry relevance: the method is training-free and applied to a frozen generator's native cache, which makes it plausible as a drop-in inference-time optimization for existing autoregressive video systems rather than a retraining project. The reported speedups target the continuation phase, which is where autoregressive rollouts accumulate cost.

Future Directions

• Bounded memory. The authors state that retained cache still grows with generation length, so memory and attention cost are not strictly bounded; they call for bounded-memory extensions.
• Separating compression error from backbone error. Few open-source baselines support reliable long-context video generation, and even LingBot World v2 has a measurable intrinsic ceiling on context consistency, making it hard to isolate pruning-induced degradation.
• Task- or query-aware retention. Denoising consistency is future-agnostic: it does not encode relevance to a future prompt and may undervalue a rare detail that is easy to denoise now but requested later, motivating combinations with query- or task-aware memory selection.
• Broader backbone validation. The main experiments use LingBot World v2, with an additional LongLive test over 18 cases on 10-second contexts; extending evaluation to more capable long-context backbones remains open.

Target Audience

Researchers and engineers working on video diffusion, autoregressive generation, and inference efficiency, particularly those concerned with KV-cache compression, long-context memory, or streaming video generation. It is also relevant to practitioners building interactive world models or streaming video systems who need to reduce inference cost without losing long-range consistency, and to benchmark designers interested in reference-grounded recall evaluation rather than general video-quality scoring. The project homepage is https://decoprune.github.io, with code at https://github.com/DeCoPrune/CMBench and the benchmark at https://huggingface.co/datasets/Aoraku/CMBench.

Authors’ abstract

Autoregressive video diffusion supports streaming generation and interactive control, but its KV cache grows continuously with the generated history. Existing compression strategies either discard history using fixed windows or select tokens through local attention and similarity signals, which do not directly measure whether the current chunk contributes information beyond the retained context. We introduce DeCoPrune, a training-free method that treats cache compression as a denoising-consistency problem. We find empirically that denoising difficulty provides a useful proxy for a token's value in long-term retention: tokens with larger step-to-final discrepancies tend to carry visual evidence that is less predictable from the retained context. DeCoPrune measures each current-chunk token's denoising difficulty using the discrepancy between its intermediate clean prediction and final denoised value, retaining high-discrepancy tokens in the long-term cache while pruning those with low discrepancy. To evaluate information retention, we introduce CMBench, comprising 58 approximately one-minute generated or real-world context episodes and 116 Reappear or Revisit continuation tasks that require recalling specific previously observed objects or scenes. Experiments with LingBot World v2 show that DeCoPrune preserves near-FullKV long-range recall while pruning over 85% of historical KV tokens and accelerating continuation generation by over $4\times$, substantially outperforming the evaluated compression baselines at comparable budgets. These results indicate that denoising consistency can serve as a model-intrinsic signal for retaining long-range information while reducing autoregressive inference cost. Our project homepage is https://decoprune.github.io. The code is available at https://github.com/DeCoPrune/CMBench, and the benchmark at https://huggingface.co/datasets/Aoraku/CMBench.