In-Distribution Forcing for Long Video Generation at Test Time

AI summary

Overview

Research area: Computer vision / generative video — specifically autoregressive (AR) video diffusion models and test-time methods for extending them to long horizons.

Technical level: Advanced. The paper assumes familiarity with diffusion denoising schedules, autoregressive chunked generation, key-value (KV) caching and conditioning in transformers, and rotary position embeddings (RoPE).

Scope: The paper identifies a failure mode it calls the "KV-provenance problem" in autoregressive video diffusion and proposes a training-free, test-time framework, In-Distribution Forcing (ID-Forcing), that keeps both KV caching and KV conditioning inside the configurations seen during training to suppress drift in minute-scale video generation.

What This Paper Is About

Autoregressive video diffusion models generate video chunk by chunk, conditioning each new chunk on a key-value (KV) cache of earlier chunks. When generation runs past the horizon the model was trained on, colors and textures shift and motion decays — a degradation called drifting. The paper's core claim is that existing fixes only modify which cached entries the model reads (KV conditioning), while ignoring how those cached entries were written in the first place (KV caching), and that this omission is the actual source of drift beyond the training horizon. The goal is a simple, training-free rule that keeps every KV operation at test time within what training executed, extending a five-second model to minute-scale generation.

Key Contributions

1. Characterizing the KV-provenance problem. The authors argue that whether a KV entry is in-distribution depends not only on its own chunk but on the other KV entries it attended to when it was cached — its provenance. Beyond the training horizon, every chunk is cached under a provenance never seen in training, producing out-of-distribution (OOD) KV entries. Because provenance is fixed at caching time, no KV conditioning policy alone can recover from it.

2. In-Distribution Forcing (ID-Forcing), a two-level test-time framework. Level 1 (KV caching) introduces self-caching: each chunk is cached without attending to any prior KV entry, so the rolling window stays exactly in-distribution. Level 2 (KV conditioning) keeps the first chunk κ₀ permanently in the window and bounds the window to length L < N, with the remaining entries re-rotated by R₋_f after each eviction.

3. Demonstrating drift suppression at minute scale. Evaluations on 120- and 240-second videos across two base models (Self-Forcing and LongLive) show large gains on the paper's color drift and motion drift metrics, plus a user study in which participants preferred the method over every baseline on every measured dimension.

4. An ablation isolating each component. Adding the first-chunk sink, then re-rotation, then self-caching cumulatively improves results, with self-caching producing the largest reduction in residual drift.

Main Findings

• KV conditioning alone is insufficient. Prior test-time methods — Deep Forcing, ∞-RoPE, and MemRoPE — all act on KV conditioning and leave untouched how each KV entry was written. The paper attributes the residual drifting they leave behind to this unaddressed caching-level OOD failure.

• Self-caching is the decisive component. In the 120-second ablation on the Self-Forcing base model, plain Self-Forcing scores 25.54 color drift and 46.88 motion drift, with Dynamic Degree 30.04. Adding the sink raises these to 48.65 / 67.19 (Dynamic Degree 46.90); adding re-rotation reaches 60.66 / 87.50 (Dynamic Degree 65.44); the full ID-Forcing reaches 72.37 color drift and 89.06 motion drift with Dynamic Degree 65.65 and Aesthetic Quality rising from 54.14 to 58.23. The authors note that Color Drift stays at 60.66 after re-rotation because KV caching is still out of distribution.

• Best drift metrics across all main settings. At 120 seconds on Self-Forcing, ID-Forcing records 72.37 color drift and 89.06 motion drift versus 59.65 / 71.88 for Deep Forcing, 54.15 / 85.16 for ∞-RoPE, and 59.25 / 82.81 for MemRoPE. At 240 seconds on Self-Forcing: 70.34 / 92.19 versus 62.52 / 88.28 (Deep Forcing), 55.49 / 87.50 (∞-RoPE), 61.41 / 85.16 (MemRoPE). On LongLive at 240 seconds: 71.76 / 96.09 versus 57.08 / 89.06, 56.72 / 94.53, and 64.23 / 96.88 respectively.

• Highest Dynamic Degree in every setting. The paper reports that ID-Forcing achieves the top Dynamic Degree score across all evaluated configurations, whereas base models such as Self-Forcing and LongLive show markedly lower dynamic degree that falls further with duration — Self-Forcing's Dynamic Degree drops from 44.58 at 30 seconds to 29.61 at 240 seconds, while ID-Forcing moves only from 65.94 to 64.76 over the same range.

• Competitive on standard quality metrics, with one stated exception. ID-Forcing also scores well on Aesthetic Quality, Imaging Quality, and Subject Consistency. The exception is Motion Smoothness, which the authors argue is inherently biased toward methods with lower dynamic degree because a collapsed, near-frozen video produces nearly identical consecutive frames that the metric rewards. They make the same argument about Background Consistency, where Self-Forcing's higher score "reflects its collapse rather than its quality."

• User study favors the method on all dimensions. Across 6 dimensions and 4 baselines, preference rates for ID-Forcing were: versus Self-Forcing 100.0 (color consistency), 99.0 (background consistency), 96.0 (subject consistency), 93.0 (dynamic degree), 98.0 (temporal flickering), 100.0 (overall preference); versus Deep Forcing 90.0 / 84.0 / 85.0 / 81.0 / 97.0 / 95.0; versus ∞-RoPE 91.0 / 90.0 / 87.0 / 75.0 / 91.0 / 93.0; versus MemRoPE 88.0 / 82.0 / 78.0 / 59.0 / 89.0 / 86.0.

• The first chunk is distributionally irreplaceable. The authors argue this is not an attention sink in the usual sense: the causal 3D-VAE encodes the initial latent frame from a single pixel frame, while every subsequent frame aggregates multiple frames. Attention sinks (such as in Deep Forcing) retain several early entries for their empirical stabilizing effect; ID-Forcing retains exactly one, for a distributional reason.

• Extending the training horizon is not a solution. The paper argues that past-horizon extrapolation is unavoidable for arbitrarily long video, that training always covers a fixed horizon, and that the cost of extending that horizon grows with target length — so a test-time extrapolation rule is needed regardless of how the backbone was trained.

Methodology in Plain English

The authors start from the observation that in autoregressive video diffusion there are two distinct places a model touches its KV cache: reading from it to generate a new chunk (KV conditioning) and writing a new entry after a chunk is produced (KV caching). Training uses the full prefix κ_{<i} for both operations, and the window grows from 0 to N−1 entries. At test time, standard rolling-window methods fix both window sizes to L < N and evict the oldest entry, so both operations now see the L most recent entries instead of the full prefix — a configuration training never executed.

Their fix has two rules:

• Self-caching (write side). When a new chunk is cached, it is not allowed to attend to the prior context. Concretely, for a conditioning window of L entries, the first ℓ entries are cached attending only to themselves, E(x_j; ∅), and the remaining L−ℓ entries are cached autoregressively on top of the last self-cached entry — mirroring training, where κ₁, κ₂, … are cached on top of κ₀. A self-cached entry carries no external provenance, so eviction later cannot leave it outside the training distribution.

• Conditioning with a fixed sink and exact rolling (read side). The window is held at length L < N with κ₀ pinned at slot 0 and the L−1 most recent entries in the remaining slots. After each chunk, the oldest non-sink entry is evicted and the survivors are re-rotated by R₋_f, so every temporal distance inside the window matches a state seen in training and the total window size stays within the training budget N−1.

Implementation details as reported: the method is built on Wan2.1-T2V-1.3B and applied to two base models, Self-Forcing and LongLive. Each video chunk is 3 latent frames, generated with a 4-step denoising schedule at timesteps {1000, 750, 500, 250}. For Self-Forcing, L = 3 and ℓ = 2; for LongLive, which is trained on minute-long videos and so has the method applied from the first chunk, L = 3 and ℓ = 3. In both cases ℓ counts the sink κ₀. Evaluation generates 120- and 240-second videos at 480×832 using prompts from MovieGen, taking the first 128 prompts refined with Qwen2.5-7B-Instruct, and reports VBench-Long metrics plus two drift-specific metrics. The user study used 20 participants comparing against 4 Self-Forcing-based baselines over 20 diverse prompts in a 2AFC protocol, with a balanced Latin-square design for baseline assignment.

The drift metrics are defined in the appendix: color drift compares L₁-normalized 180-bin Hue histograms of the first and last frames in HSV space, and motion drift is the absolute difference between Dynamic Degree computed over the first 5 seconds and over the last 5 seconds. Both are calibrated to a 0–100 scale (higher is better) and averaged over all 128 prompts.

Why This Matters

Impact on research. The paper reframes drifting from a "we are discarding useful context" problem to a distributional one, and locates the failure at the caching step rather than the attention step. If the argument holds, it changes where researchers look for fixes: it implies that window-selection and position-reassignment tricks operate downstream of a problem they cannot repair on their own, and that the two research directions — longer-context training and test-time extrapolation rules — are orthogonal, so progress in one does not obviate the other.

Real-world applications:

• World models and interactive simulation, where a model must keep simulating a consistent environment indefinitely while a user acts inside it.
• Streaming and live content generation, where output must be produced continuously rather than as a fixed-length clip.
• Long-form storytelling and narrative video, where a single generated sequence must hold its subject, palette, and motion across minutes.
• Any deployment that extends a short-horizon model past its trained duration, since the method is training-free and applied only at inference.

Industry relevance. The method requires no retraining or fine-tuning, which matters for teams that already have a short-horizon AR video model in production and want longer output without a new training run. Because it preserves active motion rather than stabilizing the video by freezing it, it targets the failure mode — a static, collapsing video — that is most visible to end users and least rewarded by frame-similarity metrics.

Future Directions

The paper does not include an explicit future-work section, so the following are logical extensions of what it reports rather than stated plans:

• Testing whether self-caching transfers to backbones other than the two evaluated. The method is demonstrated on Self-Forcing and LongLive over Wan2.1-T2V-1.3B; whether the same ℓ and L settings hold for other architectures or larger backbones is not reported.

• Reconciling the metric conflict. The paper notes that Motion Smoothness and Background Consistency reward the collapsed output of the base model. Since the reported drift metrics are the paper's own definitions, third-party validation of those metrics across methods would clarify how much the metric choice drives the reported advantage.

• Combining with longer-context training. The authors argue the two directions are orthogonal and that a better-trained backbone still needs a rule for beyond-horizon generation. An empirical test of the combination — a longer-trained backbone plus ID-Forcing — is a natural next step.

• Understanding whether more than one sink entry is ever distributionally justified. The paper keeps exactly one entry (κ₀) on distributional grounds and contrasts this with attention sinks that retain several. The question of whether other chunks have similarly unique distributional status is left open.

Target Audience

Researchers and engineers working on video diffusion and autoregressive sequence generation, particularly those concerned with long-horizon stability, KV-cache management, or exposure bias. It will be most useful to readers already comfortable with transformer attention internals and diffusion sampling, since the argument turns on the distinction between how a cache entry is written and how it is read. Practitioners deploying short-horizon AR video models who need longer output without retraining are a secondary audience, as are those studying position-embedding extrapolation, given the paper's re-rotation rule.

Authors’ abstract

Modern autoregressive (AR) video diffusion models excel at short-horizon video generation, yet generating long videos remains challenging due to drifting, where colors and textures shift, and motion dynamics decay. Existing works primarily rely on KV conditioning, which selects or modifies cached key-value (KV) entries to mitigate drifting. However, we observe that KV conditioning alone is insufficient as it assumes cached KV entries remain in-distribution. This assumption fails beyond the training horizon: nothing constrains the construction of KV entries during rollout, giving rise to the KV-provenance problem where cached entries themselves become out-of-distribution (OOD). To address this, we propose In-Distribution Forcing (ID-Forcing), a test-time framework that aligns both KV caching and KV conditioning with training configurations. Its key mechanism, self-caching, prevents OOD KV entries at their source. Each chunk is cached without attending to prior KV entry, keeping the rolling window exactly in-distribution. Consequently, ID-Forcing seamlessly extends short-horizon models to minute-scale video generation. Extensive evaluations show that our method remains competitive on standard video generation benchmark while substantially outperforming prior work in mitigating drifting, as validated by both our drift metrics and a user study.