ForcingBooster: Adaptive Patch-wise Denoising for Fast Autoregressive Video Diffusion

AI summary

Overview

Research area: Computer Vision — efficient generative modeling, specifically autoregressive video diffusion and interactive video world models.

Technical level: Intermediate. The paper assumes familiarity with diffusion/flow-matching sampling, latent patches, KV caching, and few-step distillation, though its core idea can be grasped without the math.

Scope: A framework that learns a lightweight per-patch router to stop denoising early on easy regions of each video chunk, speeding up existing few-step autoregressive video generators without retraining them.

What This Paper Is About

Autoregressive video diffusion models generate video chunk by chunk, and recent distillation methods have compressed each chunk down to four or fewer denoising steps. However, those steps are applied uniformly to every spatial patch of a chunk, even though static backgrounds, recurring content, and simple regions often converge much earlier than moving objects or newly revealed detail. The paper asks whether denoising effort can be allocated per patch instead of per chunk, so that easy regions exit early while hard regions keep refining, yielding faster generation at roughly the same visual quality.

Key Contributions

1. A patch-wise adaptive denoising router. ForcingBooster attaches a small router to a frozen few-step autoregressive generator. After each denoising step it predicts a stop probability per latent patch from the generator's own last-layer hidden features, and patches above a threshold exit and contribute their current clean-latent estimate as output.

2. Early exit grounded in clean-latent prediction. The method exploits the fact that diffusion and flow-matching models admit a clean-latent estimate (x̂₀) at every intermediate step, so a patch that stops early still has a valid final latent without altering the original sampling schedule.

3. Supervision without labels or extra data. Training uses a stopping surrogate loss (penalizing high stop probability when the prediction deviates from the full-step result) plus a denoising-cost loss, estimated with on-policy rollouts. The full-step outputs of the frozen generator serve as targets, so no additional video data or manual stopping annotations are needed.

4. Sparse inference that preserves context. Stopped patches are removed from subsequent Transformer updates once their KV cache is built, so later steps compute only active patches while stopped ones still supply context to active patches and later chunks.

Main Findings

• Patches converge at very different rates. In a controlled experiment with Causal Forcing as the base model (three latent frames per target chunk, four-step sampling, 35 dB PSNR threshold against the four-step output), 30.60% of patches reached the threshold after the first step, 12.25% at the second, 9.69% at the third, and the remaining 47.45% did not reach it within the first three steps.

• Roughly half the denoising work is removable. On the Causal Forcing 1.3B checkpoint (81 frames, 480×832, 16 FPS), ForcingBooster cut total denoising steps applied across patches by 50.19% and achieved a 1.30× end-to-end speedup (8.56 s versus 11.15 s for the four-step baseline), with VBench Quality 83.88 versus 84.54, Semantic 72.20 versus 72.21, and Total 81.55 versus 82.08.

• It beats uniform step reduction at matched runtime. At a runtime near uniform two-step sampling (8.67 s, Total 81.29), ForcingBooster scored higher on all three VBench aggregates (Quality 83.88, Semantic 72.20, Total 81.55). Uniform one-, two-, and three-step sampling also showed visible artifacts in complex texture regions that ForcingBooster avoided.

• Gains transfer across models and scales. On Wan2.1 14B, ForcingBooster reached Total 83.32 at 1.31× speedup versus the four-step baseline's 83.57; on LingBot-World-V2 14B, 78.54 at 1.25× versus 79.12; on LongLive, 78.58 at 1.15× versus 79.01. In every case it scored higher than uniform one-step sampling.

• Where patches exit matters, not just how many. Against a "Random" baseline that shuffles which patches get which exit step (Total 79.03, 1.29×) and a chunk-wise router that applies one uniform step count per chunk (Total 80.01, 1.33×), ForcingBooster scored 81.54 at 1.30×. Random showed flickering and detail loss; the chunk-wise router produced blurrier frames in low-step chunks.

• A cost penalty is required, and tuning is forgiving in a range. With zero or insufficient weight on the computational cost term, the router keeps all four steps and adds overhead. Weights between 0.015 and 0.030 gave substantial time savings with low LPIPS; larger weights gave smaller runtime gains and larger deviations from the four-step output.

• More conservative stopping does not mean better fidelity. Training separate routers with thresholds from 0.1 to 0.9 (at cost weight 0.030), higher thresholds generally increased runtime but did not consistently lower LPIPS against the four-step reference.

• Low-motion video benefits most. Grouping videos into five equal buckets by RAFT optical-flow magnitude, speedup approached that of one-step denoising on the lowest-motion bucket, with up to roughly 1.5× speedup, or about 75% reduction in patch-level denoising steps over the four-step baseline.

• Training is cheap. Roughly 500 router training steps with AdamW, learning rate 10⁻⁴, global batch size 8, while the pretrained generator and denoising schedule stay fixed.

Methodology in Plain English

The starting point is an existing autoregressive video generator that has already been distilled to very few denoising steps. The researchers freeze that model entirely and add a small side network — the router — that watches the generator's internal features.

At each denoising step, the router outputs, for every latent patch, a probability of stopping. Patches whose probability falls at or below a threshold are deactivated: their current best guess at the clean latent becomes their final output, and their key-value cache is kept so the still-active patches and future chunks can attend to them. Active patches continue through the remaining steps. Because a diffusion or flow-matching model already produces a clean-latent estimate at each step, no special schedule surgery is needed for the patches that quit early.

The tricky part is supervision, since nobody knows the "correct" stopping step for each patch. The authors sidestep this with two losses. One penalizes a high stop probability in proportion to how far the patch's current clean-latent prediction is from the fully denoised target. The other penalizes expected denoising cost, crediting a stop now with cost equal to the current step and continuing with the step at which the patch would eventually exit under the current policy. To keep that cost estimate honest, they run rollouts with the current router as training proceeds. The targets themselves come from the frozen generator: they run the full four-step schedule, record the trajectory, and replay noisy states back through the model to obtain intermediate predictions and hidden features. Training the router thus requires no new video data and no stopping labels.

Evaluation uses VBench's 16 fine-grained dimensions aggregated into Quality, Semantic, and Total scores, plus wall-clock end-to-end generation time on a single A800 GPU in BF16 at batch size 1, including text encoding, first chunk, router execution, KV cache construction, and VAE decoding.

Why This Matters

For interactive video generation and video world models, per-frame latency is the binding constraint: a system that cannot respond within a fraction of a second cannot support real interaction. This paper shows that even after aggressive step distillation, substantial redundant computation remains inside each chunk, and it can be removed spatially rather than temporally — a direction largely orthogonal to distillation, caching, and history compression, and therefore composable with them. Its practical appeal is that it requires no generator retraining, no extra data, and only a few hundred training steps, so it can be bolted onto existing checkpoints.

Real-world applications:

• Interactive game and simulation engines that need to render plausible next frames conditioned on user input at low latency, including camera-controlled navigation through generated scenes.
• Real-time video world models for robotics and autonomous systems, where a model must predict how a scene evolves in response to actions.
• Live content creation and streaming tools for previewing generated video while a creator adjusts prompts or camera paths.
• Edge and cost-sensitive deployment of video generation, where per-clip GPU time directly determines serving cost.

Industry relevance: the method's profile — a plug-in module, 500 training steps, a frozen backbone, and a 1.15×–1.31× end-to-end speedup across four different base models — matches how production video generation teams prefer to optimize: incremental, low-risk improvements to an already-deployed model rather than retraining from scratch.

Future Directions

• Joint optimization of router and generator. The router is trained on top of a frozen model; whether co-training or distilling with awareness of the routing policy could push speedups beyond the reported range is untested.
• Routing the first chunk. All reported experiments use four steps for the first chunk and apply the strategy only to subsequent chunks, so extending patch-wise allocation to the first chunk remains open.
• Combining with history compression and caching. The paper positions itself against history-compression and cache-reuse methods but does not measure the compounded savings of stacking them with patch-wise exit.
• Hardware-aware exploitation of sparsity. Removing patches from Transformer updates yields irregular active sets; converting this into realized wall-clock gains more reliably — and understanding why more conservative thresholds did not consistently reduce LPIPS — would strengthen the approach's practical ceiling.

Target Audience

Researchers and engineers working on efficient diffusion sampling, autoregressive video generation, and real-time world models will get the most from this paper, particularly those who already have a distilled few-step checkpoint and want additional speed without touching the backbone. It is also useful for graduate students studying adaptive computation, token-level or region-level sparsity, or on-policy training schemes for diffusion models, and for practitioners deciding between uniform step reduction and spatially adaptive budgets under a latency constraint.

Authors’ abstract

Autoregressive video diffusion models have become a leading approach to real-time interactive video generation and video world modeling. However, existing methods typically allocate computation uniformly across space when generating the next video chunk, updating all patches with the same denoising schedule and number of function evaluations. This ignores the spatial heterogeneity of video content: static, simple, or recurring regions are often easier to denoise, while other regions require further refinement or additional context. A uniform budget thus leads to redundant computation on easy patches, whereas reducing the fixed step count can leave difficult patches insufficiently refined. We introduce \methodname{}, a general framework that dynamically allocates denoising steps to individual patches: easy patches stop early to improve efficiency, while difficult patches receive more refinement steps. This design accelerates generation with negligible quality degradation. Our method is compatible with most existing few-step autoregressive diffusion models, requiring only a lightweight routing module with modest training. Experiments demonstrate that \methodname{} reduces the total number of denoising steps applied across video patches by about 50\% while maintaining visual quality comparable to the four-step baseline.