FilmWeaver: Weaving Consistent Multi-Shot Videos with Cache-Guided Autoregressive Diffusion

AI summary

Overview

Research area: Computer Vision — video diffusion models, specifically multi-shot and long-form video generation.

Technical level: Advanced. The paper assumes familiarity with diffusion models, DiT/MM-DiT architectures, VAE latents, CLIP embeddings, in-context conditioning, and autoregressive generation.

Scope: A cache-guided autoregressive diffusion framework that generates multi-shot videos of arbitrary length and shot count by explicitly separating inter-shot consistency from intra-shot coherence.

What This Paper Is About

Existing video generation models work well for single shots but break down on multi-shot videos: they cannot reliably keep the same characters and backgrounds looking consistent across shots, and they cannot flexibly control how long each shot lasts or how many shots a video contains. FilmWeaver's goal is to generate multi-shot videos of arbitrary length and shot count while preserving character identity, background style, and smooth motion, without modifying the underlying model architecture.

Key Contributions

1. A cache-guided autoregressive framework with a dual-level cache. A Shot Cache stores keyframes retrieved from prior shots to maintain long-term character and scene identity, while a Temporal Cache holds recent frames of the current shot to maintain short-term motion coherence. Both are injected in-context, requiring no architectural changes to the base model.
2. Broad downstream flexibility. Because of the decoupled cache design, the framework supports multi-concept character injection, interactive video extension, and mid-sequence narrative control by swapping prompts while keeping the Temporal Cache.
3. A multi-shot data curation pipeline. A method combining shot segmentation, sliding-window CLIP-based scene clustering, filtering, and group captioning with Gemini 2.5 Pro to build a high-quality, consistently annotated multi-shot video dataset.
4. Evaluation metrics and baselines. Since no suitable public benchmark existed for text-to-multi-shot video generation, the authors built a test set and defined identity consistency, background consistency, visual quality, and text-alignment metrics to compare against prior methods.

Main Findings

• Consistency gains over prior methods. On the paper's test setup, FilmWeaver scored 74.61 for character consistency and 75.12 for overall consistency, versus 73.34 / 62.40 for VideoStudio, 70.03 / 67.15 for StoryDiffusion, and 72.47 / 71.19 for IC-LoRA.
• Highest Inception Score among compared methods. FilmWeaver reported 8.57 (Incep.), above VideoStudio's 6.81, StoryDiffusion's 8.30, and IC-LoRA's 6.95. Its Aesthetics Score was 33.69, above VideoStudio (32.02) and IC-LoRA (31.78), but below StoryDiffusion (35.61).
• Best character-level text alignment. FilmWeaver reached 23.07 (Char. Align.), compared with 20.88 (VideoStudio), 20.21 (StoryDiffusion), and 22.16 (IC-LoRA). Overall text alignment was 31.23, slightly below StoryDiffusion's 30.86... actually above it, but below VideoStudio's 31.52.
• Both caches matter. Removing the Shot Cache (w/o S) dropped character consistency to 68.11 and overall consistency to 65.44. Removing the Temporal Cache (w/o T) dropped them to 70.79 and 70.57. The full model reached 74.61 and 75.12.
• Noise augmentation improves prompt adherence. Removing noise augmentation (w/o A) yielded the strongest consistency scores (72.36 Char., 75.92 All.) but the weakest text alignment (21.88 Char., 28.12 All.), which the authors attribute to a "copy-paste" over-reliance on visual context that harms prompt following.
• Shot Cache size of 3 is sufficient. The authors set K=3 based on a trade-off analysis, arguing that more keyframes yield minimal quality gains while raising compute and training complexity.
• Scale of qualitative claims. The paper demonstrates an 8-shot continuous narrative for its Scene 2 example and a video-extension case ("Extension of Shot 6") that continues a single shot using a second prompt, transitioning an athlete from "snowboarding on snowy slopes" to "surfing on ocean waves."
• Fault tolerance. Because of negative sampling during training, the model can ignore an irrelevant retrieved keyframe and still follow the text prompt rather than copying the wrong visual content.
• Computational efficiency. For generating 21 latents, a baseline processing the full sequence at once with a 3-keyframe Shot Cache processes n = 24, with attention cost proportional to 24² = 576. FilmWeaver generates in roughly 3.5 chunks over a fixed context of about 11 latents, costing approximately 3.5 × 11² = 423.5.

Methodology in Plain English

FilmWeaver builds on the HunyuanVideo model and generates video chunk by chunk autoregressively rather than all at once. Each new chunk is conditioned on three things: the text prompt, a Temporal Cache, and a Shot Cache.

The Temporal Cache is a sliding window of recently generated frames from the current shot. Because storing all past frames would be too costly, frames are stored at three fidelity tiers: the most recent latent is kept uncompressed, the next two are compressed 4x, and the final 16 are compressed 32x. A single autoregressive step produces 6 latents, equal to 24 video frames, so at 24 FPS each shot can be controlled in one-second increments.

The Shot Cache is built when starting a new shot. The system compares the CLIP embedding of the new text prompt against CLIP embeddings of candidate keyframes from previous shots and keeps the top-K (K=3) most similar ones. Both caches are injected via in-context conditioning, so the base diffusion architecture is untouched. All caches store VAE latents, not raw pixels.

Generation follows four modes depending on cache state: No Cache (first shot, plain text-to-video), Temporal Only (extending the current shot), Shot Only (starting a new shot with long-term identity carried over), and Full Cache (extending a new shot with both contexts active).

Training is done in two stages: first, the Shot Cache inputs are zeroed out and the model learns long single-shot video generation using only the Temporal Cache; second, the Shot Cache is activated and the model is fine-tuned on a mixed curriculum covering all four inference scenarios. Each stage runs for 10K steps with batch size 16. To prevent the model from simply copying its visual context, the authors apply negative sampling (randomly inserting irrelevant keyframes into the Shot Cache) and asymmetric noise — a large noise level equivalent to random diffusion timesteps between 100 and 400 on the Shot Cache, and a milder 0–100 timesteps on the Temporal Cache.

For data, the authors segment source videos into shots, cluster them into scenes using sliding-window CLIP similarity, filter out clips shorter than one second and scenes with more than three distinct individuals, then use Gemini 2.5 Pro in a "group captioning" step to describe all shots in a scene jointly so that character attributes stay consistent. A final validation step feeds each shot and caption back to the model to check accuracy. The dataset size produced by this pipeline is not reported in the paper.

Why This Matters

Impact on research. The paper reframes multi-shot consistency as two separable problems — inter-shot identity and intra-shot motion — and shows that a retrieval-based memory plus a sliding-window memory can address them without changing model architecture. This contrasts with prior approaches that either rely on complex multi-model pipelines, split a fixed-length video into short segments, or insert RNN-like mechanisms into DiT layers (TTT) or add complex positional encodings (LCT, EchoShot). The in-context injection design means the method is broadly compatible with existing pre-trained text-to-video models rather than tied to one backbone.

Real-world applications:

• Filmmaking and previsualization — generating coherent multi-shot sequences from scripted prompts for storyboarding.
• Automated storytelling and short-form content — building narrative videos with consistent characters across scene changes.
• Multi-concept character insertion — placing user-provided subject images into a generated scene while preserving their identity.
• Interactive video editing and extension — steering a narrative mid-sequence by changing the prompt while keeping motion continuous, such as switching from snowboarding to surfing.

Industry relevance. The work comes from a collaboration between Tsinghua Shenzhen International Graduate School and the Kling Team at Kuaishou Technology, and it targets a capability directly relevant to commercial video generation products: controllable, long-form, character-consistent narrative video rather than isolated clips.

Future Directions

• Improving visual quality. The authors state that aesthetics can be further enhanced through improved data curation and optimized training strategies, since their Aesthetics Score did not lead the comparison.
• Better data curation. The reported dataset size and composition are not given, and the paper treats data quality as a limiting factor, suggesting scalable annotation pipelines as an open direction.
• Alternative keyframe retrieval. The current retrieval uses CLIP cosine similarity with a fixed K=3; whether learned or prompt-adaptive retrieval would help is untested.
• Generalization beyond the evaluated backbone and dataset. The method is demonstrated on HunyuanVideo with a self-constructed test set of 20 narrative scenes of 5 interconnected shots each; behavior on other backbones and on public benchmarks remains open.

Target Audience

This paper is best suited to researchers and engineers working on video diffusion models, long-form and multi-shot video generation, and controllable generative video. It is also relevant to practitioners in film production, narrative content creation, and video editing tools who want to understand how character and scene consistency can be maintained across shots. Readers will need background in diffusion models, autoregressive generation, latent representations, and CLIP-based similarity to follow the method section.

Authors’ abstract

Current video generation models perform well at single-shot synthesis but struggle with multi-shot videos, facing critical challenges in maintaining character and background consistency across shots and flexibly generating videos of arbitrary length and shot count. To address these limitations, we introduce \textbf{FilmWeaver}, a novel framework designed to generate consistent, multi-shot videos of arbitrary length. First, it employs an autoregressive diffusion paradigm to achieve arbitrary-length video generation. To address the challenge of consistency, our key insight is to decouple the problem into inter-shot consistency and intra-shot coherence. We achieve this through a dual-level cache mechanism: a shot memory caches keyframes from preceding shots to maintain character and scene identity, while a temporal memory retains a history of frames from the current shot to ensure smooth, continuous motion. The proposed framework allows for flexible, multi-round user interaction to create multi-shot videos. Furthermore, due to this decoupled design, our method demonstrates high versatility by supporting downstream tasks such as multi-concept injection and video extension. To facilitate the training of our consistency-aware method, we also developed a comprehensive pipeline to construct a high-quality multi-shot video dataset. Extensive experimental results demonstrate that our method surpasses existing approaches on metrics for both consistency and aesthetic quality, opening up new possibilities for creating more consistent, controllable, and narrative-driven video content. Project Page: https://filmweaver.github.io