WanPE: Towards Cinematic Prompt Enhancement for Modern Text-to-Video Generation

AI summary

Overview

Research area: Text-to-video (T2V) generation, specifically the prompt-enhancement stage that sits between a user's request and a video generator.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, GRPO-style reinforcement learning, video diffusion transformers, captioning pipelines, and Bradley–Terry preference modeling.

One-sentence scope: The paper introduces WanPE, a family of prompt-enhancement models (4B to 397B parameters) trained to turn short user requests into shot-level "cinematic plans" that downstream video generators can follow, plus WanPEval, a human-annotated benchmark for measuring this capability.

What This Paper Is About

Modern video generators such as Wan3.0 and Seedance 2.5 can now produce up to 30 seconds of video and follow complex, long instructions. That means the text prompt no longer just describes a scene — it directs the production, specifying how actions, camera movement, lighting, dialogue, and sound unfold across multiple shots over time. The paper's core problem is that conventional prompt enhancers expand a user request "forward" into richer text, producing a synthetic rewriting style that does not match the video-grounded captions the generators were actually trained on, and that tends to lose or distort the user's original requirements as the plan grows longer. WanPE's goal is to close both gaps: learn cinematic planning from real videos rather than invented text, and keep user constraints intact across the whole sequence.

Key Contributions

1. A reframing of prompt enhancement as cinematic planning. The paper redesigns the enhancer so its output is a blueprint orchestrating actions, camera choreography, audio, and narrative across shots and through time before any pixels are rendered, rather than a decorated descriptive caption.

2. Video-grounded reverse SFT. Instead of expanding requests forward, WanPE derives a hierarchical cinematic target y from a real video via captioning and then reconstructs a compatible user request x, training on the resulting pairs. This is done over approximately 1.05M real-world video clips of up to 30 seconds.

3. Semantic-Consistency GRPO (SC-GRPO). A GRPO-based optimization stage using a nine-dimensional semantic reward (style, subjects, actions, dialogue, sound, camera, lighting, spatial relations, scene) that penalizes omissions, alterations, incorrect bindings, and temporal inconsistencies across shots.

4. WanPEval, a human-annotated benchmark. 249 curated requests spanning 5 to 30 seconds and three granularity levels (intent-, scene-, and shot-level), supported by roughly 11K blind pairwise video assessments, evaluated with expert preference scores and a Bradley–Terry model.

Main Findings

• WanPE-397B leads the 5–15-second comparison. On the 5–15-second subset of WanPEval under Wan3.0's video generator, WanPE-397B ranks first at every duration in both preference score S and Bradley–Terry score, with overall values of 50.61 and 61.85, and outperforms Seedance 2.0 by 7.19 points in S.

• Gains grow with duration. Against the original-request baseline under the same generator, WanPE-397B improves by 10.66, 16.28, and 18.84 points at 5, 10, and 15 seconds. On the 30-second subset it raises S from 9.38 to 60.24 — a 50.86-point gain.

• Competitive at 30 seconds. On the 30-second subset, Seedance 2.5 scores 59.76 overall versus WanPE-397B's 60.24, with WanPE scoring 81.25 on animation and 73.68 on speech categories.

• Scaling helps monotonically on the arena. Under the same Wan3.0 generator, S rises from 43.60 (WanPE-4B) to 46.00 (9B) to 47.85 (35B) to 50.61 (397B) on the 5–15-second subset.

• Reverse construction beats forward rewriting. WanPE-397B-SFT reaches an overall S of 49.86, beating a carefully tuned Forward Rewriting baseline by 10.37 points and a Forward-target SFT baseline by 14.69 points, ranking first in all seven content categories.

• SC-GRPO raises semantic consistency by 18.6–23.3 points across the four model sizes on WanPEval, and increases the share of "perfect" outputs while reducing failures. For WanPE-397B, overall consistency rises from 75.5 to 97.6, with "Perfect" going from 29.7 to 85.5 and "Failure" dropping from 36.9 to 2.8.

• Better semantic consistency also improves video. Comparing WanPE-397B before and after SC-GRPO under the same Wan3.0 generator, overall expert preference rises from 42.70 to 49.69, with gains across all seven categories.

• Cinematic plans transfer across generators. After GPT-5.4 format adaptation, WanPE-397B beats the native enhancer by 14.45 points on LTX-2.5-Base and 5.17 points on MiniMax-H3-Base.

• Granularity-level ranking. WanPE-397B ranks first on intent-, scene-, and shot-level requests with scores of 52.44, 49.26, and 49.32, gains of 18.55, 16.97, and 13.45 points over no prompt enhancement.

Methodology in Plain English

The paper treats the enhancer as a function that maps a user request to a textual condition, and defines the ideal output distribution as the video-grounded caption distribution restricted to outputs that satisfy every user constraint. Training that distribution is done in three stages.

Stage 1 — Reverse pair construction. Rather than writing prompts and expanding them, the authors start from videos. Long-form footage is split into clips of at most 30 seconds and filtered for technical validity, visual quality, and motion quality, yielding about 1.05M clips across ten content dimensions. A multimodal captioner converts each clip into a hierarchical cinematic target: a video-level summary plus timestamped shot-level descriptions covering composition, subjects, actions, lighting, camera movement, transitions, dialogue, music, and sound effects. Category-specific instructions adapt this structure for animation, music, dialect, rich-text, advertising, visual-effects, and large-motion content. A language model (gpt-5.4) then works backward from each caption to reconstruct a natural-sounding user request, using five few-shot examples drawn from a pool of 2K human-written requests; the reconstruction is constrained to include only requirements the caption actually supports.

Stage 2 — Supervised fine-tuning. The enhancer is fine-tuned by minimizing negative log-likelihood of the video-grounded cinematic target given the reconstructed request, so it learns the joint organization of cinematic elements from real films rather than from an invented rewriting style.

Stage 3 — SC-GRPO. To keep user requirements intact across shots, the authors build a set of roughly 15K prompts — mixing human-written T2V prompts with cases where videos generated from SFT-enhancer outputs failed to realize the request, as flagged by annotators. A model (Qwen3.7-Max) scores each candidate condition from 0 to 100 on nine semantic dimensions, checking for omitted, weakened, altered, or contradictory requirements, wrong subject–attribute or speaker–dialogue bindings, and ordering or cross-shot conflicts. Semantically equivalent paraphrases and compatible elaborations are accepted. Those scores become group-relative advantages in a standard GRPO objective with a clipped ratio and a KL penalty against the SFT reference.

Evaluation. WanPEval contains 249 requests stratified by duration (5–30 seconds) and granularity, plus aspect ratio, shot-count specification, audio components, and cinematic language. Text-level scoring uses the same semantic reward. Video-level scoring uses anonymous pairwise comparison by 60 experts in screenwriting, directing, and cinematography, with method identity hidden and left–right order randomized; judges choose A preferred, B preferred, both good, or both poor. The paper reports lower-bound, upper-bound, and overall preference scores, and fits a Bradley–Terry model to account for opponent strength, treating both-good and both-poor as ties. Four enhancer variants are initialized from Qwen3.5-4B, 9B, 35B-A3B, and 397B-A17B; experiments run on 512 GPUs.

Why This Matters

The paper argues that as generators get longer contexts and better instruction-following, the prompt becomes the primary control surface for the entire production, not a cosmetic detail. It shows that where the supervision comes from — real videos versus model-invented rewrites — measurably changes the quality of the resulting videos, and that a distributional mismatch between enhancer output and generator conditioning is a real, correctable problem. It also shows semantic-consistency optimization, usually treated as a text-quality concern, translates into measurable downstream video-quality gains.

Real-world applications:

• Film and advertising previsualization. WanPEval explicitly covers action, drama, advertisement, animation, speech, singing and dance, and knowledge categories, matching the content types professional storyboarding and shot-planning workflows handle.
• Multi-shot content production. Because the enhancer plans across temporally ordered shots with transitions, it targets creators who need coherent sequences rather than single clips.
• Localization and dialogue-heavy content. The speech and dialect handling in the captioning scheme, plus the speaker–dialogue binding constraint in the reward, point toward dubbing and dialogue-driven production.
• Transferable enhancement across vendors. Since format-adapted WanPE beat the native enhancers of LTX-2.5 and MiniMax-H3, a single planning model could serve as a front end for multiple generators.

Industry relevance: every evaluated commercial system (Seedance 2.5, Seedance 2.0, HappyHorse 1.1, Kling 3.0, MiniMax-H3, LTX-2.5) ships a native prompt enhancer, but the paper notes their designs and training recipes are largely undisclosed. WanPE offers a publicly described, scale-tested recipe and a benchmark for a component that is currently a black box. WanPE-397B leads all evaluated commercial offerings at 5–15 seconds and remains competitive with Seedance 2.5 at 30 seconds.

Future Directions

• Extending beyond 30 seconds. The corpus, the benchmark, and the claim of competitiveness with Seedance 2.5 all stop at 30 seconds. Whether cinematic planning remains coherent over longer horizons is not tested.

• Format adaptation without an external LLM. Cross-generator transfer currently relies on GPT-5.4 to convert WanPE outputs into each generator's conditioning format. Learning native multi-format conditioning, or removing that conversion step, is an open problem.

• Closing the remaining gap on the 30-second arena. WanPE-397B is described as competitive rather than leading at 30 seconds, and per-category numbers show it trailing Seedance 2.5 on action (50.00 vs 68.18) and drama (53.85 vs 56.00), which suggests headroom in those domains.

• Reward-model dependence. Semantic scoring uses a nine-dimensional reward produced by Qwen3.7-Max and text-level evaluation uses gemini-3.1-pro-preview. How sensitive SC-GRPO is to that evaluator, and whether the reward can be grounded in video rather than text alone, is not reported.

Target Audience

Researchers and engineers working on text-to-video generation, prompt optimization, and RL-based post-training of generative models will get the most from this paper, particularly those interested in how supervision distribution, not just data volume, shapes downstream generation quality. It is also relevant to applied teams building production pipelines on top of commercial video APIs, since it provides a benchmark (WanPEval) and a transfer test across multiple generators. Readers without a background in RL fine-tuning or diffusion video models will find the training sections dense, though the framing argument about prompts as cinematic blueprints is accessible on its own.

Authors’ abstract

Video generation begins in text space by authoring a cinematic screenplay, then materializes into pixels. As contemporary video generators scale to 30 seconds and faithfully follow complex conditions, the textual prompt largely directs the production, planning how actions, camera trajectories, lighting, and sound unfold across multi-shot sequences. In this paper, we present WanPE, a 397B-parameter prompt enhancement model trained on 1.05M real-world videos to master director-level cinematic planning. WanPE formulates shot-level cinematic plans via video-grounded reverse construction and employs Semantic-Consistency GRPO (SC-GRPO) to faithfully preserve user requirements across shots and over time. To benchmark this capability, we curate WanPEval, a human-annotated testbed covering durations from 5 to 30 seconds across varying intent granularities, supported by approximately 11K blind pairwise assessments. When powering Wan3.0's video generator, WanPE-397B boosts human preference over raw user prompts by 10.66-18.84 points at 5-15 seconds and by a dramatic 50.86 points in the 30-second arena. Ablation studies show that reverse construction demonstrates clear superiority over forward rewriting, while SC-GRPO robustly preserves semantic fidelity across model scales. Ultimately, WanPE leads all evaluated commercial offerings at 5-15 seconds and remains competitive with Seedance 2.5 at 30 seconds.