WeVisCut: The Whole Editing Room in One Agent From Narration to Long-Form Video

AI summary

Overview

Research area: Computer vision and multimodal agent systems, specifically agentic video editing and post-training of compact tool-using policies for long-form video production.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, reinforcement learning from verifiable rewards (DAPO/GRPO-style optimization), on-policy distillation, and tool-calling agent harnesses.

Scope (one sentence): The paper presents WeVisCut, a system that turns a fixed narration script into a long-form video using a single 9B policy that plans, retrieves visuals, composes, inspects, and repairs within one stateful editing environment, trained on over 130K executably-verified examples recovered from 3,750 finished videos.

What This Paper Is About

Producing a narrated long-form video requires many interdependent decisions — shot planning, visual retrieval, narration synthesis, and timeline assembly — where changing one shot's duration shifts later timestamps, and unsuitable footage can force both retrieval and planning to be redone. Existing agentic editors orchestrate general-purpose models through multi-stage workflows, but leave unclear whether a small, task-adapted policy can hold video quality while keeping token cost low. The authors' goal is to connect three things that are usually separate: stateful execution in a production harness, executable supervision derived from real tool feedback, and policy learning for a single compact model that can also recover from its own failures.

Key Contributions

1. A script-to-video agent harness that maintains shared production state, validates tool actions against typed contracts, and supports inspection, bounded repair, replanning, and final delivery — while preserving artifacts from earlier successful steps.

2. A four-stage data pipeline (editing inversion, verified replay, observe–reflect–repair, trajectory materialization) that converts finished videos — which do not record the editing actions that made them — into executable training examples. Applied to 3,750 video projects, it yields 134,771 examples.

3. A unified 9B policy (Qwen3.5-9B backbone) serving three roles — orchestrator, generator, and critic — trained in three phases: unified SFT plus runtime-verified rejection-sampled fine-tuning (RFT), then Guided-OPD with a decaying teacher, then RCF-DAPO, a role- and context-factored variant of DAPO that compares rewards only within matched roles and decision contexts.

4. A 100-task benchmark and five metrics (Script Content Fidelity, Visual–Narration Semantic Match, Timeline Plan Realization, Long-Horizon Consistency, End-to-End Rendering Yield) that link scripts, shot plans, execution traces, and rendered videos, plus a separately reported Production Cost Efficiency measure.

Main Findings

• Large gain over its own base model: WeVisCut raises Overall Score from 74.5 to 90.5 over the Qwen3.5-9B initialization — a gain of 16.0 points — improving all five metrics. SCF rises by 18.6 points and VNSM by 15.4 points, so the gains go beyond mere delivery success into script fidelity and visual grounding.

• Comparable to Claude Opus 5 at far lower token cost: Overall Score is 90.5 versus 89.9 for Claude Opus 5, with higher means on four metrics and slightly lower VNSM, and no component differing by more than 1.3 points. WeVisCut uses 2,377 generated tokens per successful output minute versus 78,763 — roughly one thirty-third as many — while holding 3.1 tool calls per successful output minute (down from 11,883 tokens for the base policy).

• Beats task-matched native editors: Against Crayotter and VideoAgent in their own native workflows, WeVisCut scores 90.5 Overall versus 84.9 and 82.5, with the highest mean on all five metrics; SCF and VNSM each improve by 6.8 points relative to Crayotter.

• Lower end-to-end cost: Per successful output minute, WeVisCut needs 2.1 wall-clock minutes versus 8.6 for Crayotter and 10.4 for VideoAgent, at estimated costs of $0.09 versus $0.97 and $0.58 respectively, with failed attempts included in the cost numerator.

• Each training stage adds measurable gains: SFT lifts ERY from 70.4 to 87.6 (the largest stage-wise delivery improvement); adding RFT improves SCF by 6.2 and TPR by 5.7 over SFT alone (Overall 83.5); Guided-OPD raises VNSM from 65.1 to 67.4 and ERY from 89.8 to 91.5 (Overall 84.9); RCF-DAPO adds another 7.4 SCF and 7.2 VNSM points, reaching Overall 90.5.

• Unadapted scaling does not substitute: Qwen3.6-35B-A3B gains 9.0 points on ERY but loses 2.5 points on VNSM, showing delivery success does not imply better visual–narration alignment. WeVisCut also exceeds Qwen3.5-397B-A17B and Gemini 3.1 Pro on all five metrics.

• Human raters agree: Three annotators rated 100 held-out tasks on a 1–5 scale across six dimensions; WeVisCut's overall preference is 4.56 versus 3.48 (Crayotter) and 3.46 (VideoAgent), with the largest gaps in narrative coherence (1.23 and 1.28 points). The authors state these descriptive means do not establish statistical significance.

Methodology in Plain English

The team built a "production state" that all tools read from and write to — it tracks narration coverage, shot order, timing, per-shot assets, audio, templates, renders, and unresolved issues, and keeps planned content separate from what has actually been produced. A single model fills three roles as the work proceeds: an orchestrator that picks the next tool and arguments, a generator that supplies structured content (script segments, shot plans, retrieval queries, template slot values), and a critic that inspects results and requests or verifies repairs.

Because finished videos on the web don't reveal the tool calls that produced them, the authors work backwards. They take 3,750 finished narration-driven videos from Bilibili across four content areas, recover timestamped narration, shot structure, and full sentence–shot alignment (editing inversion), then re-execute that recovered intent through the same toolchain used at deployment (verified replay). When replay uncovers defects, an observe–reflect–repair loop applies bounded fixes while keeping artifacts that still work. The recorded interactions — pre-action states, tool names and arguments, real returns, quality verdicts, failure–repair links — are then organized into role-specific training examples, with loss applied only to model-authored tokens and failures kept visible rather than rewritten as successes.

Training happens in three phases. Phase 1 does supervised fine-tuning with LoRA on the 134,771 examples, then adds runtime-verified rejection-sampled fine-tuning: at the first recoverable failure the teacher regenerates the rest of the sequence from the last replayable state, and both student-only successes and teacher-assisted repairs are admitted only if they pass runtime checks and a quality threshold. Phase 2 (Guided-OPD) mixes teacher and student outputs turn by turn, using forward KL on the teacher's top-K tokens and reverse KL on student-generated prefixes, while gradually reducing the teacher's probability to zero so training ends fully student-controlled. Phase 3 (RCF-DAPO) samples alternative outputs for the same decision — for example, different retrieval queries for the same shot — restores the same production state, executes each candidate with a frozen continuation policy and paired random seeds, scores them with fixed role-specific rubrics, and normalizes rewards only against candidates sharing the same role and context. A runtime also validates every tool call against schemas and prerequisites, so a stitching request with missing renders or background music returns a typed blocker instead of a false success.

Why This Matters

The paper shows that a compact, task-adapted policy with a well-designed runtime can match a frontier general model on a long-horizon creative production task while being dramatically cheaper in tokens, and can beat purpose-built multi-agent editing pipelines on quality, wall-clock time, and money. It also contributes a way to manufacture executable agent supervision from artifacts (finished videos) that contain no action traces — a pattern that generalizes beyond video.

Real-world applications:

• Automated generation of narrated explainer videos in science, history, geography, technology, and health — the domains shown in the paper's examples.
• Educational and training content production where a fixed script must be paired with sourced visuals and timed narration.
• News and commentary video assembly from an existing source pool, since the task contract fixes narration and source assets.
• Cost-sensitive content pipelines where per-minute delivery cost and reliability matter more than peak model capability.

Industry relevance: the work targets the practical trade-off that production teams actually face — quality versus per-minute generation cost and failure rate. Reporting cost per successful output minute (including failed attempts), delivery yield, and blinded human preference makes the results directly comparable to operational budgets rather than to benchmark scores alone.

Future Directions

• Isolating recovery from aggregate metrics: The authors explicitly note that aggregate metrics do not isolate failure recovery, and that their stage comparisons measure cumulative ordering effects rather than the independent contributions of turn guidance, snapshot matching, or individual role rewards. Ablations that separate these would sharpen the picture.

• Broadening the task contract: CutClaw, CineAgents, and DirectorAgent were excluded from quantitative comparison because their workflows omit or replace parts of the fixed-narration, fixed-source contract. Extending the harness and benchmark to free-form narration or open-ended source discovery is an open problem.

• Statistical validation of evaluation: The human study is descriptive only and does not establish significance or validate individual automatic metrics; the Overall Score also carries no uncertainty estimate. Formal metric-validation and significance testing remain open.

• Scaling the supervision source: The pipeline was applied to 3,750 projects to produce 134,771 examples. Whether the same editing-inversion-and-replay recipe scales to larger and more diverse video corpora, and how far the 9B policy improves with it, is untested here.

Target Audience

Researchers and engineers working on agentic systems, tool-use post-training, and multimodal video generation or editing; practitioners building automated content-production pipelines who care about per-minute cost and delivery reliability; and readers interested in methods for deriving executable agent trajectories from passive artifacts such as finished videos. The paper is not beginner-friendly — it presumes comfort with reinforcement learning for LLM policies, distillation schedules, and agent runtime design.

Authors’ abstract

Producing long-form videos from narration requires interdependent decisions that remain coherent across shots. Existing agentic editors often use multi-stage workflows driven by general models, leaving the quality-cost trade-off of compact, task-adapted policies unclear. We address this gap with WeVisCut, a long-form script-to-video system that connects tool execution with policy learning. Its agent harness maintains shared production state so that tool feedback informs subsequent decisions and repairs. Because finished videos do not expose these decisions, our four-stage data pipeline combines editing inversion with verified replay to turn recovered editing intent into executable supervision. Applied to 3,750 finished videos, this pipeline yields over 130K training examples. We use this supervision to train a single 9B policy that coordinates production and recovers from failures within the same harness. Moreover, we introduce a 100-task benchmark and five metrics assessing editing quality and delivery reliability. Under a shared tool environment, WeVisCut substantially outperforms its base model and achieves comparable performance to Claude Opus 5 while using nearly 33x fewer generated tokens per successful output minute. It also outperforms task-matched native editors on our proposed metrics.