Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning

AI summary

Overview

• Research area: Computer vision, specifically generative video editing with large language model (LLM) and vision-language model (VLM) agents.
• Technical level: Advanced. The paper assumes familiarity with diffusion-based video editing, shot boundary detection, multi-agent pipelines, and classifier-free / prompt-based generation control.
• Scope: The paper formalizes the Multi-Instruction Multi-Shot Long-Video Editing (MMLVE) task, proposes an agentic framework called MMLVE-Agent, and releases a benchmark (MMLVE-Bench) with three dedicated evaluation metrics.

The paper is arXiv:2608.26809v1 [cs.CV], dated 27 Aug 2026, by Chenyang Wu, Fuchen Long, Binyuan Huang, Xinlong Sun, Xi Chen, Chun-Le Guo, and Chongyi Li, from VCIP, CS, Nankai University and the Smart Creation Platform Department, Online Video BU, Tencent. It is noted as work done during the Tencent Qingyun Program internship, with a project page at https://wucy0519.github.io/MMLVE/.

What This Paper Is About

Existing video editing models work well on short, single-shot clips (the paper describes these as typically under 15 seconds) with simple instructions, but they break down on long videos made of many shots with several different instructions at once. The common workaround, chopping the video into fixed-duration chunks and applying every instruction to every chunk, causes the model to hallucinate entities that are not present, lose an edited entity's identity across shots, and scramble the original temporal order. The paper's goal is to define this problem formally, build an agent-based system that reasons about each physical shot before editing it, and create a benchmark plus metrics to measure success.

Key Contributions

1. A formal task definition. The authors introduce the Multi-Instruction Multi-Shot Long-Video Editing (MMLVE) task, governed by three core objectives: Cross-Shot Editing Consistency (CSEC), Multi-Instruction Decoupling (MID), and Zero-Destruction on Spatiotemporal Structure (ZDSS).
2. The MMLVE-Agent framework. A heterogeneous multi-agent pipeline built on the "Thinking on Shots" paradigm, featuring a retrieval-based on-demand editing strategy, a Global Memory Card that serves as a global visual anchor, and a closed-loop Pos-Neg Editing Feedback (P-NEF) mechanism for self-correction at both image and video levels.
3. The MMLVE-Bench dataset. A manually curated benchmark derived from the open-source UniVA-Bench, containing 25 high-quality multi-shot long-video clips (each approximately one minute) with roughly 5 distinct editing instructions per video, covering ADD, DELETE, and MODIFY operations.
4. Three MMLVE-focused evaluation metrics. Quantitative metrics for CSEC, MID, and ZDSS, each decomposed into five sub-dimensions scored on a 0 to 20 scale for a maximum of 100 points per indicator.

Main Findings

• Highest overall score on MMLVE-Bench: MMLVE-Agent achieved the best average score of 81.84, compared with Seedance 2.0 at 79.47, Kling o3 at 70.57, and HappyHorse 1.0 at 69.61.
• Strongest consistency and decoupling: MMLVE-Agent scored 84.80 on CSEC and 79.04 on MID, the highest of all evaluated methods. Seedance 2.0 scored 77.58 and 78.58; Kling o3 scored 70.00 and 72.57; HappyHorse 1.0 scored 74.68 and 67.28.
• A trade-off on structural preservation: Seedance 2.0 obtained the highest ZDSS score at 82.25 versus MMLVE-Agent's 81.68, but the authors attribute this to a conservative strategy that skips instructions in complex scenes, which they link to its lower CSEC and MID scores.
• Rejected cases in baselines: In Table 1, Seedance 2.0 and Kling o3 are marked as rejecting one to two scenarios due to internal safety mechanisms; those cases were ignored in the mean calculation.
• Human evaluation agreement: MMLVE-Agent was ranked first in 75.6% of the 45 collected rankings and in the top two in 93.3% of cases, with a best average rank of 1.36 versus 2.63 for Kling o3, 2.67 for Seedance 2.0, and 3.16 for HappyHorse 1.0.
• Statistically significant head-to-head wins: The user study reports MMLVE-Agent preferred over Seedance 2.0, Kling O3, and HappyHorse in 88.4%, 80.5%, and 93.3% of co-rated cases respectively, all significant under a two-sided Wilcoxon signed-rank test (p < 0.001, Bonferroni-corrected).
• Distinct failure profiles among baselines: Seedance 2.0 was rarely ranked best (4.7% first-place) but frequently second (48.8%), while Kling O3 was more polarized (17.1% first, 19.5% last).
• Qualitative failure modes documented: The paper shows Seedance 2.0 forgetting an added "red Christmas hat" in later shots, Kling o3 dyeing white paper blue when instructed to create a "blue sponge," and HappyHorse 1.0 hallucinating unrequested yellow and blue sponges and deleting subsequent narrative shots in favour of earlier ones.
• P-NEF ablation is qualitative only: The paper reports that using only Negative Editing Feedback causes "catastrophic forgetting" and more trial-and-error iterations, while the full P-NEF balances attention and ensures monotonic improvement; no numeric ablation values are reported.

Methodology in Plain English

The system works in three stages.

First, Instruction and Video Analysis. The video is cut into its real camera shots using the PyDetect library rather than by fixed duration. Simultaneously, an LLM Agent rewrites the user's messy prompt, resolving conflicts and separating it into distinct entities and their individual editing commands. A VLM Agent then walks through each shot, pulling out representative keyframes and describing what happens in each one.

Second, Global Memory Card Making. To keep an edited character looking the same across shots, the system needs a single visual reference. The VLM checks keyframes from all shots to find frames containing the target entity; the top six highest-confidence keyframes (k=6) are assembled into a reference grid. A generation model synthesizes an initial image of the original entity from that grid. A VLM quality-check agent then judges whether the result matches the keyframes, giving a pass or fail plus two kinds of feedback: a Negative Prompt listing things to avoid and a Positive Prompt listing features to keep. The Positive Prompt is the paper's key insight. If you only feed the model a growing list of things to avoid, its attention drifts toward those items and it damages parts that were already correct. The Positive Prompt anchors what was already right. This loop repeats until the check passes or reaches a maximum of 3 attempts (T_max = 3), after which the best-scoring candidate is chosen. The final validated image is then edited according to the user's instruction to produce the Global Memory Card, a side-by-side "before and after" reference.

Third, Multi-Shot Video Editing. For each shot, the VLM votes three times on whether the target entity is actually present. Editing is triggered only if the entity receives at least 2 positive votes; otherwise the shot is left untouched, which prevents hallucinating entities in shots where they do not exist. For shots that do get edited, the original shot, the enriched entity description, the instruction, and the Global Memory Card are sent to a video editor. A video-level version of the same feedback loop then checks whether the edit succeeded and whether unedited regions and motion were preserved, generating corrective prompts if not. Finally, edited shots and untouched shots are concatenated in original order.

For implementation, the framework uses Gemini 3.5 Flash as the unified VLM backbone for the LLM Agent, VLM Agent, and VLM QA Evaluator; Nano Banana 2 for image generation in the Global Memory Card module; and HappyHorse as the underlying video editing model. All experiments were conducted on a MacBook Pro (M5 chip).

For evaluation, each of CSEC, MID, and ZDSS is split into five sub-dimensions (15 in total), each scored 0, 10, or 20 by the VLM judge. The paper explicitly avoids CLIP-style frame-level metrics, arguing they lack the reasoning capacity to assess instruction decoupling and long-term spatiotemporal consistency. For the human study, 9 expert evaluators used a custom web platform with blind, anonymized, synchronized multi-video playback, each evaluating 5 randomly assigned cases and ranking all four results.

Why This Matters

Impact on research. The paper reframes long-video editing away from duration-based chunking toward shot-aware, agent-driven reasoning, and it provides a named task (MMLVE), three explicit objectives (CSEC, MID, ZDSS), a 25-clip benchmark, and a 15-dimension scoring rubric. That combination gives later work a common vocabulary and a measurable target, rather than leaving long-video editing as an informal extension of short-clip methods.

Real-world applications (drawn from the paper's framing of natural-language video manipulation):

• Film and television post-production, where editors need to apply the same visual change to a character across many cuts.
• Advertising and branded content, where a product or logo must be inserted or altered consistently across a multi-shot spot.
• Social media and creator content, where minute-long multi-shot videos are edited by text instruction rather than manual timeline work.
• Archive or stock footage modification, where specific objects must be added, deleted, or changed while leaving backgrounds and camera motion intact.

Industry relevance. The work is a collaboration between Nankai University and Tencent's Online Video BU, and was conducted during the Tencent Qingyun Program internship, indicating direct industrial interest in text-driven video editing at scale. The paper's critique that "blind chunking" forces hallucinated content is a practical deployment concern, not just an academic one.

Future Directions

• Reducing dependence on proprietary components. The pipeline relies on Gemini 3.5 Flash, Nano Banana 2, and HappyHorse; the paper does not report open-weight alternatives or ablations isolating each component's contribution numerically.
• Scaling the benchmark. MMLVE-Bench contains 25 clips at roughly one minute each with about 5 instructions per video; whether the findings hold for longer or larger-scale content is not reported.
• Cost and efficiency of the feedback loops. P-NEF runs up to 3 attempts at both image and video level, but the paper does not report runtime, API cost, or iteration-count statistics, aside from stating experiments ran on a MacBook Pro (M5 chip).
• Extending the ablation evidence. The P-NEF comparison against negative-only feedback is presented qualitatively in Fig. 6 with no numeric scores, leaving open how much each mechanism (Global Memory Card, retrieval voting, P-NEF) contributes quantitatively.

Target Audience

Researchers and engineers working on generative video editing, diffusion-based content manipulation, and LLM/VLM agent pipelines for multimodal tasks. It is most useful to readers who already understand how prompt-conditioned generative models work and want a concrete formulation of the multi-shot long-video problem, a benchmark to test against, and a detailed agentic architecture to build on. Practitioners in video production tooling and content platforms will also find the failure-mode analysis of chunking-based baselines directly relevant, though the paper is not written as an introductory text.

Authors’ abstract

While generative AI has significantly advanced video editing, existing methods primarily focus on single-shot or short video clips. Editing long videos with multiple instructions remains a formidable challenge. Naive chunking strategies, e.g., fixed-duration segmentation, often lead to entity fragmentation, severe editing hallucinations, and disrupted temporal continuity. To bridge this gap, we introduce the Multi-Instruction Multi-Shot Long-Video Editing (MMLVE) task, which is structured around three core objectives: Cross-Shot Editing Consistency (CSEC), Multi-Instruction Decoupling (MID), and Zero-Destruction on Spatiotemporal Structure (ZDSS). To tackle these three unique challenges, we introduce an agentic editing framework that leverages the synergy of Large Language Models (LLMs) and Vision-Language Models (VLMs) to achieve shot-level video decoupling and precise instruction parsing. Furthermore, to comprehensively evaluate this task, we construct MMLVE-Bench, which is an MMLVE-focused dataset characterized by complex real-world spatiotemporal dynamics, high-density heterogeneous instructions, and sparse, random entity distributions. Three MMLVE-focused evaluation metrics are further exploited to assess the quality of the editing results. Extensive experiments demonstrate that our MMLVE-Agent outperforms existing closed-source SOTA approaches (e.g., Seedance 2.0), successfully eliminating editing hallucinations, preserving cross-shot editing consistency, and attaining seamless spatiotemporal transitions.