Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editing

AI summary

Overview

Research area: Computer vision / generative video editing — specifically multi-turn (iterative) video-to-video diffusion.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiTs), VAE latent spaces, flow-matching training, attention mechanics, and camera pose parameterization.

One-sentence scope: The paper proposes Memory-V2V, a memory-augmented framework that lets a pretrained video-to-video diffusion model condition on selected and compressed outputs from earlier editing turns, so that independently denoised edits stay consistent with one another.

What This Paper Is About

Video-to-video diffusion models work well for a single edit, but real editing is iterative: users refine a result over several turns. The paper identifies a failure mode it calls cross-turn consistency — when edits are applied sequentially, each turn is treated independently, so previously generated regions drift or get overwritten (for example, an apple turned into an orange becomes a visually different orange in each segment). The goal is a framework that treats prior edits as structured constraints rather than as extra temporal context, achieving consistency across turns without letting computation grow linearly with the number of past edits.

Key Contributions

1. Formulates multi-turn video editing as a distinct problem setting, highlighting the challenge of preserving consistency across edits that are each denoised independently.
2. Proposes Memory-V2V, a memory-augmented framework in which prior edits are treated as structured constraints rather than temporal continuations, combining retrieval with relevance-aware compression.
3. Introduces task-aware, non-temporal retrieval mechanisms: a geometric Field-of-View (FOV) retrieval for video novel view synthesis, and semantic/DINO-feature-based segment retrieval for text-guided long video editing. It also develops dynamic tokenization and adaptive token merging for scalable multi-turn conditioning.
4. Demonstrates improved cross-turn consistency on iterative novel view synthesis and long video editing, and introduces a data augmentation pipeline that temporally extends only target videos from short source–target pairs, avoiding the need for long paired video-to-video training data.

Main Findings

• Multi-turn novel view synthesis (Table 2, evaluated on 40 publicly available videos, three iterative outputs per input): Memory-V2V reports the lowest average MEt3R multi-view consistency score (0.1357) versus ReCam (AR) 0.1485, TrajCrafter 0.1818, and ReCam (Ind) 0.1892. It also reports the lowest rotation error (1.65) and translation error (13.47), compared with ReCam (Ind) 1.97 / 24.23 and TrajCrafter 3.66 / 57.44. Subject consistency is reported as 0.9494 for Memory-V2V, 0.9483 for ReCam (Ind), and 0.9452 for TrajCrafter.

• Baseline failure patterns: TrajCrafter and ReCam (Ind) lose cross-consistency as iterations increase. ReCam (AR), which chains the previous output as the next input, keeps adjacent iterations consistent (e.g., 2nd vs 3rd) but fails across distant ones (e.g., 1st vs 3rd).

• Long video editing (Table 3, 50 videos from the Señorita test set): Memory-V2V reports the highest scores among listed methods on Subject Consistency (0.9326), Background Consistency (0.9233), Imaging Quality (0.6759), Temporal Flickering (0.9862), and Motion Smoothness (0.9939). DynVFX reports the highest Aesthetic Quality (0.5179), and FlowEdit the highest DINO-F (0.8612) and CLIP-F (0.8943). The paper cautions that cross-frame DINO/CLIP similarity must be read alongside edit success: overly high values may indicate insufficient editing, and overly low values unintended changes or temporal instability.

• Qualitative baselines: TokenFlow and RAVE maintain coarse edited-object consistency but often lose fine-grained subject identity or unintentionally modify background regions; CCEdit shows limited editing controllability; LucyEdit preserves identity and background via token reuse but struggles with edited-object consistency over long sequences; FlowEdit often fails to follow the instruction; DynVFX may place the new object at unintended locations.

• Ablation on components (Table 4): Dynamic tokenization alone takes 980.80 s with MEt3R 0.2234; adding video retrieval gives 965.95 s / 0.2169; adding adaptive token merging gives 661.31 s / 0.2344; the full model gives 648.5 s / 0.2208.

• Token responsiveness is layer-dependent (Table 1): In a 30-block DiT, Block 1 correlates weakly with Blocks 2–30 (Pearson 0.608 ± 0.137, Spearman 0.506 ± 0.267, bottom-k overlap 0.730 ± 0.160), while later stages stabilize (Block 11: 0.723 ± 0.115 / 0.657 ± 0.139 / 0.758 ± 0.120; Block 21: 0.753 ± 0.144 / 0.683 ± 0.126 / 0.793 ± 0.114). The authors therefore apply token merging at Blocks 10 and 20.

• Merging beats discarding: Discarding low-responsive tokens degrades fine visual details and motion and introduces artifacts, while merging them reduces computation without affecting cross-consistency or visual fidelity. Merging or discarding highly responsive tokens breaks consistency in overlapping novel-view regions.

• Computational savings: Compared with naive context-window scaling, the method reduces FLOPs and latency by over 90%; adaptive token merging reduces FLOPs and latency by a further 30%. Gains increase as the number of memory videos grows.

• Training cost is modest: Both models are trained with a rectified flow matching loss on 32 A100 GPUs with total batch size 32, converging within 1–2K finetuning steps.

• Not reported: The provided content does not include a human perceptual/user study, and the supplementary sections referenced (retrieval algorithms, RoPE design, ground-truth comparisons) are truncated and their contents are not available here.

Methodology in Plain English

The framework wraps a pretrained single-turn video-to-video model (ReCamMaster for novel view synthesis, LucyEdit for text-guided editing) with an external cache of previously generated outputs.

Step 1 — Store as you go. After each editing turn j, the latent of the generated video is stored in a cache, indexed by its camera trajectory (for novel view synthesis) or simply as a latent (for text-guided editing).

Step 2 — Retrieve selectively. Rather than conditioning on the whole editing history, the method ranks cached items and keeps only the top-k most relevant. For novel view synthesis, relevance is a VideoFOV score: a unit sphere is placed at the first target camera position, M = 64,800 viewing directions are sampled, per-frame fields of view are computed and unioned into a video-level FOV, and two similarity measures (overlap and containment) are combined with λ = 0.5. For text-guided editing, text similarity is considered unreliable due to ambiguous instructions, so segments are retrieved by visual similarity of their source segments using DINO features.

Step 3 — Spend tokens where they matter. Retrieved videos are tokenized at different spatio-temporal compression factors: 1×2×2 for the user input, 1×4×4 for the top-3 most relevant retrieved videos, and 1×8×8 for the rest. This preserves detail for relevant memory while keeping the token count manageable.

Step 4 — Compress the redundant. For each frame, key features are spatially averaged and the frame's responsiveness is computed as its maximum attention response to the target queries. Low-responsiveness frames are fused by a learnable convolutional operator with a reduction factor scaled to the number of conditioning videos, instead of being thrown away.

Step 5 — Extend to long video editing. A long video (over 200 frames) is split into segments that fit the base model's ~81-frame window and edited iteratively, with past edited segments in memory; the segments are then stitched. Because long paired source–target data is hard to obtain, training uses short source–target pairs from a public dataset (56K samples filtered from Señorita-2M), with only the target video temporally extended by an off-the-shelf video extension model.

Why This Matters

Impact on research. The paper reframes memory in video generation. Existing memory-augmented long video generators assume temporal continuity; here the memory must act as a constraint across independently denoised edits.

Authors’ abstract

Video-to-video diffusion models achieve impressive single-turn editing performance, but practical editing workflows are inherently iterative. When edits are applied sequentially, existing models treat each turn independently, often causing previously generated regions to drift or be overwritten. We identify this failure mode as the problem of cross-turn consistency in multi-turn video editing. We introduce Memory-V2V, a memory-augmented framework that treats prior edits as structured constraints for subsequent generations. Memory-V2V maintains an external memory of previous outputs, retrieves task-relevant edits, and integrates them through relevance-aware tokenization and adaptive compression. These technical ingredients enable scalable conditioning without linear growth in computation. We demonstrate Memory-V2V on iterative video novel view synthesis and text-guided long video editing. Memory-V2V substantially enhances cross-turn consistency while maintaining visual quality, outperforming strong baselines with modest overhead.