Streaming Video Editing with Easy Adaptation

AI summary

Overview

Research area: Computer vision, specifically video diffusion models and real-time (streaming) video editing.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiTs), autoregressive generation, LoRA adapters, key/value caching, and singular value decomposition.

Scope: This paper introduces SVEET, a framework that learns video editing capability on a frozen bidirectional video diffusion model and transfers that capability zero-shot to a frozen causal (streaming) backbone, achieving 15 FPS on a single H100 GPU.

What This Paper Is About

Existing video editing models process whole clips bidirectionally, which makes them incompatible with streaming applications where frames must be produced causally and with low latency. Training a streaming editing model from scratch is prohibitively expensive, and the authors cite prior streaming distillation work requiring 128 H100 GPU days plus synthesized ODE pairs. SVEET asks whether high-quality streaming video editing can instead be obtained by training only on a pretrained bidirectional model and transferring the learned editing ability directly to a streaming backbone without retraining it.

Key Contributions

1. First study of bidirectional-to-streaming video editing transfer without retraining or distilling the streaming backbone (to the authors' knowledge).
2. Two identified principles for streaming-compatible transfer — backbone feature disentanglement (control stays decoupled from the base model) and conditional frame independence (source encoding avoids cross-frame dependencies) — implemented as a temporally independent control branch that replaces full spatiotemporal self-attention with per-frame 2D spatial attention.
3. Orthogonal Decoupled Training (ODT), which uses ridge regression and a thin SVD to estimate the dominant bidirectional-to-causal feature discrepancy subspace per DiT block, then constrains the trainable control update to the orthogonal complement of that subspace.
4. Empirical validation across three editing tasks (style transfer, inpainting, depth-to-video) showing stronger metrics than the compared baselines and real-time performance at 15 FPS on a single H100 GPU with no auxiliary acceleration, plus theoretical bounds for the decoupling strategy in Theorems 1 and 2.

Main Findings

• Architecture matters for transfer: In comparative experiments, Wan-Fun fine-tuning suffered severe degradation and almost completely failed to transfer; full spatiotemporal VACE retained only partial editing capability with unstable temporal behavior; the temporally independent 2D-attention variant transferred substantially more reliably.
• Best overall quantitative results: SVEET reported the highest VLM editing-accuracy score in all three tasks — 7.4322 for style transfer, 8.4667 for inpainting, and 8.4083 for depth-to-video — against baselines including SDEdit+CF, StreamDiffusionV2, Daydream+CF, LiveEdit, a 3D VACE attention variant, and channel concatenation.
• User study preference: With 10 participants scoring 15 generated videos, SVEET scored 9.0800 (editing correctness), 9.2067 (structural preservation), and 8.1133 (overall smoothness), versus 7.8533 / 8.6200 / 7.9533 for DayDream+CF and lower values for SDEdit+CF and SDV2.
• Ablation confirms complementarity: 2D attention alone improved source preservation and streaming stability (VLM 7.1898, motion smoothness 0.9830); adding ODT gave the best result (7.4322, 0.9898), compared with 7.0653 / 0.9809 for 3D without ODT and 7.3315 / 0.9856 for 3D with ODT.
• Alternative transfer strategies underperformed: Inference-stage projection scored 6.4250 VLM and two-stage teacher forcing 7.0927, and the latter still required additional adaptation of the streaming backbone.
• Adaptive rank beats fixed rank: Fixed ranks (k=32, 64, 128) generally underperformed adaptive energy-based selection, with r=80% achieving comparatively better performance across most metrics.
• Discrepancy energy is concentrated: Appendix A reports that approximately 4%–21% of singular directions explain 80% of the spectral energy across different layers, supporting the compact discrepancy-subspace assumption.
• No key/value cache needed in the control branch: Because each frame's conditioning depends only on that frame, the control branch adds no cache-growth overhead on top of the autoregressive backbone.

Methodology in Plain English

The authors start from an existing conditional video diffusion design (VACE) that adds a separate control branch to a frozen diffusion transformer and injects its features into the backbone additively. They modify that branch so that each frame's tokens only attend to other tokens in the same frame, rather than attending across time. Temporal coherence is left entirely to the streaming backbone's own autoregressive history.

The harder problem is that the control branch is trained against a bidirectional model but deployed on a causal one with different parameters and feature distributions. To handle this, the authors run a small calibration set of video–prompt pairs through both models, record hidden states at each block, and fit a ridge-regression linear map from bidirectional features to causal features. Subtracting the identity from this map gives a residual that represents the direction of the bidirectional-to-causal shift. A truncated SVD of that residual yields the dominant directions of the mismatch; the authors keep enough directions to cover 80% of the cumulative spectral energy per block and build a projector onto the complementary space. The trainable control update — parameterized with LoRA adapters of rank 128 — is then constrained to live in that orthogonal complement, so controllability learning does not consume the directions responsible for the causal/bidirectional gap.

Training uses Wan2.1-1.3B-VACE as the bidirectional backbone and chunk-wise Causal Forcing as the streaming backbone, both frozen, with AdamW (learning rate 1×10⁻⁴, weight decay 10⁻²), 10 epochs per task on a single NVIDIA A100 (80GB), batch size 1, 81-frame clips at 480×832 resolution. Data comes from Ditto (style transfer) and VPData (inpainting, depth-to-video), with depth maps extracted by Video-Depth-Anything; each task has roughly 6K–13K training samples. Evaluation uses held-out sets of 120 Ditto video pairs for style transfer and 80 VPData samples each for inpainting and depth-to-video, scored by GPT-4o (1–10 Likert), CLIP-T (CLIP ViT-L/14), and six VBench metrics.

Why This Matters

The work reframes streaming video editing as a transfer problem rather than a from-scratch training or distillation problem, which the paper positions against adaptation pipelines costing 128 H100 GPU days. It suggests that the expensive part — the pretrained generative prior — can be reused untouched, with only a lightweight control pathway and a feature-space constraint added. The two design principles (backbone feature disentanglement, conditional frame independence) are stated generally enough to guide other streaming adaptation efforts.

Real-world applications named or implied in the paper:

• Live style transfer on a video stream, where the target appearance is swapped in real time.
• Online inpainting, filling or removing regions frame by frame during capture or broadcast.
• Interactive content creation, where users edit video with low-latency feedback.
• Real-time avatar synthesis, which the paper lists among low-latency applications that causal frame generation supports.

Industry relevance: 15 FPS on a single H100 GPU without auxiliary acceleration is the kind of throughput that matters for live production tooling, and the fact that the streaming backbone is never retrained lowers the barrier for teams that already have a bidirectional editing model in place across heterogeneous backbones.

Future Directions

• Stronger base editing models: The authors note their method still relies on the capability of the underlying bidirectional editing model, so transfer quality is bounded by what that model can already do.
• Broader editing tasks: Extending the transfer paradigm beyond the three evaluated tasks (style transfer, inpainting, depth-to-video) is stated as future work.
• More heterogeneous backbones: The paper reports transfer results on additional streaming backbones in Appendix D.2 and names more heterogeneous backbone combinations as an open direction.
• Subspace selection policy: The adaptive rank (r=60% vs r=80% energy retention) was better than fixed ranks, but the paper does not report a principled rule for choosing the retained proportion per layer or across backbone pairs.

Target Audience

Researchers and engineers working on video diffusion models, streaming or causal video generation, and controllable generation adapters (LoRA-style control branches). It is most useful to readers already comfortable with diffusion transformer internals, autoregressive key/value caching, and linear-algebraic analysis of parameter updates; readers looking for an introductory treatment of video editing will find the presentation dense.

Authors’ abstract

In this paper, we propose SVEET, a framework that requires merely training on a pretrained bidirectional video diffusion model but supports high-quality streaming video editing in an auto-regressive fashion. To tackle this problem, we first systematically revisit existing video-to-video diffusion approaches and identify two key principles for such streaming adaptation: backbone feature disentanglement and conditional frame independence. Building on these insights, we develop a novel paradigm for controllable video generation. At its core, an auxiliary model branch encodes source video inputs with temporally independent self-attention, and the intermediate features are injected into the corresponding backbone blocks for streaming-compatible control. Moreover, to bridge the discrepancy between the feature spaces of bidirectional and streaming models, we propose a decoupled training scheme that explicitly enforces the orthogonality between the optimization directions of video controllability and model causality. Such disentanglement ensures compatibility between the two objectives at inference and facilitates smooth zero-shot knowledge transfer across heterogeneous backbone architectures. Extensive experiments demonstrate that SVEET achieves superior editing quality while maintaining real-time performance, attaining 15 FPS on a single H100 GPU 17 without any auxiliary acceleration techniques. Codes are available at https://github.com/YujiaHu1109/SVEET.