EditaLive! Unified Character Video Editing for Live Streaming

AI summary

Overview

• Research area: Computer vision, specifically instruction-based character video editing and real-time (streaming) generative video.
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching models, distillation, autoregressive streaming generation, RoPE positional encoding, and KV-caching.
• Scope: EditaLive is a unified framework that edits the appearance of a primary character in a live video stream from a text instruction, in real time, while preserving the source character's facial expressions and body motion.

What This Paper Is About

Most video-editing research targets scene- or object-level content, but live streaming centers on the human subject, and existing editors transfer poorly to that setting. The authors identify three obstacles: synthetic training data that carries facial-expression inconsistencies, offline pipelines that need dozens of denoising steps plus bidirectional attention over complete clips, and long-term appearance drift in autoregressive streaming. EditaLive addresses all three by reformulating character editing as an appearance transformation under explicit motion control, adapted to causal streaming inference and compressed to a two-step sampler.

Key Contributions

1. A unified real-time streaming character video editing framework. EditaLive supports diverse character appearance edits together with motion-driven animation, built on a single model rather than a cascade of separate editing and animation stages.
2. An appearance–motion decoupled editing paradigm. The model is built on Wan-Animate, a pretrained image animation model that naturally separates appearance from motion. Character video editing is reformulated as appearance transformation under explicit motion control, trained via a reconstruction objective on motion-aligned real videos, with the CharEdit-50K dataset constructed to support it.
3. Causal streaming adaptation. The offline bidirectional model is converted into a chunk-wise causal generator that keeps bidirectional attention inside each chunk and restricts cross-chunk attention to preceding context, enabling continuous inference from cached history.
4. Aligned self-rollout distillation. A distillation strategy combining Align Forcing, Fixed RoPE, and First-frame Preserved Sparse Attention (FPSA) compresses the model into a two-step sampler, reduces training–inference discrepancies, and mitigates appearance drift over long generations.

Main Findings

• Instruction-following with expression preservation: In qualitative comparisons, EditaLive performs both local appearance edits and global style transformations while preserving source facial expressions, body poses, and scene structure; streaming methods either fail to fully apply the requested edits or introduce facial-expression inconsistencies, and bidirectional baselines show similar limitations at much higher inference cost.
• Ranking on CharEdit-Bench-S: The paper reports that EditaLive ranks first or second across all eight quality metrics in Table 1. Its scores include ID-SIM 0.550, AED 0.499, APD 0.124, TA 2.796, EQ 2.609, BC 2.024, SR 0.720, and Pick Score 19.61. The character-consistency metrics of LiveEdit and LucyEdit are omitted because those methods do not support global style editing.
• Efficiency on a single NVIDIA H100 GPU: EditaLive reaches 14.47 FPS end-to-end throughput with 0.829 seconds average inter-chunk latency. Dedicated streaming baselines are faster in raw terms, but their edit success rates are at most 0.428, versus 0.720 for EditaLive.
• Speedups over baselines: Compared with LucyEdit, the fastest bidirectional baseline, EditaLive delivers 4.7× higher throughput and reduces latency by a factor of 31.7. Against the cascaded Qwen-Image-Edit plus Wan-Animate baseline, even excluding the one-time reference-image editing cost, it is 11.4× faster and reduces latency by a factor of 76.7.
• Decoder swap: Replacing the standard Wan-VAE decoder with Flash-VAED increases EditaLive's inference speed to 16.4 FPS.
• Reference conditioning in causal adaptation: Optimizing both reference and target positions (Ref.+Tgt.) introduces conflicting supervision, because the reference latent must reconstruct its unchanged input while targets must apply the edit. Keeping the reference clean and computing the Flow Matching loss only over noisy target positions (Tgt. only) avoids contradictory gradients.
• Align Forcing is the single most impactful distillation component: Replacing it with Self Forcing drops ID-SIM from 0.492 to 0.252 and SR from 0.817 to 0.367, because the KV cache is built from an intermediate rollout state rather than the final prediction.
• Fixed RoPE matters for long videos: Removing it exposes the model to unseen positional offsets and weakens reference conditioning, producing progressive appearance drift and an ID-SIM of only 0.381.
• Sparse attention helps, and first-frame preservation helps further: The variant without FPSA reaches ID-SIM 0.455; sparse attention without first-frame preservation reaches 0.466 but still shows visible changes such as hair variation at frame 900; full FPSA reaches ID-SIM 0.492 and lowers AED and APD to 0.576 and 0.109.
• NFE comparison in distillation: Under two-step distillation, Self Forcing requires an average of 2.5 NFEs per training iteration, whereas Align Forcing requires 2 NFEs, with only the first chunk needing one additional t0 cache forward pass to initialize the attention sink.

Methodology in Plain English

The authors start from Wan-Animate, a pretrained model that already separates a character's appearance from the motion driving it. To turn it into an instruction-following editor, they avoid building synthetic edited videos, which tend to alter facial expressions in unintended ways. Instead, they synthesize only the reference image: an off-the-shelf image editor applies a forward instruction to a chosen frame, and a reverse instruction is written to undo that edit. The model is then trained to reconstruct the original real video when given the edited reference, the reverse instruction, and the source motion signals (a skeleton sequence and implicit facial representations). This teaches appearance transformations without disturbing genuine motion.

For this, they build CharEdit-50K from SpeakerVid-5M, the Seamless Interaction Dataset, and additional internet videos. A high-quality frame is selected per video based on facial sharpness and completeness plus hand visibility and sharpness, excluding frames with severe occlusion, motion blur, or truncation. GPT-5.5 generates candidate instruction pairs across four categories: add–remove, remove–add, change–restore, and stylization, with category and color balancing. Qwen-Image-Edit and Nano Banana 2 synthesize candidate images, and GPT-5.5 filters them for edit accuracy, target clarity, character consistency, and reverse feasibility.

Next, the bidirectional model is converted to chunk-wise causal generation: frames attend bidirectionally within a chunk of size 3, and each chunk attends only to previous clean context. Training still runs all chunks in parallel via a causal attention mask, with the loss computed only over noisy target positions.

Finally, self-rollout distillation compresses the model to two sampling steps (timesteps set to [1000, 250]). Three fixes align training with streaming inference. Align Forcing computes gradients only at the selected denoising step but completes the remaining rollout with stop-gradient, so the propagated KV cache always comes from the final prediction, matching inference. Fixed RoPE assigns positions by role rather than absolute timestamp, using indices 0–9 for the reference sink, attention sink, local-window context, and current chunk respectively, divided as 0, 1–3, 4–6, and 7–9. FPSA partitions queries and keys into spatio-temporal blocks, scores them at block level, keeps the top K = (1 − ρ)B blocks with a sparsity ratio of 75%, and always includes all key blocks from the first generated frame as a persistent appearance anchor.

Training uses LoRA (rank r and scaling factor α both 128) on 8 NVIDIA H100 GPUs with AdamW and a global batch size of 8. Stages 1 and 2 use 33-frame, 480p clips for 15K steps at a learning rate of 1×10⁻⁴; Stage 3 uses 69-frame, 480p clips for 2K steps at 2×10⁻⁶. The Stage 3 student initializes from the Stage 2 checkpoint, and its real-score and fake-score branches initialize from the Stage 1 checkpoint.

Why This Matters

The work shifts video editing from clip-level, offline post-production toward interactive, character-centric streaming, where the human subject—and specifically their facial expressions—is the thing that must not break. It also contributes a dataset construction philosophy: synthesizing only the reference image while supervising on authentic motion-aligned video avoids the expression corruption that plagues synthetic video-pair datasets.

Real-world applications:

• Live streaming overlays and persona customization: streamers applying text-prompted appearance changes (outfit colors, accessories, stylistic looks) on the fly without changing how they actually move or emote.
• Virtual avatars and VTubing: driving a customized character with the performer's own body motion and facial dynamics via the motion-driven animation capability.
• Video conferencing and telepresence: consistent real-time appearance adjustments while retaining genuine expression cues, with sub-second inter-chunk latency.
• Content production and post-production: applying a single instruction across a long take without the flicker of frame-by-frame editing or the drift of naive autoregressive streaming.

Industry relevance is direct for platform and device companies: the affiliations include vivo BlueImage Lab, and the engineering constraints the paper targets—low latency, bounded historical context, a two-step sampler, and a faster VAE decoder—are deployment concerns rather than purely academic ones.

Future Directions

• Dataset scale and diversity: The provided content names CharEdit-50K but does not report per-category sample counts, source-video duration distributions, or how the dataset generalizes beyond its source pools (SpeakerVid-5M, Seamless Interaction, internet videos).
• Pushing beyond two-step sampling and 75% sparsity: The ablation shows FPSA already trades quality against history length; the paper does not report results for more aggressive sparsity ratios or a one-step sampler.
• Wider motion conditions and shot changes: The framework relies on extracted skeleton sequences and implicit facial representations; behavior under camera cuts, occluded hands, or atypical body types is not reported in the provided content.
• Evaluation under real streaming conditions: Metrics are computed on CharEdit-Bench-S and CharEdit-Bench-L; it is not reported how performance holds under real network jitter, variable frame rates, or multiple simultaneous viewers, or how the 0.829-second inter-chunk latency translates to perceived audio-visual synchronization.

Target Audience

Researchers and engineers working on video diffusion models, real-time generative media, and virtual avatars will get the most from this paper, particularly those interested in distillation for few-step streaming inference, causal attention conversion, and appearance–motion decoupling. It is also relevant to dataset builders working on instruction-based video editing and to product teams evaluating generative character editing for live streaming platforms. Readers without a background in flow matching, KV-cache mechanics, or autoregressive video generation will find the implementation sections dense.

Authors’ abstract

Conventional video editing primarily focuses on scene-level content, whereas live streaming places greater emphasis on the human subject. However, directly applying existing video-editing methods to human-centric live streaming remains challenging, as they may introduce facial-expression inconsistencies and typically depend on multiple offline inference steps, making them unsuitable for real-time interaction. We propose EditaLive, a novel framework for real-time streaming character video editing. In detail, we start from a pretrained image animation model (Wan-Animate), which naturally decouples appearance from motion, and repurpose it as the base model for instruction-based human-centric video editing by reference frame editing and video reconstruction via the collected CharEdit-50K dataset. Besides, we adapt the model from offline bidirectional to causal streaming generation, and design an aligned self-rollout distillation strategy that compresses the model into a two-step sampler, where fixed RoPE and align forcing reduce training--inference discrepancies, and first-frame preserved sparse attention filters redundant historical information to mitigate appearance drift. Extensive experiments demonstrate that EditaLive delivers state-of-the-art editing performance with faithful preservation of facial expressions and low-latency real-time streaming inference.