JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion

AI summary

Overview

Research area: Computer vision — instruction-guided video editing, specifically streaming/causal video-to-video (V2V) editing built on autoregressive diffusion models.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, flow matching, distribution matching distillation (DMD), classifier-free guidance, KV caching, and autoregressive causal attention.

Scope: This paper describes a 16B-parameter autoregressive diffusion system that edits video in a streaming, open-ended fashion (no future frames, no predefined duration) at approximately 30 FPS for 720p output on a single Nvidia B200 GPU, and validates it on short-video, long-video, human, and latency evaluations.

What This Paper Is About

Most high-quality video editors are offline: they process a fixed-length clip using bidirectional or global temporal attention and only emit output after iterative denoising finishes, which means they need future frames, their memory grows with input length, and they cannot produce frames causally. This paper's goal is a streaming editor that emits edited frames as the source video arrives, with low latency, bounded computation, and stable quality over an unknown duration. The authors argue that simply causalizing an offline editor is not enough, because autoregressive models trained on clean target history drift when they must consume their own imperfect predictions at inference time.

Key Contributions

1. A 16B-parameter autoregressive diffusion framework for real-time open-ended video editing. The system combines an MLLM condition encoder, a causal video VAE, and a multimodal diffusion transformer (MM-DiT), and supports both instruction-guided V2V editing and reference-conditioned image-and-video-to-video (IV2V) editing. It operates without access to future frames or a predefined video duration.

2. Chunk-wise autoregressive adaptation with bounded temporal state. The bidirectional editor is converted into a chunk-wise causal editor with bidirectional attention inside each chunk and causal attention across chunks. Cross-chunk attention is restricted to a sliding window of recent chunks plus the first chunk, which is retained as a global sink, so the temporal state and per-chunk compute stay constant regardless of stream length.

3. Source-Anchored Distribution Matching Distillation (SA-DMD). This distills the iterative diffusion process into a two-step generator. It applies classifier-free guidance to the real-score teacher along independent text-conditioning and source-fidelity axes, using each temporally aligned source chunk to counteract drift from imperfect autoregressive history. The source-aware guidance is applied only to the distillation target and absorbed into the generator, so deployment uses a single conditional branch.

4. Long-Horizon Autoregressive Distillation (LHAD) and the LongV2VBench benchmark. LHAD performs segmented optimization over extended multi-chunk rollouts, computing the SA-DMD backward pass per clip, clearing the graph, and accumulating gradients for one optimizer step under bounded memory. LongV2VBench is a constructed one-minute video-editing benchmark of 229 tasks across five categories.

Main Findings

• Short-video editing (OpenVE-Bench): JoyAI-Video-Edit achieves an overall score of 3.60, outperforming SANA-Streaming (2.62) by 0.98, LiveEdit (2.00) by 1.60, XMax-X2.0 (1.87) by 1.73, and StreamDiffusionV2 (1.23) by 2.37 points. It ranks first among streaming methods in four of the five categories, with particularly large gains in local change and local removal. It is described as comparable to strong offline editors such as Kiwi-Edit (3.02), Bernini-R (3.72), Kling-3.0 Omni (3.64), and Kling-O1 (3.62).

• Highest local-removal score overall and tied best local-change score: On OpenVE-Bench, the model scores 4.06 for local removal, the highest among all compared methods, and 4.47 for local change, matching the best score (Bernini-R also reports 4.47).

• Long-video editing (LongV2VBench): The model achieves an overall score of 3.30 and ranks first in all five editing categories, exceeding the strongest baseline XMax-X2.0 (1.71 overall) by 1.59 points. Category scores are 2.49 background change, 3.85 global style, 3.10 local add, 4.09 local change, and 2.99 local remove.

• Full-pipeline throughput of 30.19 FPS: On LongV2VBench, JoyAI-Video-Edit runs at 30.19 FPS at 720×1280, reported as 44.4% faster than XMax-X2.0 (20.90 FPS) at a substantially lower resolution, and more than twice as fast as SANA-Streaming (14.51 FPS) at a comparable resolution.

• Human preference: In pairwise evaluation on single-shot videos shorter than 10 seconds, JoyAI-Video-Edit receives 90% of preference votes against LiveEdit, 87% against SANA-Streaming, 81% against XMax-X2.0, and 87% against StreamDiffusionV2. Against Bernini-R it receives 48% versus Bernini-R's 44%. Kling-3.0 Omni and Seedance 2.0 each receive 56% of preference votes in their respective comparisons.

• Deployment latency: On 81-frame inputs with batch size 1, the complete pipeline measures 2.68 s latency and 30.19 FPS, versus StreamDiffusionV2 (4.48 s, 18.07 FPS), LiveEdit (5.24 s, 15.45 FPS), and SANA-Streaming (5.58 s, 14.51 FPS). This is 1.67× StreamDiffusionV2's throughput, 1.95× LiveEdit's, and 2.08× SANA-Streaming's. The VAE component reaches 200.00 FPS (0.405 s), while the DiT component measures 37.21 FPS (2.18 s). Notably, SANA-Streaming has a faster standalone DiT in this evaluation (54.36 FPS).

• Per-stage runtime on a single Nvidia B200: VAE encoding 22 ms, DiT denoising 185 ms, and VAE decoding 19 ms per chunk, giving a request-to-response latency of 226 ms. Clean KV-cache construction adds 31 ms and pseudo encoding adds 9 ms, making the complete 266-ms cycle equivalent to 30.1 FPS.

• Ablation of the two distillation components (LongV2VBench): The baseline without either component scores 2.81 overall. Adding SA-DMD alone raises the score to 3.23, lifting global style from 3.61 to 4.24 and local change from 3.43 to 4.00. Adding LHAD alone raises the overall score to 3.06, particularly benefiting background change and local removal. Enabling both gives the best overall score of 3.30 and dominates all three local-editing tasks.

Methodology in Plain English

The authors start from a strong bidirectional editor and progressively make it causal and fast.

Building the base model. They first pretrain a text-to-image model at 256×256 and then 512×512 to establish visual priors, then initialize text-to-video training from that checkpoint, keeping the T2I objective alive so spatial quality does not degrade. Video training starts at low resolution (256p at 12 and 24 fps) and 360p, then supervised fine-tuning and continual training at 480p. Image-editing supervision (I2I) is layered in to teach instruction-conditioned transformation and content preservation, and this checkpoint initializes bidirectional video-editing training on 720p V2V and IV2V data, jointly optimized with all preceding tasks.

Making it causal. Each editing pair is split along time into aligned source and target chunks (one latent frame per chunk in their implementation). Attention is bidirectional inside a chunk but causal across chunks. To keep cost flat, an active chunk attends only to a fixed number of recent chunks and to the first chunk as a global sink, so the attention context is constant no matter how long the video runs. During training they pack the noised active chunk together with source tokens, condition tokens, optional reference tokens, and clean history tokens, with a mask controlling visibility. Following Diffusion Forcing, each target chunk gets an independent noise level under a masked flow-matching objective. Because teacher forcing on clean history mismatches streaming inference, they follow Resampling Forcing and replace clean history with an on-policy estimate: each historical chunk is regenerated via a single-step denoising rollout and detached from gradients.

Making it fast without losing the source. Standard Distribution Matching Distillation pulls a student generator toward a teacher using the discrepancy between real and fake score models. The authors' addition, SA-DMD, guides the real-score teacher with classifier-free guidance on two separate axes: text condition and source fidelity. Because the guidance is applied only to the training target and gets absorbed into the generator, the deployed model preserves source fidelity through a single conditional forward pass. The system uses a causal generator, a trainable fake-score model, and a frozen real-score model, all initialized from the previous stage under a shared-backbone LoRA configuration, and produces a two-step generator.

Stabilizing long rollouts. Short rollouts do not capture the compounded errors of long-horizon inference. LHAD divides a long sequence into shorter consecutive clips, computes the SA-DMD backward pass per clip, clears the graph, and accumulates gradients across clips for a single optimizer step, reflecting the full horizon's gradient within bounded memory. When target rollouts exceed the available source video, a dynamic mirror looping strategy alternates forward and reversed sequences to preserve temporal continuity without duplicating tensors.

Deployment. The incoming stream is divided into consecutive eight-frame chunks, each encoded by the causal VAE, edited by the few-step DiT using the current condition and cached history, then immediately decoded. FP8 quantization, operator fusion, computation-graph compilation, pipelined execution, startup warm-up, persisted compilation artifacts, and retained memory pools reduce overhead. Clean KV states are cached for later chunks, retaining the first chunk as a global sink plus a sliding window of recent chunks.

Data. T2V data comes from diverse image and video sources covering people, lifestyle, entertainment, nature, objects, and urban scenes, filtered for sharpness, aesthetic score, motion magnitude, camera stability, and temporal validity, then clustered and downsampled to avoid overrepresenting frequent concepts, with manual inspection for high-quality fine-tuning data. Because paired video-editing data is hard to collect at scale, they transfer image-editing supervision from JoyAI-Image using two pipelines: (1) edit a representative keyframe, then propagate the edit across the video with an image-and-video-to-video model, and (2) generate paired videos from an original image and its edited counterpart using latent-shared I2V generation where both branches share early denoising latents. Pairs are filtered for quality, editing correctness, content preservation, and temporal consistency, then an MLLM compares source and edited videos and refines the instructions.

Why This Matters

The paper reframes video editing as a continuously available capability rather than a fixed-clip post-production step, and shows that a 16B model can sustain 720p editing at roughly 30 FPS on a single GPU. Its distinctive technical claim is that the hard part is not causalization alone but balancing three competing conditions — the current source chunk, the generated history, and the edit instruction — and that source-anchored distillation plus long-horizon optimization are highly complementary ways to do so.

Real-world applications the paper names:

• Live broadcasting and video communication, where edited frames must be produced as the source arrives.
• Real-time digital humans, telepresence, and embodied simulation.
• Interactive entertainment and game content creation, where users apply or revise instructions during playback rather than waiting for a full render.
• Real-time content creation platforms, where continuous preview and instruction-level control shorten iteration cycles and enable personalized visual effects at scale.

Industry relevance: The reported numbers matter for deployment economics — 30.19 FPS at 720×1280 on a single B200, with a 266-ms complete cycle broken down into measurable components, plus FP8 quantization, graph compilation, and bounded KV caching. The B2B2C framing (platforms and enterprises applying adaptive content transformation at scale) and the open-sourced code at https://github.com/jd-opensource/JoyAI-Video-Edit position this as an engineering-oriented system contribution, not only a modeling one.

Future Directions

• Pushing resolution and latency further. The system targets 720p at approximately 30 FPS. Whether the same chunk-wise causal design and two-step distillation scale to higher resolutions while preserving the 266-ms cycle is not established here.

• Beyond one-minute evaluations. LongV2VBench covers 229 one-minute tasks; the paper notes that existing benchmarks predominantly consist of clips shorter than 10 seconds and are insufficient for assessing error accumulation and temporal degradation. How quality holds over substantially longer, open-ended streams — the stated target of the method — remains an open question.

• Interaction and control. The paper frames the system as enabling users to revise instructions during playback, but the evaluation protocols (OpenVE-Bench, LongV2VBench, and the human study) are single-shot editing comparisons. Multi-turn, interactive editing sessions are not evaluated.

• Closing the remaining gap to offline editors. Automatic results are described as competitive with offline systems, and human preference against Bernini-R is 48% versus 44%, while Kling-3.0 Omni and Seedance 2.0 each receive 56%. The remaining quality gap in specific categories (for example, local addition scored 3.10 on LongV2VBench and 2.97 on OpenVE-Bench) suggests targeted improvement is still needed.

Target Audience

Researchers working on video diffusion, autoregressive and streaming generation, and few-step distillation; systems and deployment engineers interested in real-time generative video pipelines (FP8 quantization, KV caching, graph compilation); and developers or product teams building interactive video editing, live broadcasting, telepresence, or digital-human applications. Readers without background in diffusion sampling, flow matching, and distillation will find the training sections dense, though the architecture and results sections are comparatively accessible.

Authors’ abstract

Real-time video editing requires low-latency causal generation with bounded computational resources while preserving source fidelity and long-term temporal consistency. We present JoyAI-Video-Edit, a 16B-parameter autoregressive diffusion framework for real-time, open-ended video editing without access to future frames or a predefined video duration. Our method combines chunk-wise autoregressive adaptation, Source-Anchored Distribution Matching Distillation (SA-DMD), and Long-Horizon Autoregressive Distillation to reduce train--inference mismatch, preserve source fidelity during two-step generation, and mitigate accumulated temporal drift. Extensive automatic and human evaluations show that JoyAI-Video-Edit substantially outperforms existing streaming editors and remains competitive with strong offline systems on both short and long videos. The complete system achieves end-to-end 720p video editing at approximately 30 FPS on a single Nvidia B200 GPU. Code is available at https://github.com/jd-opensource/JoyAI-Video-Edit.