MotionStream: Real-Time Video Generation with Interactive Motion Controls

AI summary

Overview

Research area: controllable video generation, specifically real-time streaming video diffusion with interactive motion control.

Technical level: Advanced. The paper assumes familiarity with diffusion models, distillation, autoregressive generation, KV caching, and attention mechanisms.

Scope: A single paper describing a motion-conditioned video generation system that runs in real time on one GPU and can stream arbitrarily long videos under user-drawn trajectory control.

What This Paper Is About

Existing motion-controlled video generators are slow (the paper cites 12 minutes to synthesize a 5-second clip with Motion Prompting) and non-causal, meaning they need the entire motion specification up front and cannot show partial results during generation. MotionStream aims to make motion-controlled video generation interactive by streaming frames out in real time as the user provides trajectory, camera, or drag controls, while sustaining that generation indefinitely without quality degrading.

Key Contributions

1. A streaming motion-conditioned video generation pipeline reported to run at 29.5 FPS on a single H100 GPU, described by the authors as the first of its kind and fast enough for real-time interaction.
2. A system design combining a lightweight track head and conditioning modules with a distillation process that folds joint text-motion guidance into the training objective, plus a Tiny VAE for faster decoding.
3. A distillation strategy for long-video generation that applies attention sinks and local attention with extrapolation-aware training, which the authors state is explored systematically for the first time and prevents drift during extended streaming.
4. State-of-the-art reported results on motion transfer and camera control at orders-of-magnitude faster speeds, including zero-shot generalization to 3D novel view synthesis.

Main Findings

• Speed: MotionStream reaches 17 FPS at 480P and 10 FPS at 720P on a single H100 GPU with sub-second latency, and up to 29 FPS with the Tiny VAE decoder. The distillation makes the system "two orders of magnitude faster" than prior motion-conditioned methods.
• Motion transfer quality: On the DAVIS validation set (30 videos), the distilled causal Wan 2.1-1.3B model achieves PSNR 16.20, SSIM 0.447, LPIPS 0.443, and EPE 7.80, versus the teacher's 16.61 / 0.477 / 0.427 / 5.35. On the 20-video Sora subset, the same distilled model scores 16.67 / 0.531 / 0.360 / 4.21 compared with the teacher's 17.82 / 0.586 / 0.333 / 2.71.
• Baseline comparison: Distilled MotionStream models beat prior methods such as Image Conductor, Go-With-The-Flow, Diffusion-As-Shader, and ATI on motion-following metrics while running at 16.7 FPS (480P) and 10.4 FPS (720P), where baselines run between 0.23 and 2.98 FPS.
• Camera control: On the LLFF dataset, the causal 1.3B model reaches PSNR 15.7, SSIM 0.38, LPIPS 0.23 at 16.7 FPS, and the causal 5B model reaches 15.0 / 0.39 / 0.23 at 10.4 FPS, outperforming DepthSplat, ViewCrafter, and SEVA (which score PSNR 13.9, 14.0, 14.1 respectively) while being more than 20 times faster.
• Track representation: Sinusoidal positional-encoding tracks with a learnable head (PE-Head) take 24.8 ms versus 1053 ms for RGB-VAE encoding, roughly 40 times faster, while also improving PSNR/SSIM/LPIPS/EPE (e.g., DAVIS PSNR 16.29 vs 16.03, EPE 6.54 vs 8.57).
• Attention sink and window: The minimal configuration of one sink chunk and a one-chunk window (c3s1w1) performs best. Adding sinks gives only marginal gains while raising latency, and larger windows actually degrade performance. Removing the sink (c3s0w1) speeds things up slightly but harms long-term stability.
• Chunk size trade-off: A chunk size of 1 causes significant quality degradation (LPIPS 0.597, EPE 76.21 on Sora Extended), while chunks larger than 3 add latency unsuitable for real-time interaction, so chunk size 3 is selected.
• Guidance balance: Joint text-motion guidance with w_t = 3.0 and w_m = 1.5 balances trajectory adherence against natural dynamics. Pure motion guidance gives the best trajectory accuracy but rigid, simplistic motion; text guidance allows effects like weather changes or object appearances.
• Stochastic masking: A masking probability p_mask = 0.2 during fine-tuning addresses the ambiguity between occluded tracks and unspecified tracks, reducing artifacts where objects abruptly appear or disappear.
• Tiny VAE: A compact decoder improves Wan 2.1 from 16.7 FPS with 0.69 s latency to 29.5 FPS with 0.39 s latency, and Wan 2.2 from 10.4 FPS with 1.1 s latency to 23.9 FPS with 0.49 s latency. Full VAE decoding consumed 47% of chunk generation wall time for Wan 2.1 (1.3B) and 35% for Wan 2.2 (5B).
• Latency stability: The chosen c3s1w1 configuration keeps latency (0.70 ± 0.01 s) and throughput (16.92 ± 0.80 FPS) stable, whereas a vanilla sliding window shows larger fluctuations (0.80 ± 0.08 s latency, 14.96 ± 1.42 FPS).

Methodology in Plain English

The authors work in two stages. First, they take an existing image-to-video diffusion model (Wan 2.1 at 1.3B parameters and Wan 2.2 at 5B) and teach it to follow motion. Each 2D track gets a sinusoidal embedding, the authors place those embeddings at downsampled spatial positions to build a conditioning signal, and a small track head compresses them and concatenates them with the video latents, avoiding the doubled compute of a ControlNet-style design. This teacher is trained with a flow matching objective on OpenVid-1M plus synthetic data from Wan text-to-video models (70K samples at 832×480 and 30K at 1280×704), using tracks from a 50×50 grid extracted with CoTracker3. Because zero-valued tracks can mean either "occluded" or "not specified," the authors fine-tune with random mid-frame masking at p = 0.2. They also combine text and motion guidance so that text supplies natural dynamics while tracks enforce trajectory adherence.

Second, they distill the slow bidirectional teacher into a fast causal student. They adapt the student to causal attention and few-step trajectories using regression on ODE pairs from the teacher, then apply Self Forcing-style autoregressive rollout with distribution matching distillation. The real score comes from the frozen teacher with joint guidance, while the fake score comes from a trainable critic, which lets the student "bake in" guidance at a single function evaluation. The generator and critic are updated at a 1:5 ratio, and gradients flow only through one randomly sampled denoising step to save memory, with cached keys and values treated as stop-gradient.

The distinctive part is how they handle long videos. Inspecting attention maps, they found many heads persistently attend to the initial frame's tokens, similar to observations in large language models. They therefore keep the initial frame's tokens as a fixed "sink" while rolling a fixed-size local window of recent chunks through a KV cache. Because RoPE positions are assigned by cache slot rather than absolute frame index, and because this exact mechanism (self-rollout with rolling cache and sinks) is used during training as well as inference, there is no train-test mismatch and cost stays constant no matter how long the video runs. Finally, a Tiny VAE decoder trained from scratch with adversarial and LPIPS losses removes the decoding bottleneck.

Why This Matters

The paper argues that interactive motion control has been blocked less by quality than by latency, causality, and duration limits. By making generation streaming and constant-cost, MotionStream turns video generation from a "render-and-wait" loop into something closer to live operation, and the authors show the same model transfers to tasks it was not trained for, such as zero-shot 3D novel view synthesis.

Real-world applications the paper demonstrates or implies:

• Motion transfer, using tracks from offline or online trackers to drive a performance in real time.
• Mouse-based drag control, where a user paints a trajectory and sees the object follow immediately.
• Camera control, including 3D camera moves derived from depth and camera parameters.
• Interactive demos in which user input during streaming alters the output as it unfolds, including text-driven effects like a rainbow appearing alongside a dragged object.

Industry relevance: the results are reported on a single H100 GPU, and the Tiny VAE work shows decoding (not just diffusion) is a real bottleneck in production streaming pipelines. Real-time, controllable video generation is directly applicable to content creation tools, visual effects, gaming and virtual production, and interactive media, which is consistent with the author affiliations spanning Adobe Research, Seoul National University, Carnegie Mellon University, and Morpheus AI.

Future Directions

• Push beyond the 480P and 720P settings tested here while keeping latency sub-second, since the 5B 720P causal model runs at 10.4 FPS before VAE optimization.
• Investigate whether the minimal one-sink, one-window configuration remains optimal at longer horizons or higher resolutions; the paper reports that larger windows degrade performance because errors accumulate in context tokens, which is not fully explained.
• Address the ambiguity between occluded and unspecified tracks more fundamentally, since the current fix is a masking heuristic with p_mask = 0.2.
• Read Appendix F for the limitations and future directions the authors say they discuss there; the truncated content provided does not include those details, nor the ethics and reproducibility statements in Appendix G.

Target Audience

Researchers and engineers working on video diffusion, autoregressive generative models, and real-time inference systems will benefit most, particularly those interested in distillation, KV-cache and attention design for long-horizon generation, or efficient decoders. Practitioners building interactive creative tools, virtual production pipelines, or game and simulation engines will also find the practical latency and throughput numbers directly relevant. Readers without a background in diffusion models, distillation, and attention mechanisms will find the technical sections difficult, though the problem framing in the introduction is broadly accessible.

Authors’ abstract

Current motion-conditioned video generation methods suffer from prohibitive latency (minutes per video) and non-causal processing that prevents real-time interaction. We present MotionStream, enabling sub-second latency with up to 29 FPS streaming generation on a single GPU. Our approach begins by augmenting a text-to-video model with motion control, which generates high-quality videos that adhere to the global text prompt and local motion guidance, but does not perform inference on the fly. As such, we distill this bidirectional teacher into a causal student through Self Forcing with Distribution Matching Distillation, enabling real-time streaming inference. Several key challenges arise when generating videos of long, potentially infinite time-horizons -- (1) bridging the domain gap from training on finite length and extrapolating to infinite horizons, (2) sustaining high quality by preventing error accumulation, and (3) maintaining fast inference, without incurring growth in computational cost due to increasing context windows. A key to our approach is introducing carefully designed sliding-window causal attention, combined with attention sinks. By incorporating self-rollout with attention sinks and KV cache rolling during training, we properly simulate inference-time extrapolations with a fixed context window, enabling constant-speed generation of arbitrarily long videos. Our models achieve state-of-the-art results in motion following and video quality while being two orders of magnitude faster, uniquely enabling infinite-length streaming. With MotionStream, users can paint trajectories, control cameras, or transfer motion, and see results unfold in real-time, delivering a truly interactive experience.