Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation

AI summary

Overview

Research area: Generative computer vision, specifically real-time interactive video generation, streaming video editing, and stereoscopic (spatial) video synthesis.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, autoregressive/streaming inference, distillation objectives (DMD), preference optimization, and GPU kernel-level acceleration.

Scope: A single sentence — Vidu S2 is a two-part real-time video system (an interactive digital-character model and an instruction-guided streaming editor) that runs at 720p and 25–42 FPS and can optionally output synchronized stereo video for VR headsets.

What This Paper Is About

Most state-of-the-art video generators (Sora, Veo, Wan, Seedance) work offline: a user submits a prompt, waits minutes, and receives a finished clip with no opportunity to intervene. That paradigm cannot support live conversation, streaming, games, or face-to-face interaction, where content must respond to the user immediately and can be redirected at any moment.

The paper asks whether high-quality video generation and editing can be made fully interactive — generated chunk by chunk, steered in real time, and delivered as spatial video — without the drift, blurring, and quality collapse that normally plague long autoregressive rollouts.

Key Contributions

1. Vidu S2-Avatar, a real-time interactive digital-character model that raises output from Vidu S1's 540p to 720p, supports reference images that can be swapped at any point mid-stream, and follows a wider range of instructions including large body motion such as dancing.

2. Vidu S2-Editing, a model that edits an incoming video stream on the fly for style transfer, virtual try-on, character replacement, and background replacement, using frame-aligned attention so the edited output inherits the source's exact motion and timing.

3. Real-time spatial video generation and editing, converting avatar-generated or edited monocular streams into synchronized left/right eye views, or jointly editing existing stereoscopic input for delivery to VR head-mounted displays.

4. A training and serving stack combining Self-Replay Forcing (an on-policy distillation method that allows gradient flow across autoregressive block boundaries), layer-wise hybrid attention, per-block W8A8 quantization, kernel fusion, CUDA Graphs, and multi-GPU context parallelism with quantized communication.

Main Findings

• Resolution and speed improved together: Vidu S2-Avatar generates 720p real-time video at 25–42 FPS, up from 540p in Vidu S1, achieved by keeping the backbone at low resolution and adding a single-step latent-space Refiner rather than making the backbone slower.

• Self-Replay Forcing outperforms Self-Forcing: Prior self-forcing feeds self-generated history back clean and detached from the computation graph. SRF re-noises the entire student-generated trajectory along Diffusion Forcing and replays it in one gradient-enabled causal pass, so a later segment's loss can shape preceding segments. This improves data efficiency and training quality without backpropagating through the original rollout.

• Asymmetric cache noise separates motion from detail: The backbone attends to a high-noise historical cache for long-range temporal structure, while the Refiner attends to a low-noise high-resolution cache. This division lets the backbone preserve motion consistency while the Refiner restores fine detail without disturbing the trajectory.

• Instruction following extends to whole-body motion: Adding solo dance videos and 2D/3D animation, stabilizing backgrounds instead of discarding videos with camera movement, and captioning events in chronological order (with temporal boundaries and outcomes) lets the model handle dancing and other large motion that Vidu S1 could not.

• Frame-aligned attention preserves timing in editing: Each target frame reads only the source frame at the same timestep, while the reference image stays visible to all frames. Edited video therefore keeps exactly the input's motion and timing, and source frames can be discarded from the cache after use.

• Style transfer generalizes broadly: The reconstruction-based data pipeline, built on surface-normal video conditioned on a reference image, generalizes to more than 50 photorealistic and non-photorealistic styles, including cel shading, cyberpunk, Monet-style painting, sketching, and Chinese gongbi painting.

• Reported superiority over all baselines: The authors state that Vidu S2 outperforms every baseline while still meeting real-time inference constraints, with a playable online demo available.

• Demand argument for interactivity: Because offline videos are replayed and shared, their generation demand scales roughly as β×N/m (m = average views), while interactive demand scales as α×N. With m > 100, real-time interactive generation represents a far larger demand pool.

Methodology in Plain English

The system is built by first training a standard bidirectional diffusion transformer that can generate video (and audio, for the avatar model) from a reference image and per-segment captions. This teacher model is high quality but not streamable.

To make it streamable, the authors replace bidirectional temporal attention with a block-wise causal mask, so each segment can only see the reference plus its own past. They train this causal model with a mix of Teacher Forcing (clean ground-truth history) and Diffusion Forcing (deliberately noised history) to build robustness against its own accumulated errors.

The central training innovation is Self-Replay Forcing. The student generates a long rollout on its own; that rollout is detached so gradients do not flow through it. Then the whole self-generated trajectory is re-noised at various levels and replayed in a single differentiable pass. Because all replayed blocks sit in one computation graph, the loss on a late segment can still influence earlier ones — recovering cross-block credit assignment that Self-Forcing loses, without paying the memory cost of the full rollout. A perceptual loss is added to prevent mode collapse.

Quality is then sharpened with preference optimization at two stages: diffusion-based DPO on the bidirectional teacher, and Streaming Negative-aware Fine-Tuning on the causal student, both using self-generated trajectories so the optimization matches inference-time conditions.

For the editor, the same recipe is applied, but conditioning is source video plus optional reference image plus a text instruction. The source and target streams interact only at aligned timesteps.

On the systems side, the authors pick a different attention kernel per layer based on sensitivity (SageAttention, SpargeAttention, or Sparse-Linear Attention), quantize linear layers with per-block W8A8 GEMM to handle outliers, fuse operators into custom Triton/CUDA kernels, replay stable execution sequences via CUDA Graphs, and split work across GPUs with Ulysses-style context parallelism using quantized tensor exchange.

For spatial output, the monocular frames are depth-estimated, converted to horizontal disparity, and warped in opposite directions to form left and right views, with hole filling and temporal depth stabilization. For stereo input, left and right views are concatenated horizontally, edited in one pass with shared references, then split back apart.

A vision-language model agent wraps the avatar model: it transcribes speech, writes structured prompts covering identity, expression, pose, and held objects, inspects generated frames to judge whether an action completed, and issues follow-up prompts that preserve state (for example, continuing to hold a cup while smiling).

Why This Matters

Impact on research. The paper targets the gap between offline one-shot video generation and interactive media, and it contributes two reusable technical ideas: Self-Replay Forcing, a general recipe for training streaming autoregressive generators with cross-block gradients, and frame-aligned attention for streaming video editing that decouples edit content from source motion. It also demonstrates that real-time 720p and stereo output are feasible on ordinary GPUs, which reframes the practical target for the field.

Real-world applications.

• Live streaming and virtual presence, where a digital character responds to viewer speech and can change outfit, background, or held objects mid-broadcast.
• Video conferencing and telepresence with real-time background replacement, character replacement, or stylization of the camera feed.
• E-commerce virtual try-on, letting shoppers see clothing applied to a live video stream with the reference garment supplied on demand.
• Immersive VR/AR entertainment, where generated or edited characters are delivered as stereo video to head-mounted displays with depth cues preserved.

Industry relevance. The demand argument and the 720p/25–42 FPS figures on low-cost GPUs make a direct case for deploying this class of model as a consumer product rather than a batch service. The inference stack — hybrid attention, per-block quantization, kernel fusion, CUDA Graphs, quantized multi-GPU parallelism — is itself an engineering contribution for anyone serving diffusion models under latency budgets.

Future Directions

• Closing the interaction loop further. The current system relies on a VLM agent that inspects frames and revises prompts; end-to-end learned state tracking (holding objects, maintaining identity across long streams) without an external agent remains open.
• Extending beyond 720p. The backbone deliberately runs at low resolution with a one-step Refiner. Whether the same split scales to 1080p or 4K at interactive frame rates is untested.
• Broadening the editing vocabulary. The four supported tasks (style transfer, try-on, character replacement, background replacement) are a fixed set; more compositional and multi-edit instructions are not evaluated.
• Stereo quality and evaluation. Depth-based warping introduces holes and boundary artifacts, mitigated only by lightweight streaming heuristics. Objective benchmarks for streaming and spatial video generation, beyond qualitative demos, are largely absent.
• Long-horizon consistency and cost. Drift resistance over very long streams, and the actual serving cost per concurrent user under the multi-GPU configuration, are not reported in the available text.

Target Audience

Researchers and engineers working on diffusion-based video generation, streaming and autoregressive generative models, or efficient inference systems will get the most from this paper. It is also directly relevant to product teams building digital humans, live-streaming avatars, video-editing tools, and VR/AR immersive experiences, and to practitioners who need concrete techniques for serving large generative models at interactive latency on commodity GPUs.

Authors’ abstract

We present Vidu S2, which comprises Vidu S2-Avatar, a real-time interactive digital-character model, and Vidu S2-Editing, a real-time video editing model. Moreover, we explore the feasibility of real-time spatial video generation for both Vidu S2-Avatar and Vidu S2-Editing. Compared with Vidu S1, Vidu S2-Avatar supports real-time 720p video generation, generation with dynamic references that can be updated at any moment, and stronger instruction following, such as dancing. Vidu S2-Editing supports editing a video stream in real time, including style rendering, clothing replacement, character replacement, and background replacement. Experiments show that Vidu S2 outperforms all baselines. A playable online demo is available at https://vidu.com/vidu-stream.