DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution

AI summary

Overview

Research area: Computer vision / generative modeling — specifically native joint audio-video generation, reinforcement-learning post-training, and high-resolution video refinement.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching transformers, cross-attention, LoRA, distribution-matching distillation, and RL fine-tuning of generative models.

Scope: A single paper describing DreamX-Creator 1.0, a 7B open-weight system that jointly generates synchronized audio and video from a first frame plus a text prompt, and refines the video output to 2K resolution with an autoregressive 1-step refiner.

What This Paper Is About

Most video generators either produce no audio or generate audio in a separate stage after the video, which prevents the two modalities from informing each other. This paper builds a compact (7B) generator that denoises audio and video streams together inside one network, so that visible events and their sounds are produced as a single joint process. It then adds reinforcement learning with modality-aware feedback and a 2K refinement stage, releasing both the generator and the refiner openly.

Key Contributions

1. Audio-Video Data System. A unified pipeline that builds and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools for targeted training.

2. Native Joint Audio-Video Generation. A dual-stream generator with Gated Cross-Modal Attention and Progressive Joint Training, followed by Audio-Video Reinforcement Learning using Modality-Aware Multimodal Feedback.

3. Autoregressive 1-Step 2K Refinement. A bidirectional multi-step teacher is adapted into an autoregressive multi-step refiner and then distilled into a 1-step student requiring one denoising evaluation per temporal chunk.

4. A Compact 7B Open-Weight Model. The authors release both the 7B generator and the 2K Refiner, and state that the generator is the smallest open-weight native joint model by disclosed total backbone size in their capability comparison table.

Main Findings

• Joint generation is competitive with open systems. The abstract states that DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems, and the contributions section describes "top-tier benchmark performance." The numeric benchmark tables are not present in the provided content, so no specific scores can be reported here.

• Smallest disclosed open joint model. In Table 1, among systems with a disclosed total backbone size, DreamX-Creator 1.0 (7B) is the smallest model that combines downloadable weights, native joint audio-video generation, and an officially supported local path at 2K or above. For comparison, the table lists LTX-2.3 at 22B (open, joint A/V, local up to 4K), MOVA at 32B total / 18B active (open, joint A/V, no 2K path), LingBot-Video at 30B / 3B (open, video only), MAGI-2 Preview at 114B / 6B (open, joint A/V, no 2K path), MiniMax H3 at 33B (base only), and Seedance 2.5 and Kling AI 3.0 with parameter counts not disclosed and weights not released.

• Speech and event sounds dominate the training data. Figure 4 is reported to show that speech accounts for 45.0% of the training data, while event sounds constitute another 33.4%, covering human actions, object interactions, transportation, and other physical events. The remainder covers music, natural sounds, and mixed content.

• Cross-modal interaction is deliberately delayed. The first half of the network processes audio and video streams independently; only the latter half adds paired audio-to-video (A2V) and video-to-audio (V2A) paths through Gated Cross-Modal Attention. Cross-modal positions are mapped to a shared temporal coordinate system with temporal rotary position encoding applied to cross-modal queries and keys, avoiding resampling of either latent sequence.

• A2V and V2A are internal paths, not separate inference tasks. Training mixes A2V, V2A, and Joint conditioning configurations throughout, distinguished by direction masks and relative corruption ordering. In directional modes the target stream is noisier and the conditioning stream cleaner, with stop-gradient applied before the key and value projections so the target-stream loss cannot update the conditioning backbone through cross-modal attention.

• Output gating rather than token routing. Each enabled cross-modal path uses a token- and head-wise sigmoid output gate conditioned on both the target hidden state and the cross-modal attention output, scaling each attention-head output before head concatenation, output projection, and residual addition. The gate and the direction mask act at different levels: the gate scales active head outputs, the mask selects whether the whole A2V or V2A residual is added.

• Separate rewards for separate modalities. A single global reward was judged insufficient because video and audio quality vary independently. Feedback is decomposed into video-specific, audio-specific, and cross-modal components, with the cross-modal term shared by both streams and their bidirectional interaction modules.

• Refinement targets synchronization-relevant regions. Training begins with nearly uniform token weights and progressively focuses on regions contributing more to audio-visual synchronization, using video-to-audio responses from selected interaction blocks. Depth-dependent gradient scaling attenuates gradients entering the audio stream more strongly in shallow blocks and restores them in deeper blocks.

• Distillation uses self-rollout to avoid exposure bias. Because the 1-step student consumes its own previously refined chunks at inference, training it only with ground-truth past chunks would let small errors accumulate. Distillation therefore generates complete videos with the student and applies the DMD objective to those rollouts, combining distribution matching with perceptual (DISTS) and ℓ2 reconstruction losses, both weighted at one.

• Evaluation is on one benchmark family. Evaluation uses Verse-Bench, which has three evaluation sets: Set 1 and Set 2 cover diverse general audio-visual events, and Set 3 focuses on speech-centric scenarios with visible speakers. The paper notes that benchmarks such as VMBench evaluate general video quality but are not designed to assess audio quality or audio-visual alignment.

Methodology in Plain English

Data first. The team pooled public datasets (Koala-36M, VGGSound, AudioSet, OpenHumanVid, SpeakerVid-5M, Action-100M, Talker-T2AV) plus internal data. Raw videos were cut at scene boundaries with PySceneDetect, near-silent segments were removed by audio RMS energy, and three frames were trimmed from both ends of each segment to reduce boundary artifacts. Clips were then filtered for visual quality (Q-Align), motion (UniMatch-based optical flow), audio quality (Audiobox Aesthetics), and synchronization (Synchformer for general audio-visual events, SyncNet for visible-speech clips).

Annotation. Qwen3-Omni-30B-A3B-Instruct jointly analyzes video and audio to produce one annotation describing subjects, actions, scene context, camera view, speech, music, sound events, and ambience in temporal order. Qwen3-ASR-1.7B transcribes spoken content, and Qwen3.6-27B merges annotation and transcript into the final caption used as the conditioning signal.

Training the generator. Two modality-specific backbones are used. Stage 1 freezes the first half of both and trains rank-256 LoRA adapters (scaling factor 128, zero dropout) on the latter half plus cross-modal modules and gates. Stage 2 merges the adapters and trains everything jointly. Stage 3 is a curated High-Quality Finetuning run using synchronization metrics, aesthetic scores for both modalities, resolution, and duration, with OmniShotCut used to detect and discard segments containing shot transitions. Both streams are trained with flow matching: for each modality, clean latents are interpolated with independently sampled Gaussian noise and the model predicts the velocity field, with a token- and feature-normalized loss. A base timestep is drawn from a discrete 1,000-timestep flow schedule with a shift factor of 5.0. Pre-training uses equal modality weights of 0.5 and 0.5, while High-Quality Finetuning uses 0.5 for video and 0.1 for audio. Optimization uses AdamW with betas (0.9, 0.99) and epsilon 1e-10, a 200-step linear warm-up, bfloat16 mixed precision, and gradient-norm clipping at 1.0.

Reinforcement learning. Starting from the pretrained base model, the policy generates a group of joint audio-video candidates per conditioning pair from a training set of 1,000 first-frame–prompt pairs. Each candidate is scored on video quality, audio quality, prompt consistency, and audio-visual synchronization; rewards are normalized within each group to produce relative advantages. Advantages are routed to their corresponding streams, cross-modal feedback is shared, and a regularization term keeps the updated policy close to the base model to preserve diversity. Training alternates generation, evaluation, and policy updates in an online loop.

2K refinement. A bidirectional multi-step diffusion-transformer teacher is trained with flow matching on LR-conditioned refinement, using a degradation curriculum that progresses from mild to strong degradations including temporally correlated corruptions, since the inputs are generated videos with flicker and jitter rather than camera captures. The teacher is converted into an autoregressive multi-step refiner by teacher forcing over temporal chunks, then distilled with DMD (following DMD2) into a 1-step student under self-rollout. At inference the refiner processes temporal chunks sequentially, performing one denoising evaluation per chunk conditioned on the LR video and previously refined HR chunks, leaving audio unchanged.

Why This Matters

Research impact. The paper frames model scale and availability as practical barriers: many existing joint systems use tens of billions of parameters, offer only hosted access, or are hard to reproduce. Releasing a 7B generator and a 2K refiner provides a reproducible base for research on unified audio-video generative modeling and allows downstream adaptation without multi-billion-parameter compute budgets.

Real-world applications

• Dialogue and talking-head content, where speech drives facial and articulatory dynamics (the paper's speech and dialogue data pool supports A2V and joint training).
• Foley and action sound, where visible events such as impacts, tools, object interactions, and vehicles provide cues for temporally aligned sound generation.
• Scene-level ambience and music, where visual context provides cues for sustained acoustic content such as crowds, weather, and background music.
• High-resolution post-production, where the 2K refiner sharpens generated footage while preserving motion and audio-aligned timing.

Industry relevance. The capability comparison table positions the work against LTX-2.3, MOVA, LingBot-Video, MAGI-2 Preview, MiniMax H3, Seedance 2.5, and Kling AI 3.0. Several of these are released only through online services or with base checkpoints only, while DreamX-Creator 1.0 is listed as open weights with a local 2K path. Local, downloadable joint generation at modest scale matters for teams that cannot rely on hosted APIs for cost, latency, privacy, or reproducibility reasons.

Future Directions

• Pushing beyond 2K. The comparison table lists LTX-2.3 as supporting local output up to 4K, while DreamX-Creator 1.0 is listed as local without a stated ceiling above 2K. Extending a comparable refinement path further is an obvious next target.

• Better balancing of modality rewards. The paper keeps video, audio, and cross-modal feedback separate precisely because they can trade off. How to weight and schedule these terms, and how much regularization toward the base model is optimal, remains open.

• Data imbalance and capability coverage. Speech at 45.0% and event sounds at 33.4% dominate the corpus, so capabilities such as music, ambience, and mixed cinematic content receive less supervision. Rebalancing and expanding those pools is a natural extension.

• Scaling the architecture. The generator is deliberately compact at 7B, and its joint capacity relative to much larger systems (for example MAGI-2 Preview at 114B total, MOVA at 32B total / 18B active) is not resolved by the available content.

• Benchmark coverage. Evaluation here uses Verse-Bench alone. The paper itself notes that existing general video benchmarks are not designed for audio or audio-visual alignment, so broader joint evaluation protocols are needed.

Target Audience

Researchers and engineers working on generative video and audio, particularly those interested in unified multimodal generation, cross-modal attention design, RL post-training of generative models, and efficient high-resolution refinement. It is also relevant to practitioners who need reproducible, locally deployable audio-video generation with openly released weights, and to teams evaluating the trade-off between model scale, openness, and output resolution.

Note: the provided paper content is truncated in the middle of Section 6.2, so the quantitative benchmark results, metric definitions beyond the video-quality metrics (Aesthetic Predictor, MUSIQ, MANIQA, DINOv3), and the full audio-quality dimensions are not available here.

Authors’ abstract

Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator. Conditioned on a first frame and a text prompt, the generator jointly denoises modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning further post-trains the generator with Modality-Aware Multimodal Feedback that routes video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.