SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model

AI summary

Overview

• Research area: Audio-visual generative modeling and audio source separation — specifically, extending a pretrained joint audio-video generator so it emits individual per-source waveforms (stems) rather than only a single mixed soundtrack.
• Technical level: Advanced. The paper assumes familiarity with flow-matching diffusion, latent audio/visual autoencoders, transformer attention routing, LoRA adapters, and classifier-free / attention guidance.
• Scope in one sentence: SepGen adapts a frozen 22B joint audio-video generator (LTX-2.5) with 327M trainable LoRA parameters so that a single checkpoint outputs the video, the audio-mix, and one captioned waveform per source, operating in either a generation mode or a separation mode selected solely by the audio-mix noise level.

What This Paper Is About

Joint audio-video generators can synthesize a video with a synchronized soundtrack, but that soundtrack arrives as one mixed waveform in which the individual sound sources are not separately accessible. That matters because rendering a scene from a new camera viewpoint (the "4D" audio-visual setting the paper motivates) requires each source to be available as its own waveform, so it can be localized and propagated to the listener before mixing.

SepGen addresses this by extending a pretrained audio-video generator so that, from a scene caption plus one caption per source, it produces the video, the audio-mix, and one waveform per captioned source in a single sampling run. The same weights also act as a separator that pulls captioned stems out of an observed video and its existing audio-mix.

Key Contributions

1. SepGen framework. To the authors' knowledge, the first framework that turns a pretrained joint audio-video generator into a multi-stem generator, emitting the video, its soundtrack, and one waveform per captioned source in a single sampling run.
2. Mechanisms that protect the pretrained output while keeping stems caption-faithful. A protected audio-mix channel that bypasses the low-rank adapter, aligned rotary positions across audio channels, block-diagonal caption routing (each stem sees only its own caption, the mix sees only the scene caption), and a schedule in which stems attend to an estimated clean audio-mix during generation.
3. One set of weights for both modes. The mode is chosen by the audio-mix noise level: separation holds the observed mix at zero noise, generation denoises it together with the stems. A separation-trained checkpoint already generates stems; a brief continuation (3,000 additional updates) reduces leakage between generated stems, improves speech intelligibility, and stabilizes results across seeds while preserving separation.
4. Evaluation across generated, cross-generator, and real scenes, using automatic metrics plus a human A/B listening study, with comparisons against language-queried separation baselines and an audio-visual masking baseline.

Main Findings

• Generation beats cascades. On the generation benchmark (134 sound scenes, 60 two-speaker dialogues; 194 prompts total), SepGen's joint-run stems lead every cascade baseline on every metric. The joint run scores Judge 3.82, J-swap 2.07, CLAP 0.15, C-swap 0.18, own-line WER 0.07, and speaker cosine 0.66, versus FlowSep at Judge 3.35 / J-swap 1.26 and SAM large-tv at Judge 2.96 / J-swap 0.41.
• The widest margin is in swap margins. J-swap is the score against a stem's own description minus the score against its sibling's, on which a copied audio-mix scores zero. Because a cascade answers each query in isolation and can return the same source twice, it trails badly here (SepGen joint run 2.07 vs. FlowSep 1.26).
• Self-separation trades a second run for higher scores. Running the same weights again in separation mode over the generated video and mix raises every sounds column (Judge 4.21, J-swap 2.29, CLAP 0.16, C-swap 0.20), at the cost of a second sampling run, while the joint-run stems keep a lower own-line WER (0.07 vs. 0.10).
• Separation on rendered Veo 3.1 Lite scenes. Across 149 scenes (104 sounds, 45 speech), both checkpoints surpass every baseline on Judge and both swap margins. The separation checkpoint reaches Judge 4.36, J-swap 2.61, CLAP 0.19, C-swap 0.23; the generation checkpoint reaches 4.39, 2.59, 0.20, 0.23. AudioSep, the strongest cascade on sounds, reaches 3.46 / 1.60 / 0.19 / 0.16.
• The lead survives removal of quoted lines. Speech cells are reported with and without the quoted lines in the captions. SepGen's separation checkpoint scores WER 0.05 with lines and 0.50 without; speaker cosine 0.61 with lines and 0.53 without. The paper notes that without the lines, 4 to 5 of the 90 stems fall silent, which partly explains why speaker cosine improves while word error rises.
• Visual masks do not close the gap. Prompting SAM large-tv with SAM3 video masks of each source instead of text (SA×SAM3) yields Judge 3.07 and J-swap 0.27 on the Veo benchmark.
• Real recordings. On 172 real recordings from SAM-Audio-bench, scored with a Judge from the same encoder family as SAM Audio, the generation checkpoint reaches Judge 3.70 and J-swap 1.52, and the separation checkpoint 3.70 and 1.42, versus the strongest SAM Audio checkpoint at 3.12 / 0.78.
• Users prefer SepGen. In a separation study against SAM Audio with 48 listeners and ten clips, SepGen received 0.58 of judgements and SAM Audio 0.25 (0.64 vs. 0.20 on real recordings; 0.51 vs. 0.30 on the Veo benchmark). SAM Audio's released pipeline ranks the best of eight candidates per source while SepGen contributed a single draw. In generation against AudioSep, with 51 listeners and ten clips, SepGen received 0.63 and AudioSep 0.28 (0.87 vs. 0.10 on speech; 0.49 vs. 0.38 on sounds).
• Jointness is the critical ingredient. In ablation on the LTX-2.3 benchmark (separation checkpoint, seed 0), separating each caption in a standalone pass without the sibling stem collapses every metric — Judge drops from 4.07 to 2.43, J-swap from 1.99 to 0.21, sibling similarity rises from 0.01 to 0.13, WER from 0.09 to 1.05, cosine from 0.66 to 0.95 — returning two near-copies of the audio-mix.
• The video contributes little to separation. Blank frames, an unrelated clip, or no video at all leave every separation metric statistically unchanged (Judge 4.09, 4.08, and 4.06 respectively, versus 4.07 baseline). Captions and the audio-mix drive separation, though each stem's video attention still finds its named source.
• Temporal alignment matters. Retraining with a concatenated rotary timeline instead of aligned positions drops Judge from 4.07 to 3.61, J-swap from 1.99 to 1.36, and more than triples WER from 0.09 to 0.31.
• The pretrained output is provably untouched. A generation pass run with the adapter enabled and disabled from identical inputs yields bit-identical audio-mix tokens and video at every layer and at the output, while stem tokens differ.
• Two-stage training is measurable. Going from the separation checkpoint to the generation checkpoint raises J-swap from 1.95 to 2.07, lowers own-line word error from 0.10 to 0.07, and reduces the seed spread of J-swap from 0.13 to 0.01.

Methodology in Plain English

Starting point. The authors take a pretrained joint audio-video generator (LTX-2.5, a 22B flow-matching diffusion model with separate transformer streams for video tokens and audio tokens) and freeze it entirely.

Adding stems as extra audio channels. Each sound source is represented as additional tokens appended to the audio sequence, so the audio stream carries three channels concatenated along the token axis: stem 1, stem 2, and the audio-mix. Every channel keeps the temporal shape of the pretrained audio latent, so the j-th token of every channel refers to the same time instant.

Flow of information is one-directional. Each stem may attend to itself, the other stem, the audio-mix, and the video, but only to its own caption. The audio-mix attends only to itself and the scene caption. Nothing flows back from the stems into the mix or the video. Latent projections on the mix tokens bypass the adapter with a learned gate, so the frozen projection is used there and the adapted projection only on stem tokens. This is what produces the bit-identical mix and video result.

What is trained. Only a LoRA adapter on the audio stream: rank and scale 128, 480 modules, 327M trainable parameters, applied to audio self-attention, the query and output projections of text cross-attention and video-to-audio attention, and the audio feed-forward block. The loss is computed on the stem tokens alone; predictions on video and mix tokens are not supervised and serve purely as inputs that tell the stems what the scene contains.

Two training stages. Stage 1 (12,000 updates) trains separation only: the audio-mix is always presented clean (σ = 0), so stems learn to decompose it. Stage 2 (3,000 more updates) continues from that checkpoint, changing only how the audio-mix noise level is drawn: with probability 0.3 the mix stays clean, otherwise it is noised to a level capped at the stem noise level. The cap covers exactly the two conditions seen at generation time.

How the mode is selected at inference. Because the backbone is a denoising model, the mix noise level chooses the behavior. Separation starts stems from noise while holding the observed mix clean at every step. Generation denoises everything together; above a threshold of 0.97 the stems attend to the noisy mix, and below it they attend to the current step's estimate of the clean mix — a scheme the authors call Estimated Separation.

Keeping stems apart. Block-diagonal caption routing gives each stem only its own caption. Cross-Stem Attention Guidance, a variant of Normalized Attention Guidance at scale 2, subtracts an attention branch computed with the sibling's caption as a negative prompt, so a sound fitting both captions is pushed away from one of them.

Data. Training uses 4,843 two-source segments: CelebV-HQ talking-head pairs (n = 1,658), URMP two-instrument scenes (n = 1,437), generated two-source clips (n = 749), and cross-video composites (n = 999) of VGGSound and MUSIC-21 clips. Ground-truth stems exist in every family because each mix is the sum of two known sources. Each source caption comes in three registers — semantic, positional, and short — sampled with equal probability. A "source box" bias enforces positional captions during training but is removed at inference.

Comparison setup. Cascade baselines (FlowSep, AudioSep, two released SAM Audio checkpoints, and SAM large-tv prompted with SAM3 masks) each receive the same audio-mix and the same captions and separate them; SepGen produces both stems jointly in one pass.

Why This Matters

• Impact on research. The paper reframes multi-source audio as a property of a joint audio-video generator rather than as a post-hoc cleanup step. It shows that captioned stems can be added to a frozen backbone without perturbing the backbone's video or soundtrack, and that a noise-level parameter can double as a mode switch between generation and separation. It also demonstrates a concrete failure of query-based separation — two queries for the same category can both return the louder voice — and offers joint prediction as the fix.
• Real-world applications.
  • 4D scene replay and free-viewpoint video: rerendering a captured or generated scene from a new camera pose and propagating each sound source to the new listener position, which requires per-source waveforms to place in space.
  • Dialogue and podcast post-production: pulling per-speaker stems out of a recording from free-text descriptions so lines can be edited, re-leveled, or reordered individually.
  • Film and game audio pipelines: extracting or generating speech, music, and effects tracks described in text, and placing them spatially in a scene.
  • Content localization and accessibility: isolating a speaker's line for dubbing, captioning, or enhancement without the original multi-track session files.
• Industry relevance. The method is a lightweight adaptation (327M trainable parameters on a frozen 22B backbone, trained on a single NVIDIA A6000 with int8 quantization), which is an economically realistic path to adding source-level control to an existing generator rather than training a new one. The evaluation spans a proprietary generator's output (Veo 3.1 Lite) and real recordings, and the code, checkpoints, and datasets are released.

Future Directions

• Extending beyond two sources. The current layout is fixed at two stems; a general N-source layout is listed explicitly as future work.
• Spatial placement from stems. Evaluating a placement readout derived from each stem's video attention and integrating it into a spatial renderer for the 4D scenes that motivate the work.
• Handling scene acoustics. Stems retain the reverberation and distance cues of their source scene, so a spatial renderer must account for them before propagating each source to a new viewpoint.
• Waveform fidelity. The audio VAE decoder keeps each stem's envelope and spectrum but regenerates phase, so SI-SDR against a reference waveform measures phase resynthesis rather than separation quality — an open issue for any downstream use that needs sample-accurate waveform recovery.

Target Audience

Researchers and engineers working on audio-visual generative models, language-queried audio source separation, multi-source audio generation, and spatial/4D audio rendering. It is also relevant to practitioners who need per-speaker or per-source control over generative video soundtracks, and to readers interested in parameter-efficient adaptation strategies that add capabilities to a frozen large generative backbone without altering its original outputs. The paper is written for an expert audience: it assumes fluency with flow matching, diffusion sampling schedules, transformer attention masking, LoRA, and guidance techniques.

Authors’ abstract

A 4D audio-visual scene comprises a video, the dynamic geometry it depicts, and the sound sources that populate it, each with its own position and trajectory. Rendering such a scene from a novel viewpoint requires that every source be available as an individual waveform, so that it can be localized in the scene and propagated to the observer before the signals are mixed. Joint audio-video generators can synthesize the video and its soundtrack, but the soundtrack is emitted as a single audio-mix in which the sources are not individually accessible. We present SepGen, which extends a pretrained audio-video generator to emit the video, the mixed soundtrack, and one waveform per captioned source in one joint sampling run. SepGen supports two complementary modes: generation and separation. In generation mode, each source caption specifies what its stem contains. A two-speaker dialogue, for example, comes out as one stem per speaker in the original turn order. In separation mode, the input audio-mix remains clean while the captions specify what to extract, so the model can decompose a recording from a free-text description. We evaluate generation on scenes synthesized from text, and separation on scenes rendered by other generators and on real recordings of speech, music, and sound effects. Given an audio-mix and captions that carry the spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech, and it keeps the lead when the lines are removed from the captions. Code, checkpoints, and datasets are available at https://sepgen.github.io/