Hear What Matters! Text-conditioned Selective Video-to-Audio Generation

AI summary

Overview

Research area: Multimodal generative audio — specifically video-to-audio (V2A) generation, sitting at the intersection of computer vision, audio synthesis, and cross-modal representation learning.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, transformer cross-attention, teacher-student distillation, and V2A evaluation metrics.

Scope: The paper defines a new task — text-conditioned selective video-to-audio generation — and introduces SelVA, a model that uses a text prompt as an explicit selector for which sound source in a multi-object video should be synthesized, along with a new benchmark called VGG-MonoAudio.

What This Paper Is About

Existing video-to-audio models produce a single holistic soundtrack for an entire video, so if a user wants only one sound out of several — say the cat meowing and not the dog barking — they must re-synthesize the whole track. The authors reformulate this as a selective generation problem, where the target sound source is specified purely by a text prompt rather than by spatial cues such as segmentation masks or points. Their goal is to generate audio that matches the user's textual intent while staying semantically aligned with the video and temporally synchronized with it.

Key Contributions

1. SelVA, the first text-conditioned selective V2A framework that relies solely on text prompts, without any spatial guidance such as masks or points. Prior selective approaches depend on visual prompts (points, masks) or computationally expensive segmentation models, and they struggle with diffuse or non-localizable sounds such as rainfall or wind.

2. Learnable supplementary [SUP] tokens prepended to text embeddings inside a cross-attention block in the video encoder. These are designed to suppress high-norm attention artifacts — cases where models highlight irrelevant tokens due to spurious correlation — and thereby sharpen intent-relevant cross-modal grounding.

3. An autonomous video-mixing scheme for self-supervised training. Two videos are horizontally concatenated with a random ratio, and the audio and text of one of them serve as the target, which sidesteps the absence of source-separated mono audio supervision.

4. VGG-MonoAudio, a new evaluation benchmark of clean single-source videos, curated from UnAV-100 clips that overlap with the VGGSound test set. It contains 67 clean single-source videos spanning 39 unique events across 8 categories (human, music, vehicle, tool, animal, nature, sport, other), expanded into 1,071 mixed pairs — 560 inter-class and 511 intra-class.

Main Findings

• State-of-the-art on VGG-MonoAudio. On the inter-class subset, SelVA records an FAD of 51.7, KAD of 0.676, IS of 13.07, KL of 1.85, CLAP of 0.292, IB of 0.3251, and DeSync of 0.721. On the intra-class subset it records FAD 37.0, KAD 0.492, IS 9.62, KL 1.04, CLAP 0.280, IB 0.3262, and DeSync 0.639. The paper states SelVA achieves the best scores in both semantic and temporal audio-video alignment.

• Baselines fall short in different ways. MMAudio-S-16k, which the paper describes as overlooking the text modality, reaches a lower CLAP than SelVA (0.270 inter-class, 0.276 intra-class), while ReWaS (DeSync 1.364 inter-class, 1.377 intra-class) and VinTAGe (1.292, 1.304) show poor temporal alignment. The VOS+MMAudio segmentation-based baseline is competitive on semantic alignment (CLAP 0.291 inter-class) but weaker on temporal synchronization (DeSync 0.991 inter-class).

• [SUP] tokens matter most for temporal alignment. Removing them degrades DeSync to 0.868 inter-class (from 0.721) and 0.734 intra-class (from 0.639), while the paper reports negligible sacrifice in audio quality and semantic alignment. Attention visualizations show high-norm outlier patches without [SUP] and intent-focused attention with it.

• Five [SUP] tokens is the sweet spot. An ablation identifies 5 tokens as achieving the best DeSync score while maintaining comparable CLAP, consistent with observations from prefix-tuning that too few tokens convey insufficient conditioning and too many lead to redundancy and overfitting.

• Joint (single-stage) training hurts. Optimizing both losses simultaneously causes notable drops in semantic and temporal alignment. In the intra-class benchmark, IB (0.3229 vs. 0.3248) and DeSync (0.777 vs. 0.670) are reported as worse than the frozen MMAudio baseline.

• Generator-only finetuning causes a shortcut. Finetuning only the V2A generator with a frozen video encoder yields marginal gains in audio quality and semantic alignment but noticeably degrades audiovisual temporal synchrony, because the generator produces sounds that align with text semantics but drift from actual video events.

• Human study agrees with objective results. 26 participants rated overall audio quality, text-audio alignment, and audiovisual temporal synchronization on a 5-point Likert scale across 16 videos. SelVA outperformed MMAudio and the VOS baseline on all criteria and reached audio quality comparable to ground truth. The VOS baseline scored 3.78 versus 4.53 for SelVA on text-audio alignment despite a comparable CLAP score, which the authors attribute to human listeners being more sensitive to off-screen noises loosely aligned with the text prompt.

• CLIP features were removed from conditioning. On the VGGSound test set, MMAudio-S-16kHz drops in CLAP from 0.197 to 0.235 when CLIP features are replaced by a null embedding, an improvement of 0.038, while DeSync stays essentially steady (0.486 vs. 0.492). Removing the Synchformer embedding instead degrades IB (0.2927 to 0.2394) and DeSync (0.486 to 1.243).

Methodology in Plain English

SelVA has two modules: a video encoder that learns to focus on what the text describes, and an audio generator that produces sound conditioned on that focused visual signal plus the text.

The video encoder starts from a pretrained Synchformer model. The authors insert a cross-attention block that lets text features act as keys and values while video features act as queries, so text can modulate the visual representation. They prepend several learnable [SUP] tokens to the text embeddings to help the attention concentrate on the intended sound source and ignore distractors. Only small parts of the encoder — the cross-attention and spatial attention pooling layers — are trained; the paper reports 19M trainable parameters, 14% of the encoder's total.

Training happens in two stages. In the first stage, a frozen teacher encoder sees a single-source video and produces a reference feature; the student encoder must produce a matching feature while seeing a mixed video containing the target plus a distractor, given the target's text prompt. The student is optimized with an L2-norm regression loss toward the teacher. In the second stage, the video encoder is frozen and the audio generator — an MM-DiT architecture following MMAudio, initialized from MMAudio-small-16kHz weights — is trained with conditional flow matching. The only generator sub-modules finetuned are the video-feature projection layer and the adaLN modules of the audio latent branch, totaling 22M parameters, 14% of the generator's total.

The self-supervised mixing scheme concatenates two videos horizontally at a random ratio λ drawn from a beta distribution, clipped so the target occupies more than 0.2 of the width, with mixing applied within each minibatch at probability 0.75. For classifier-free guidance, video and text features are replaced with learned null embeddings at probability 0.1, and the text feature is dropped with an additional probability of 0.5 to strengthen visual fidelity. Inference uses an Euler solver with 25 linear sampling steps and guidance strength γ = 4.5.

Why This Matters

Impact on research. The paper reframes text from an auxiliary semantic hint into an explicit selector that modulates visual embeddings, defining a task and a benchmark where no direct comparison existed before. It also shows that parameter-efficient finetuning of a video encoder — with trainable tokens and a mixing scheme — can outperform both frozen-encoder baselines and segmentation-based pipelines, while avoiding the computational overhead and boundary failures of segmentation models on fast-moving, motion-blurred, or diffuse sources.

Real-world applications:

• Film and game Foley and post-production, where sound designers layer individually crafted tracks rather than sonifying every visible object and need precise per-source control.
• Audio post-production and mixing sessions in which speech, music, and sound effects must be synthesized under separate, independently controllable conditions.
• Content creation workflows where a creator wants to add or suppress a single sound in a scene without re-synthesizing the entire audio track.
• Interactive media and editing tools, where the paper argues language edits offer more intuitive control than complex visual manipulations such as drawing masks.

Industry relevance. Media production pipelines need per-source audio stems for editing, mixing, and mastering. A model that produces a targeted track from a text prompt — and that a user can redirect simply by changing the wording — fits directly into that stem-based workflow.

Future Directions

• Cleaner training data. The authors state that performance is currently limited by the noisiness of VGGSound, and suggest more rigorous data filtering or refining the auto-mixing process with cleaner source data.
• Deeper text understanding. Because labels are typically simple noun-verb conjunctions, the authors propose improving fine-grained cross-modal distinction (for example, separating "male singing" from "male burping") and attribute controllability (for example, a dog barking "aggressively").
• Residual sound substitution. Although the method significantly alleviates the issue, cases remain when the video encoder fails to consistently track a target movement change.
• Full-scale training. The authors leave comprehensive full training of the model as future work.

Target Audience

Researchers and engineers working on multimodal generative audio, video-to-audio synthesis, and controllable audio generation; practitioners in film, game, and media post-production evaluating AI-assisted Foley and stem generation; and readers interested in parameter-efficient cross-modal grounding, learnable token mechanisms, and self-supervised training strategies for tasks lacking source-separated supervision.

Authors’ abstract

This work introduces a new task, text-conditioned selective video-to-audio (V2A) generation, which produces only the user-intended sound from a multi-object video. This capability is especially crucial in multimedia production, where audio tracks are handled individually for each sound source for precise editing, mixing, and creative control. We propose SELVA, a novel text-conditioned V2A model that treats the text prompt as an explicit selector to distinctly extract prompt-relevant sound-source visual features from the video encoder. To suppress text-irrelevant activations with efficient video encoder finetuning, the proposed supplementary tokens promote cross-attention to yield robust semantic and temporal grounding. SELVA further employs an autonomous video-mixing scheme in a self-supervised manner to overcome the lack of mono audio track supervision. We evaluate SELVA on VGG-MONOAUDIO, a curated benchmark of clean single-source videos for such a task. Extensive experiments and ablations consistently verify its effectiveness across audio quality, semantic alignment, and temporal synchronization.