Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models

AI summary

Overview

Research area: Generative computer vision and audio — specifically video-to-audio (V2A) synthesis, extended to long-form video-to-audio (LV2A) generation.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, transformer architectures, positional embeddings, state-space models (Mamba), and multimodal alignment objectives such as CLIP, Synchformer, PANNs, PaSST and VGGish embeddings.

Scope: The paper proposes MMHNet, a hierarchical multimodal video-to-audio model built on Non-Causal Mamba-2, and shows that a model trained only on 8-second clips can generate coherent, synchronized audio for videos several minutes long.

What This Paper Is About

Existing video-to-audio systems are designed for short clips, typically 8–10 seconds, and break down when asked to produce audio for much longer videos. Public long audio-video datasets mostly cover videos up to about 1 minute, so models cannot simply be trained on long examples.

The paper frames this as a "train-short, test-long" problem: can a model trained on fixed-length 8-second segments generalize at inference to variable-length audio outputs exceeding 5 minutes? The authors identify explicit positional embeddings — a core component of the transformer backbones used by prior V2A models — as a key reason these models fail to generalize in length, and they build an architecture that avoids them.

Key Contributions

1. Introduces the length generalization challenge for V2A. The authors define a setting where models are trained on short, fixed-length audio-visual data and evaluated on long-form video-to-audio generation using the UnAV100 and LongVale datasets.

2. Proposes MMHNet. A multimodal hierarchical network that integrates MMAudio with the HNet hierarchical architecture, replacing attention modules in the single-modality blocks with Non-Causal Mamba-2 to eliminate the need for explicit positional embeddings.

3. Adds hierarchical token routing. MMHNet uses temporal routing (selecting tokens at sound-event boundaries), multimodal routing (selecting tokens with high cross-modal similarity), and chunking/dechunking with downsampling and upsampling, so that only the most informative tokens reach the main network.

4. Builds and reports a long-form LV2A benchmark. Extensive experiments on UnAV100 and LongVale show MMHNet outperforming prior methods, including a demonstration of generation beyond 5 minutes where prior V2A methods fall short.

Main Findings

• Positional embeddings are the bottleneck. In a pilot study, an MMAudio model trained on 8-second audio-visual data and tested on 40 seconds degraded as duration increased, with a reported 3–4 point drop in distribution matching (FD_PANNs) and multimodal alignment (IB) scores. Removing positional embeddings entirely produces homogeneous, orderless audio dominated by prominent visual objects; adjusting them improves alignment but degrades sound quality over long sequences.

• MMHNet-L leads on UnAV100. With 1.09B parameters, MMHNet-L achieves FD_VGG 1.80, FD_PANNs 5.29, FD_PASST 209.06, KL_PASST 1.49, ISC_PANNs 8.10, ISC_PASST 7.35, IB-Score 36.27 and DeSync 0.410 — the best values in the comparison table across those columns.

• Large IB-Score margin. The IB-score, which measures video–audio alignment, surpasses HunyuanVideo-Foley by 3.9 on the UnAV100 dataset. MMHNet-S reaches an IB-Score of 36.82 versus 32.90 for HunyuanVideo-Foley-XXL (5.13B parameters).

• Strong long-duration results on LongVale. MMHNet-S reaches IB-Score 30.62 and DeSync 0.438, and the paper reports a substantial margin of 0.23 on DeSync scores over the second-best method on that dataset. LongVale contains samples with durations up to 7 minutes, where earlier methods struggle with alignment and synchronization.

• Autoregressive models struggle with length. V-AURA, an autoregressive baseline, performs comparatively poorly on the long-form benchmarks, which the authors attribute to error accumulation and limited length generalization.

• Parity on matched-length VGGSound. When training and testing use the same duration (approximately 10 seconds), MMHNet performs on par with MMAudio and surpasses it on several metrics — MMHNet-L achieves ISC_PANNs 20.52 and ISC_PASST 15.34 versus 17.40 and 13.33 for MMAudio-L.

• Non-Causal Mamba-2 beats alternatives. In the core-network ablation on UnAV100, Non-Causal Mamba-2 gives FD_PANNs 5.87, ISC_PANNs 7.62, IB-Score 36.82 and DeSync 0.439, versus transformers (9.00, 6.42, 28.41, 0.638) and causal Mamba-2 (9.18, 5.85, 33.32, 0.497). The same ordering holds on LongVale.

• Hierarchical processing helps. The hierarchical variant outperforms the non-hierarchical one on UnAV100 (FD_PANNs 5.87 vs 6.31; FD_PASST 217.00 vs 264.43; IB-Score 36.82 vs 35.00; DeSync 0.439 vs 0.621) and on LongVale (FD_PANNs 10.10 vs 11.76; FD_PASST 323.39 vs 442.12; IB-Score 30.62 vs 26.34; DeSync 0.438 vs 0.669).

• A 0.5 routing threshold is best. Sweeping thresholds from 0.3 to 0.7 on UnAV100, the 0.5 setting produced the strongest results across all metrics (FD_VGG 1.80, FD_PANNs 5.29, ISC_PANNs 8.10, IB-Score 36.27, DeSync 0.410), while 0.7 collapsed performance (FD_VGG 15.03, IB-Score 0.02).

Methodology in Plain English

Base model. MMHNet starts from MMAudio's structure, which follows the MM-DiT block design from SD3 and FLUX: some blocks process multiple modalities together, others process a single modality. This lets the network go deeper without paying the full parameter cost of processing every modality at every layer, and it supports joint training on audio-visual and audio-text data. Training uses conditional flow matching in the latent space, and the model predicts a velocity field used by an ODE solver at inference.

Conditioning. Global context is injected through adaptive layer normalization (adaLN) using average-pooled visual and text features. For finer synchronization, the model also uses token-level conditioning from CLIP semantic video features, Synchformer motion-audio synchronization features, and CLIP text features.

Replacing attention. Instead of transformer attention (which needs positional embeddings to know where tokens are), the single-modality blocks use Non-Causal Mamba-2, a state-space model that carries contextual information through a transition matrix rather than through explicit positions. Two reasons are given for non-causality: video is available offline, so tokens do not need to be processed in order, and multimodal fusion is hard when modalities must be merged in a fixed scanning order. The non-causal formulation also avoids the modulation decay that plagues causal models over long sequences, because it defines the mask term using the inverse of each transformation matrix rather than a cumulative product.

Hierarchical token handling. Long videos and audio contain a lot of redundancy. The framework therefore computes cosine similarity between tokens and routes only the important ones forward. Temporal routing keeps tokens where the context shifts (sound-event boundaries) and masks tokens that are too similar to their predecessor. Multimodal routing keeps tokens whose similarity to the reference modality exceeds 0.5. Chunking with downsampling compresses encoder outputs by directly selecting boundary-marked vectors; dechunking with upsampling restores the original length, using a straight-through estimator so gradients still flow to unselected tokens.

Scale variants. The small version (S, 157M parameters) uses N=5 multimodal blocks and N'=4 single-modal blocks; the large version (L, 1.09B parameters) uses N=10 and N'=7.

Evaluation. Metrics cover distribution matching (Fréchet Distance with VGGish, PANNs and PaSST, plus KL divergence with PANNs and PaSST), audio quality (Inception Score with PANNs), semantic consistency (IB-Score via ImageBind cosine similarity between visual and audio features), and temporal synchronization (DeSync, computed with Synchformer over a 4.8-second context window). Because the pretrained classifiers were not trained on long audio, evaluation is performed over multiple chunks.

Why This Matters

Research impact. The paper isolates positional embeddings as a concrete architectural cause of length-generalization failure in V2A, and demonstrates that a non-causal state-space backbone sidesteps it without any inference-time modification or extrapolation technique such as NTK scaling. It also introduces a reproducible long-form evaluation setup on UnAV100 and LongVale, which prior short-clip metrics could not meaningfully cover.

Real-world applications:

• Film and television sound design, where foley and ambience must stay synchronized with scenes far longer than 10 seconds.
• Game development, where ambient audio needs to track long, variable-length gameplay footage.
• Video post-production and content creation, where automatically generated audio tracks must remain coherent across a full multi-minute clip rather than restarting every few seconds.
• Accessibility and archival, where silent or poorly described long-form video can be given contextually matched audio.

Industry relevance. The result that a 157M-parameter model (MMHNet-S) can outperform a 5.13B-parameter baseline (HunyuanVideo-Foley-XXL) on several long-form metrics — and that training only requires 8-second clips — makes long-form audio generation substantially cheaper to train and deploy. Avoiding positional-embedding extrapolation also removes a per-duration tuning step that would otherwise complicate production pipelines.

Future Directions

• Beyond five minutes. The paper demonstrates generation past 5 minutes but does not report where degradation begins, nor how quality scales toward the 7-minute ceiling present in LongVale. Establishing and testing a practical length limit is an open question.
• Perceptual validation. The evaluation relies entirely on learned embedding metrics (FD, KL, IS, IB-Score, DeSync). Human listening tests comparing MMHNet with LoVA and V-AURA are not reported, and the paper does not report training compute or wall-clock cost.
• Training on longer data. The entire premise is train-short/test-long. Whether performance improves further — or whether the architectural advantage persists — when longer training clips become available is not addressed.
• Combining paradigms. The paper positions agent-based segmentation and autoregressive generation as alternatives, noting their failure modes (fragmented transitions and error accumulation respectively). A hybrid that uses MMHNet's global receptive field alongside explicit segmentation or retrieval remains unexplored.

Target Audience

Researchers and engineers working on multimodal generative models, audio synthesis, and video understanding, particularly those interested in length generalization, state-space architectures, or efficient long-sequence modeling. It is also relevant to practitioners in film, gaming, and media production who need automated audio generation for long-form video, and to readers tracking the competition between transformer, Mamba, and autoregressive backbones in generative multimodal tasks. Due to the density of architectural detail and metric tables, some familiarity with diffusion/flow-matching models and audio evaluation metrics is assumed.

Authors’ abstract

Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations.