MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models

AI summary

Overview

• Research area: Audio and speech processing (cs.SD), specifically benchmarking and inference-time improvement of large audio-language models (LALMs) on multi-audio tasks.
• Technical level: Intermediate. The paper is readable without deep modeling background, but it assumes familiarity with multimodal LLMs, multiple-choice evaluation, and decoding strategies such as self-consistency.
• Scope: The paper introduces MUGEN, a 35-task, 1,750-instance benchmark for multi-audio understanding across speech, general audio, and music, evaluates seven LALMs plus a cascaded ASR+LLM baseline, and tests training-free strategies to improve performance.

What This Paper Is About

Large audio-language models are usually tested on a single audio clip at a time, even though real applications often require comparing several audio segments at once. The authors build MUGEN, a benchmark where a model must pick which of several audio clips best satisfies a written constraint (for example, "select the audio with the angriest emotion"), so every candidate is itself an audio signal rather than text. The paper then measures how current models fail under this setting and tests simple, training-free prompting strategies that reduce those failures.

Key Contributions

1. A new benchmark, MUGEN (Multi-audio Grounding and Understanding Benchmark): 35 audio-grounding tasks totaling 1,750 test instances across 7 dimensions spanning speech, general audio, and music, with 9,250 audio clips in total.
2. An audio-as-option design: Each task requires selecting the best-matching audio from five candidates, and ten tasks additionally include a reference audio, giving six audio inputs and requiring reference-conditioned comparison. Because answers are audio rather than text, models must compare acoustic features directly.
3. Identification of specific weaknesses: The paper documents that LALMs are consistently worse on non-semantic and paralinguistic attributes than on semantics, and that accuracy degrades markedly as the number of concurrent audio inputs grows, identifying input scaling as a systematic bottleneck.
4. Validation of training-free improvement strategies: The authors show that Audio-Permutational Self-Consistency (APSC), which randomly permutes the order of audio candidates before each inference, yields up to 6.28% accuracy gains, rising to 6.74% when combined with Chain-of-Thought.

Main Findings

• MUGEN's construction: The benchmark contains 35 tasks and 1,750 test instances over 7 dimensions, with 9,250 audio clips; mean audio duration is 8.60 ± 8.79 seconds and mean instruction length is 13.91 ± 5.46 words. Each task uses five audio candidates, and ten tasks add a reference audio for six inputs.
• Seven evaluation dimensions: Semantics & Pragmatics (S&P), Speaker & Demographics (S&D), Affective & Paralinguistic (A&P), Temporal Awareness (TA), Acoustic Scene & Event Analysis (AS&E), Music Analysis (MA), and Compositional Acoustic Reasoning (CA).
• Open-source LALMs lag: Overall accuracy (micro-average) was 24.91 ± 2.03 for DeSTA2.5-Audio, 28.69 ± 2.12 for Qwen2.5-Omni, 17.43 ± 1.78 for Audio Flamingo 3, 27.83 ± 2.10 for Voxtral-Mini-3B, 28.63 ± 2.12 for Voxtral-Small-24B, and 26.29 ± 2.06 for Phi-4-multimodal-instruct.
• Cascaded systems are competitive: An ASR+LLM cascade (Whisper-large-v3 transcription with Gemini-3-pro answering) reached 29.09 ± 2.13 (Low) and 30.06 ± 2.15 (High) overall, comparable to open-source end-to-end LALMs, and it did noticeably worse on non-semantic dimensions.
• The proprietary model leads but is far from solved: Gemini-3-pro scored 67.66 ± 2.19 overall (Low thinking level) and 69.60 ± 2.16 (High), substantially above every open-source model, yet still far from perfect.
• Semantic tasks are easier than non-semantic ones: All LALMs performed consistently better on the semantic dimension than on others, indicating that non-semantic perceptual reasoning remains underdeveloped.
• Temporal reasoning is a specific blind spot: Reasoning over and comparing temporal information was consistently difficult, with substantially lower performance even for the proprietary model. Model-specific weaknesses also appeared, such as music analysis for Audio Flamingo 3 and compositional acoustic reasoning for Voxtral-Small-24B.
• Performance degrades as audio inputs increase: Using reduced variants of MUGEN with two to five candidate audios across the remaining 32 tasks (ranking-based tasks were excluded), accuracy fell as candidates increased from two to five. With five candidates, Qwen2.5-Omni preserved only about 66% and 48% of its two-candidate accuracy in tasks without and with reference audio, respectively, while Gemini-3-pro retained around 80% in both cases. The decline already emerged with only three additional audio inputs, and Gemini-3-pro's two thinking levels followed similar trends.
• Chain-of-Thought alone does not help: CoT changed Gemini-3-pro (Low) by +0.23, Gemini-3-pro (High) by +0.86, and slightly degraded Qwen2.5-Omni (-0.12).
• Self-Consistency helps more: SC changed Qwen2.5-Omni by -0.69, Gemini-3-pro (Low) by +2.40, and Gemini-3-pro (High) by +3.14.
• APSC is the strongest single strategy: APSC raised Qwen2.5-Omni to 30.69 (+2.00), Gemini-3-pro (Low) to 73.94 (+6.28), and Gemini-3-pro (High) to 74.97 (+5.37).
• APSC + CoT gives the best results: This combination produced 31.31 (+2.62) for Qwen2.5-Omni, 74.40 (+6.74) for Gemini-3-pro (Low), and 75.26 (+5.66) for Gemini-3-pro (High).
• Evaluation reliability: Automatic judging used Claude Haiku 4.5 (claude-haiku-4-5-20251001) at temperature 0, with 99% agreement with human annotators on 400 randomly sampled instances.

Methodology in Plain English

The authors designed a benchmark where every question is a selection task: a short written constraint is given, and the model must pick the audio clip out of several candidates that best fits it. Options are audio, not text, so the model has to listen to and compare clips directly rather than reason only over transcripts. Distractors are deliberately chosen or synthesized to vary the specific attribute in question, so superficial shortcuts do not work. Tasks are grouped into seven dimensions covering speech content, speaker traits, emotion and prosody, timing, environmental sound, music, and combined reasoning.

Data came mostly from public corpora; synthetic generation with open-source speech synthesis models or GPT-4o mini TTS was used only where precise attribute control was needed, and all synthesized samples were manually verified. All instructions were manually authored.

Seven LALMs were tested, plus a cascade baseline that transcribes audio with Whisper-large-v3 and answers with Gemini-3-pro. Open-source models ran with the vLLM framework. Decoding was greedy except for Voxtral (temperature 0.2, top-p 0.95, as recommended by its authors) and Gemini-3-pro (temperature 1).

To study input scaling, the authors built reduced variants of the benchmark by removing one non-ground-truth option at a time, shrinking candidates from five down to two while leaving instructions unchanged, and excluding ranking-based tasks.

To test improvement strategies, they compared Chain-of-Thought prompting, Self-Consistency (multiple sampled responses aggregated by majority voting), and Audio-Permutational Self-Consistency (one response per each of 10 random audio orderings, mapped back to the original indexing and aggregated by majority voting). Both SC and APSC generated 10 responses, so the two methods have comparable computational cost.

Why This Matters

The paper reframes multi-audio comprehension as a prerequisite rather than a niche capability, and shows that current evaluation, built around single-audio settings, has hidden a systematic weakness. It also shows that the bottleneck is perceptual and comparative rather than purely logical, since Chain-of-Thought prompting barely helps while changing the ordering of audio inputs does.

Real-world applications the paper points to:

• Audio-based in-context learning, where several audio demonstrations guide a model's adaptation.
• Speech retrieval-augmented generation (RAG), where retrieved audio must be compared against a query.
• Multi-speaker analytics, which requires distinguishing and tracking speakers across clips.
• Cross-utterance event matching, which requires reconciling information spread across separate audio segments.

Industry relevance: Any product built on voice agents or audio assistants that must handle several clips at once faces exactly the failure modes MUGEN measures. The finding that ordering-sensitive aggregation (APSC) produces large gains without retraining offers a low-cost deployment path, though the paper notes that generating multiple responses per instance incurs considerable computational overhead.

Future Directions

• Mitigating input scaling: The authors identify input scaling as a systematic challenge and note that increased thinking depth in Gemini-3-pro did not mitigate the degradation, leaving open how architectures or training should handle more concurrent audio inputs.
• Strengthening non-semantic perception: Since temporal awareness and other non-semantic dimensions remain weak, the paper suggests targeted training strategies or architectural adjustments may be required.
• Cheaper aggregation: APSC and APSC+CoT improve accuracy but require many generations per instance; reducing that overhead is an obvious open engineering question.
• Extending beyond the current design: MUGEN tops out at five candidates (six with a reference audio) and excludes ranking-based tasks from the scaling analysis, so behavior at larger input counts and under ranking-style instructions is not reported.

Target Audience

Researchers and engineers working on large audio-language models, speech and audio understanding, and multimodal benchmarking benefit most. The paper is also useful for practitioners building voice agents, speech RAG systems, or multi-speaker analytics who need to know where current models break down, and for benchmark designers interested in the audio-as-option evaluation design.

Authors’ abstract

While multi-audio understanding is critical for large audio-language models (LALMs), it remains underexplored. We introduce MUGEN, a comprehensive benchmark evaluating this capability across speech, general audio, and music. Our experiments reveal consistent weaknesses in multi-audio settings, and performance degrades sharply as the number of concurrent audio inputs increases, identifying input scaling as a fundamental bottleneck. We further investigate training-free strategies and observe that Audio-Permutational Self-Consistency, which diversifies the order of audio candidates, helps models form more robust aggregated predictions, yielding up to 6.28% accuracy gains. Combining this permutation strategy with Chain-of-Thought further improves performance to 6.74%. These results expose blind spots in current LALMs and provide a foundation for evaluating complex auditory comprehension.