The MUSE Benchmark: Probing Music Perception and Auditory Relational Reasoning in Audio LLMS

AI summary

Overview

• Research area: Multimodal audio large language models (audio LLMs), music perception, and benchmark design at the intersection of AI evaluation and music cognition.
• Technical level: Advanced. The paper assumes familiarity with multimodal LLM evaluation, prompting strategies (standalone vs. chain-of-thought), in-context/few-shot learning, and basic music-theoretic vocabulary (chord quality, key modulation, meter, syncopation).
• One-sentence scope: The paper introduces and releases the MUSE Benchmark, ten music-perception tasks with a 200-participant human baseline, and uses it to measure how far four state-of-the-art audio LLMs fall short of human listeners, particularly on relational and music-theoretic reasoning.

What This Paper Is About

Existing audio benchmarks mostly test surface-level classification, tagging, captioning, and in-the-wild question answering, which means a model can appear to "understand music" by exploiting co-occurrences such as timbre or tempo cues rather than by representing musical structure. The authors argue that current evaluations miss abstract, relational abilities that human hearing relies on, such as recognizing a melody after it is transposed, tracking melodic contour, or perceiving chord harmonic function. MUSE (Music Understanding and Structural Evaluation) is built to probe those abilities directly and to compare model performance against a large human sample, including a small subgroup of expert musicians.

Key Contributions

1. The MUSE Benchmark: An open-source resource of 10 tasks (each with 20 trials), split into a "Beginner" tier of five core-perception tasks and an "Advanced" tier of five music-theoretic tasks, with stimuli, task descriptions, and inference scripts released publicly (GitHub: brandoncarone/MUSE_music_benchmark; an Airtable link and an OSF human-data link are also provided).
2. A large human comparison baseline: Data from 234 online participants, reduced to a final sample of 200 after excluding 34 who failed an in-experiment headphone check (105 males, 89 females, 6 non-binary; mean age = 38.76, SD = 12.79), plus an "Expert Musician" subgroup (N=6) defined as scoring at or above the 90th percentile on the Gold-MSI Musical Training scale.
3. A four-model evaluation under two prompting regimes: Audio Flamingo 3, Qwen2.5-Omni, Gemini 2.5 Flash, and Gemini 2.5 Pro were tested in Standalone and Chain-of-Thought (CoT) conditions, with three random seeds per task and condition, yielding 240 runs total.
4. Diagnostic analyses of model reasoning: Error-level inspection of Gemini Pro's CoT logs and a post-hoc comparison of model in-context learning (varying shots) against human musical training using GLM and GLMM models.

Main Findings

• A persistent human-machine gap on relational tasks: Expert musicians reached 100% on Pitch Shift Detection, whereas the best model (Gemini Pro) reached 81.36% in the Standalone condition and 98.33% with CoT. On Chord Sequence Matching, musicians scored 85.00% versus Gemini Pro's 66.67%; on Key Modulation Detection, 91.70% versus 88.33%; on Meter Identification, 73.30% versus 46.67%.
• Classification is not the bottleneck, reasoning is: Gemini Pro and Qwen both hit 98.33% on Instrument Identification in the Standalone condition, essentially matching experts at 98.30%. Gemini Pro also reached 100.00% on Oddball Detection and 96.67% on Rhythm Matching, but its accuracy fell on advanced tasks such as Chord Sequence Matching (66.67%) and Meter Identification (46.67%).
• Critical failures in some SOTA models: Qwen's Standalone accuracy on Melody Shape Identification was 23.33%, at roughly the 25.00% chance level, indicating a failure to use relative pitch direction. Audio Flamingo 3 performed at or just above chance on nearly all 10 tasks (for example, 25.00% on Melody Shape Identification, 50.00% on Oddball Detection, Rhythm Matching, Pitch Shift Detection, Chord Sequence Matching, and Syncopation Comparison).
• Melody Shape Identification is a high-variance outlier, not a uniform gap: Expert musicians scored 95.00% and Gemini Pro 96.67%, while other SOTA models failed dramatically on the same task.
• CoT prompting is inconsistent and often harmful: CoT produced one dramatic gain (Gemini Pro's Pitch Shift Detection, 81.36% to 98.33%) but degraded Gemini Pro on Rhythm Matching (96.67% to 88.33%) and Syncopation Comparison (69.49% to 61.67%), and worsened Qwen's below-chance Melody Shape Identification (23.33% to 18.33%). Across models, Standalone performance matched or exceeded CoT performance overall.
• Confident-sounding reasoning that is often wrong: In Syncopation Comparison, Gemini Pro's reasoning was directionally consistent in all 37 correct trials, but exact off-beat counts were correct in only 4/37. In Chord Quality ID it correctly identified the defining major or minor third in 34/60 trials. In Chord Sequence Matching it correctly identified I–V–vi–IV 19 times and vi–IV–I–V 8 times but never correctly identified other progressions (for example, I–IV–V). In Key Modulation Detection, for 30 modulation items it incorrectly asserted "no modulation" in 10 cases, and across the 27 trials it did describe, mean absolute error was 3.04 scale degrees with only one exact match.
• More in-context examples do not substitute for musical training: Varying the number of few-shot examples (0, 1, 2, 4, 8 or 0, 1, 3, 6, 9 depending on the number of response options) produced a significant positive effect on accuracy for only one of the four examined tasks, Melody Shape Identification (p < .001), and only for Gemini Flash. Shots had no statistically significant effect on Key Modulation Detection, Chord Sequence Matching, or Syncopation Comparison. Human musical training, by contrast, had a significant positive effect on accuracy across all four tasks.

Methodology in Plain English

The authors wrote and recorded 200 musical stimuli (mean length 14.1 seconds; min 3 seconds; max 46 seconds) using Logic Pro X, an Apollo Twin X interface, Yamaha HS8 monitors, and a 2021 16" MacBook M1 Pro. Guitars were recorded with a PRS McCarty Hollowbody II and a Schecter Solo-6 through Neural DSP Tim Henson Archetype and Cory Wong Archetype plugins; bass with a Fender Squier Classic Vibe '60s Mustang Bass through the Cory Wong Archetype plugin; piano with an Arturia KeyLab Essential Mk3 49-Key MIDI controller and Analog Lab V; drums with a Roland TD-17 Electronic Drum Kit and Superior Drummer 3.

Each of the 10 tasks presents one or two audio clips and a small set of answer choices (for example, Same/Different, Major/Minor, Groups of 3/Groups of 4/Groups of 5). Five tasks (Instrument Identification, Melody Shape Identification, Oddball Detection, Rhythm Matching, Pitch Shift Detection) assess fundamental perception; five (Chord Identification, Syncopation Comparison, Key Modulation Detection, Chord Sequence Matching, Meter Identification) require formal music-theoretic knowledge.

Models were run through custom inference scripts with identical system instructions and few-shot examples for the Standalone condition, using chat modes for multi-turn stateful interaction. The CoT condition added multi-step analytical prompts with in-context demonstrations of the reasoning process. Audio Flamingo 3 was an exception: preliminary testing showed it did not follow instructions reliably with chat history or few-shot examples (performing effectively at chance), so it was tested without chat history or examples, using a combined system and per-trial prompt, and its two stimuli for comparison tasks were concatenated into a single file separated by spoken cues ("Here is the first excerpt," "Here is the second excerpt") and 1–2 second silences. Each task script was run three times with different random seeds and accuracies were averaged.

Human participants were recruited via Prolific and New York University's student population, tested in PsychoPy hosted on Pavlovia, and screened with an in-experiment headphone check. To reduce fatigue, the benchmark was split into two halves with random assignment, and accuracy was computed by pooling correct responses across both groups per task. Task and stimulus order were randomized. Before each task, participants received instructions and few-shot examples for every possible answer choice, matching the models' Standalone condition.

Why This Matters

Impact on research. The paper argues that current audio benchmarks probe surface-level classification and can be passed via learned co-occurrences rather than genuine structural representations. MUSE offers a diagnostic instrument aimed at invariant musical representations, and the post-hoc analysis suggests that scaling in-context examples is not a substitute for the kind of rule internalization that musical training produces in humans.

Real-world applications.

• Music education and practice tools that need to judge melodic contour, meter, or chord quality rather than just label genres.
• Music recommendation, search, and cataloguing systems that must match melodies across transpositions or keys.
• Assistive and accessibility technologies that describe or query music by structural content rather than timbre and tempo.
• Automated music analysis and production assistants that reason about harmony, meter, and syncopation.

Industry relevance. Companies building audio-capable foundation models can use these results as a diagnostic of where capability claims rest on shallow cues, and as evidence that better prompting (CoT or more few-shot examples) is not a reliable fix for missing perceptual competence.

Future Directions

• Change training paradigms rather than prompts: The authors argue that bridging the human-machine gap likely requires fundamental changes in model architecture and training paradigms, possibly mimicking how humans learn music, instead of more data or more complex prompts.
• Build genuinely invariant musical representations: The failures at or near chance on relational tasks point to a shared absence of invariant representations, which the paper frames as an open modeling problem.
• Determine why CoT helps or hurts: CoT improved Gemini Pro on Pitch Shift Detection but degraded performance on Rhythm Matching and Syncopation Comparison; what distinguishes beneficial from detrimental step-by-step reasoning in audio is unresolved.
• Re-examine in-context learning as a proxy for expertise: The analysis used number of shots as a proxy for learning and found effects for only Melody Shape Identification (p < .001, Gemini Flash only); whether any in-context regime can approximate human training effects remains open.

Target Audience

Researchers and engineers working on multimodal and audio LLM evaluation, music information retrieval, and music cognition; benchmark designers who need tasks that separate surface classification from structural reasoning; and product teams building music-aware audio systems who want a diagnostic of where current models actually fail.

Authors’ abstract

Multimodal Large Language Models (MLLMs) have demonstrated capabilities in audio understanding, but current evaluations may obscure fundamental weaknesses in relational reasoning. We introduce the Music Understanding and Structural Evaluation (MUSE) Benchmark, an open-source resource with 10 tasks designed to probe fundamental music perception skills. We evaluate four SOTA models (Gemini Pro and Flash, Qwen2.5-Omni, and Audio-Flamingo 3) against a large human baseline (N=200). Our results reveal a wide variance in SOTA capabilities and a persistent gap with human experts. While Gemini Pro succeeds on basic perception, Qwen and Audio Flamingo 3 perform at or near chance, exposing severe perceptual deficits. Furthermore, we find Chain-of-Thought (CoT) prompting provides inconsistent, often detrimental results. Our work provides a critical tool for evaluating invariant musical representations and driving development of more robust AI systems.