Audio Captioning, Retrieval, and Question Answering

Asking whether a particular sound is in a clip should be easy for a system that can already describe the clip. It is not.

A team at National Taiwan University took AudioCaps clips and wrote 15,110 object-presence questions whose answer was yes. Then 15,110 matched questions whose answer was no. On captioning, the audio-language models they tested were comparable to specialised audio captioning models. On the questions they collapsed. SALMONN-7B answered “Yes” on 92.2% of the random-sampled questions, which bought it an F1 of 24.4. Qwen-Audio-Chat-7B answered “Yes” 79.3% of the time. Kuan and colleagues published the result at Interspeech in 2024.

A system that says yes to nine questions in ten is not answering nine of them. It is producing the shape of an answer. The matched negatives are what make that visible. A question set whose answer is always yes cannot separate a model that listened from a model with a habit.

A review that stops at caption quality never sees any of this. The descriptions read as well as a specialist's. The yes/no answers underneath them were close to a reflex. That gap is what the rest of the lesson is about: producing words about a recording, and being able to defend them against the recording. It runs in both directions.

Both of those failures are about words attached to a recording. The three tasks that attach them are not interchangeable. Captioning produces an open-ended description of a recording. Cross-modal retrieval ranks audio and text against each other, and its useful evidence is recall and rank under independent splits. Audio question answering responds to a targeted query about audible evidence, which may require temporal or quantitative reasoning.

MMAU puts a number on the distance between them. It is a 2024 benchmark from the University of Maryland and Adobe: 10,000 human-annotated audio clips spanning 27 skills. Human annotators scored 82.23% on the test-mini split — 86.31% on sound, 78.22% on music, 82.17% on speech. The best audio-language model, Gemini Pro v1.5, scored 52.97%. The best open model, Qwen2-Audio, scored 52.50%.

The highest score of all did not come from an end-to-end audio model at all. A cascade captioned the audio first and then prompted a text-only LLM. GPT-4o with strong captions reached 57.30% on test-mini and 58.74% on test. “The top-performing LALM achieves only 53% accuracy, while the best-cascaded captioning + LLM approach reaches just 59%. In comparison, human performance achieves 82%,” the authors write.

Splitting the problem into captioning and then reasoning over the caption beat doing both at once. And every one of those systems sat far below the annotators who listened.

Evidence that establishes one of the three tasks establishes nothing about the other two. A clip can be ranked first for the query that found it and still carry a caption nobody could defend. A model can caption it well and still fail the question about what is in it.

Underneath all three sits one line, and it has to be drawn in every one of them: between what was directly heard and what is merely plausible around it. A fluent caption can supply location, identity, intent, emotion, cause, or danger beyond the waveform. It costs the model nothing to make the leap.

The AudioCaps probe gave that leap a measurement. 15,110 positive and 15,110 matched negative object-presence questions, each asking an audio-language model whether a given object was there. The line is not a matter of taste once you can count what crosses it.

The finding is worth having in the authors' own words: “Our findings reveal that LALMs are comparable to specialized audio captioning models in their understanding of audio content, but struggle to answer discriminative questions, specifically those requiring the identification of the presence of particular object sounds within an audio clip,” the abstract says. Descriptive fluency and discriminative evidence came apart in the same systems.

Two different questions follow from the line, and they need different numbers. Retrieval recall and rank under independent splits asks whether the right recording comes back. Caption coverage, precision, and unsupported-claim rate asks whether the words about it were earned. Neither number should stand in for the release decision on its own. A system can be comparable to a specialist captioner on the first kind of evidence while answering “Yes” to 92.2% of random-sampled object-presence questions.

Keeping those four apart means keeping four stages apart that a single score folds together: measurement, modeling, decision, and verification. The decision comes first, and it is a written one. An evidence policy says what a caption may claim from audio alone.

Define the evidence policy, separating audible observation, uncertain interpretation, metadata, and prohibited inference. Represent time and query, using clip, segment, event, or grounded intervals according to the task. Generate or retrieve, conditioning on audio and text while preserving uncertainty and source boundaries. Then evaluate semantically and behaviorally, combining retrieval metrics, human judgments, grounding, hallucination, and abstention.

Write down what may not be inferred and an object-presence dispute stops being a matter of taste. Identity, intent, cause, diagnosis and emotion go on the list of things a waveform does not establish. The argument is settled before anyone hears the clip.

It is the same move DCASE made about contamination. Rather than argue case by case over which Freesound recordings a team should have avoided, the challenge published the forbidden sound IDs as a list. That made the rule checkable in advance.

That is the case for writing the rubric down rather than carrying it in your head. A rubric for audible evidence is what lets another team dispute a judgement you made about a caption, including the captions the references say nothing about.

Four steps. List the allowed claims: events, counts, timing, order, and acoustic properties supported by audio. List the restricted claims: identity, intent, cause, diagnosis, and emotion without adequate evidence. Create contrast prompts — answerable, ambiguous, misleading, and impossible questions. Review the explanations, requiring time spans or source segments for high-impact claims.

The third step is the one teams skip. It is also the one the AudioCaps probe was built around: 15,110 positive questions were matched with 15,110 negatives, precisely because a question set whose answer is always yes cannot tell a listening model from a reflex.

Say what your evidence policy assumes, then give one counterexample. A matched negative object-presence question where the honest answer is no is one. A Clotho-AQA question whose three annotators did not answer identically is another. Then say what the semantic and behavioral evaluation does about each of them.

Whose ear you trust is not a report. The report has to be a portfolio rather than a headline. And both halves of the portfolio have to be read with their own limits in view.

Start with the ranking half. DCASE 2024 Task 8, Language-Based Audio Retrieval, evaluates on a 1,000-clip set with one caption per clip. “Submitted methods will be ranked by the mAP@10 metric”, the task description states. The official baseline scored R@1 0.130, R@5 0.343, R@10 0.480 and mAP@10 0.222 on the Clotho development-testing split. The top-ranked submission, from Primus at CP-JKU Linz, scored 0.416. Even the winning system leaves most queries without the right clip at the top. Recall was never going to carry a release decision alone.

The caption half is no safer. A 2022 paper asked whether audio captions can be evaluated with image caption metrics, and then did the work to answer. Its authors built the AudioCaps-Eval and Clotho-Eval human-judgement benchmarks. They used pairwise comparison rather than absolute rating, because that was what gave usable inter-annotator agreement. Then they checked the metrics everyone had borrowed from image captioning against those judgements. “Current metrics are found in poor correlation with human annotations on these datasets”, their abstract reports of SPICE and CIDEr. Their replacement, FENSE, beat those metrics by 14–25% accuracy.

So give each metric its unit, the clips it was computed over, how uncertain it is, and the split it was taken under. Then say which of them has been checked against a human judgement at all.