Skip to content
AI.info

Speech and audio

Audio Foundation Models and Audio–Text Representation

Explain audio encoders, spectrogram Transformers, audio–text contrastive learning, zero-shot transfer, prompting, adaptation, and limits.

By the end you can

Key idea

The caption told the model where it came from

An audio foundation model goes wrong in four ways. They are worth naming together, because a strong transfer score survives all four.

Take the first, which has been measured rather than merely warned about. A 123,000-pair audio–text pretraining set was stripped of every word that was not a noun or a verb. The model was trained again on what was left. Retrieval did not degrade. Recall@10 went 0.798 → 0.796 on AudioCaps, 0.505 → 0.524 on Clotho and 0.477 → 0.506 on Clotho 2022. Two thousandths lower on one benchmark, higher on the other two. The 2023 paper that ran the experiment is titled Audio-Text Models Do Not Yet Leverage Natural Language, and it states the result plainly: “For all three benchmark datasets, the model trained on nouns and verbs performs similarly or better than the model trained with full sentences.” Whatever the grammar of those captions was contributing, the model was not using it. It was matching keywords. Three retrieval benchmarks had no way to say so.

That is the shape of all four faults. Caption text hands the model something the audio never carried — the dataset it came from, its context, its keywords — and the model learns that instead of the sound. Zero-shot labels get asked for concepts outside the ontology the model actually covers. Long clips are compressed into a single embedding, and a brief event inside one leaves nothing behind in it. And fine-tuning gains get credited to pretraining when no matched baseline was ever run.

Two of those concern what the representation is able to hold: a concept it never learned, an event too short to survive the averaging. The other two concern how the result was read. None of them shows up as a low number. That is the whole difficulty. Broad, high transfer figures do not rule out a single one.

Strip every word that is not a noun or a verb out of 123,000 caption pairs and Recall@10 moves 0.798 → 0.796 on AudioCaps: the score cannot tell you the model had stopped reading the sentences.

Breadth follows the captions, not the ambition

So it is worth being exact about what the object is. Audio foundation models pretrain encoders or multimodal spaces on broad datasets and objectives, then transfer to real work through probes, fine-tuning, retrieval, or text prompts. What such a model can do follows from four things: the training corpus, the temporal resolution it was built at, the language its labels were written in, and the objective it was optimized for.

The operational boundary follows from those same four. With AudioSet it can be counted rather than described. Its project page states the scale: “AudioSet consists of an expanding ontology of 632 audio event classes and a collection of 2,084,320 human-labeled 10-second sound clips drawn from YouTube videos.” That is about 5.8 thousand hours of released data, split by the download page into 20,383 evaluation segments, 22,176 balanced-train segments and 2,042,985 unbalanced-train segments. Those clips carry labels for 527 of the 632 classes. The ontology names more sounds than the dataset ever labels. That difference is the whole of what “outside the model's actual ontology coverage” means: an event can have a name in the taxonomy and no examples behind it. Coverage stops where the captions stop. Quiet events, rare species, clinical cues, industrial faults and languages nobody wrote captions for sit outside whatever breadth the pretraining bought.

And a zero-shot text match is a ranking against the other candidates on offer. It is not a calibrated probability. It carries no evidence about when in the recording anything occurred. Work that holds up therefore runs from choosing the representation objective through to validating task evidence, with someone named at each end.

Whatever the captions never described stays outside the model's reach: 632 classes in the AudioSet ontology, 527 of them with labels in the released data, and the gap surfaces first in the rare case someone was counting on it to cover.

Example

Asked which sound came first, the model answered at chance

The last of those limits — that a ranking says nothing about when — was put to a direct test in the same paper.

The test was a balanced set of 176 sentences. The same events were described in the two possible orders, and the model had to pick the caption that got the order right. On a balanced two-way choice, a coin scores 50%. The paper reports what the model scored: “The baseline (MLP) yields roughly 50%, indicating it cannot capture the difference between “before” and “after”.”

The preposition test set gave the same answer from the other direction. The baseline's Recall@1 was 0.405 on the original captions and 0.419 when the clauses were swapped into reverse order. Describing the events in the wrong sequence cost the model nothing. On this measure it did slightly better. The embedding held which sounds were present. It did not hold what happened first. A system asked whether one specific event occurred at one specific moment is asking exactly the question these two tests show is not being answered.

A single transfer score, averaged over many tasks, is exactly the kind of number under which that hides.

  • The decision underneath the whole subject is how to explain audio encoders, spectrogram Transformers, audio–text contrastive learning, zero-shot transfer, prompting, adaptation, and where each of them stops.
  • The failure measured here is the third one: a clip compressed into a single embedding that keeps the inventory of sounds and drops their order, 0.405 against 0.419 when the order is reversed.
  • The evidence a claim like this rests on is frozen and fine-tuned transfer measured across diverse tasks — none of which had asked the question the 176-sentence test asked.
  • The practical response is to make that task set include speech, music, environment, localization, and one domain-specific event of the kind the deployment will actually meet.

Case

One frozen embedding, nineteen tasks, twenty-nine models

That test asked a general-purpose embedding to do one specific job. That is a claim testable at scale, and one benchmark tested it properly. HEAR, run as a NeurIPS competition in 2021, evaluated 29 submitted models from 13 external teams. The rule was that a single frozen embedding had to serve 19 tasks built from 16 datasets, with no fine-tuning. Speech, music and environmental sound were all in scope. Breadth was not something an entrant could assert about itself. It was the condition of entering.

The organisers then refused to do the thing benchmarks are usually asked for. “Although it is tempting to give a single score for every model, we believe that would strip out important nuances shown in the full score table,” they write. The nuances ran in both directions. CP-JKU's frozen PaSST embeddings set a new state of the art of 0.641 mAP on FSD50K with no fine-tuning at all — a general representation beating the specialists on their own task. The same organisers reported a negative result on the tasks that defied categorisation, among them Gunshot Triangulation, which has only 88 instances. An average across 19 tasks would have absorbed both.

Breadth of that kind still follows the corpus. Whisper was trained on 680,000 hours of weakly supervised audio. Of those, 117,000 hours spanned 96 languages other than English. That figure says how much of the corpus was not English. It does not say that any particular language of yours was among the 96. Its own paper makes the dependence explicit: “We find a strong squared correlation coefficient of 0.83 between the log of the word error rate and the log of the amount of training data per language.” The hours predict the errors, with the word error rate halving for every 16× increase in data per language. The training set also contains no data at all for 20 of the 102 languages in FLEURS, which upper-bounds language-identification accuracy at 80.4%. It scores 80.3% on the 82 languages that do overlap. Essentially all of the remaining headroom is the twenty it never heard.

Nothing in the model announces where its own boundary sits. A model that has heard little of your domain will still return a confident embedding for it, in the same way that a language with zero training hours still gets identified as something.

Example

One embedding, four jobs

A single frozen embedding serving many tasks is not only a benchmark rule. It is also the product proposition, and four jobs lean on the same embedding for four different purposes.

They are not measured in the same units. Retrieval quality, annotation saved, query success, and whether an event was caught at the right moment are four different questions. Evidence that settles one of them does not settle another. The ordering result is the reminder of which one gets left out. Monitoring is the job the other three cannot stand in for, because it is the only one of the four whose answer depends on when.

  • Audio search leans on shared embeddings for retrieval and for suggesting metadata — the same retrieval that nouns and verbs alone were enough to win on AudioCaps, Clotho and Clotho 2022.
  • Few-label classification uses a pretrained encoder to reduce how much has to be annotated by hand, which is the trade wav2vec 2.0 priced at ten minutes of labels.
  • Multimodal systems depend on audio and text being aligned, which is what makes natural-language queries possible — and what makes the caption's own shortcuts available to the model.
  • Monitoring is the domain-specific event, where the 50% before/after result says the temporal claim has to be validated on your own task rather than inherited from the encoder.

Comparison

Where the language enters the objective

That alignment between audio and text — the thing making a natural-language query possible at all — comes from one particular choice of objective. There are three on offer worth separating. Each has a canonical result attached, and the results are in different units, which is the point.

Audio-only pretraining learns structure from acoustic prediction or discrimination, with no labels and no language involved. wav2vec 2.0 is what that buys. Its 2020 abstract: “Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER.” Those are the Librispeech clean and other test sets. The comparison that gives the number its meaning is the same benchmark with all of Librispeech's labels available: 1.8/3.3 WER. Ten minutes of labels does not equal the full label set. On 53,000 hours of unlabeled speech it comes within a few points of it.

Supervised broad tagging brings labels into the objective. The Audio Spectrogram Transformer, published in 2021, was the first convolution-free, purely attention-based audio classifier. Its abstract states the result: “We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.” Three benchmarks, three different units, one architecture. And 0.485 mAP on AudioSet and 98.1% on Speech Commands V2 are not comparable quantities. Even a single model's own breadth has to be read task by task.

Audio–text contrastive learning brings language into it, aligning recordings and language descriptions in a shared space. It is also the only one of the three whose objective reads captions. That is why the shortcut belongs to it and not to the other two: a model trained on nouns and verbs alone cannot be beaten by full sentences if the sentences were never being used. The ordering here locates the first failure mode exactly. Leakage of the dataset source is a risk that arrives with the language, and not before it.

FigureComparison · 3 columns

Audio-only pretraining

Learns structure from acoustic prediction or discrimination.

  • Decision focus: Choose the representation objective
  • Useful evidence: Frozen and fine-tuned transfer across diverse tasks
  • Watch for: Caption text leaking dataset source and context shortcuts
  • Best used when its assumptions are documented for audio foundation models and audio–text representation

Supervised broad tagging

Learns from a large ontology of clip labels.

  • Decision focus: Audit pretraining coverage
  • Useful evidence: Zero-shot ranking, calibration, and prompt sensitivity
  • Watch for: Zero-shot labels outside the model’s actual ontology coverage
  • Best used when its assumptions are documented for audio foundation models and audio–text representation

Audio–text contrastive learning

Aligns recordings and language descriptions in a shared space.

  • Decision focus: Select adaptation mode
  • Useful evidence: Temporal localization and short-event recall
  • Watch for: Long clips compressed into one embedding that misses brief events
  • Best used when its assumptions are documented for audio foundation models and audio–text representation

Example

A method, a measurement, an objective, an artefact

Contrastive learning was just called an objective, and the word was chosen deliberately. Four terms that circulate together name four different kinds of thing. Reporting them as one capability is how a transfer result turns into a claim nobody tested.

  • Zero-shot classification is a method: it ranks labels without task-specific supervised training, often through text prompts — the ranking that nouns and verbs alone were enough to win.
  • A linear probe is a measurement — a simple classifier trained on representations that stay frozen, which is the kind of measurement left available once a benchmark forbids fine-tuning, as HEAR did across its 19 tasks.
  • Contrastive learning is an objective, one that brings matched pairs closer together and separates the alternatives, and the only one of the three objectives here that has captions to read.
  • A foundation model is the artefact itself, broadly pretrained and then adapted to multiple downstream tasks — wav2vec 2.0 on 53,000 hours, AST on AudioSet, an audio–text encoder on millions of pairs.

Analogy

A multilingual encyclopedia of sounds

Calling that artefact a foundation invites a comparison it half deserves. Picture an encyclopedia built from many recordings and their captions. Broad associations everywhere, local facts missing, headings that exist with nothing filed under them — 632 classes in the AudioSet ontology, 527 with labels in the released data. And entire languages with no entries at all, the way 20 of the 102 FLEURS languages have no Whisper training data behind them. Worse for our purposes, the articles have no sense of sequence. A page can list two events and not say which came first.

The comparison breaks at the point that matters most. An article can be looked up, cited, and corrected. An embedding is a geometric relation that was learned, and there is no entry to check it against. That is why the ordering failure had to be measured with a purpose-built 176-sentence test rather than noticed by reading.

Broad representation is a starting point, not task evidence.

Visual

Compare against a simple domain baseline

Which leaves the check to be built rather than looked up. Two ends were named earlier: choosing the representation objective, and validating task evidence. The first decides what an audit of pretraining coverage would even be looking for — which classes have labels behind them, which languages have hours behind them. The second is where the two are found to agree, or not.

The weakness is that the second end inherits the first. A task set assembled by whoever chose the objective tends to be a task set that objective does well on. What breaks the circle is a comparison from outside it: a simple domain baseline, built on your own domain, run against the foundation model on your own task. That is the discipline behind reporting 4.8/8.2 WER next to the 1.8/3.3 obtained with the full label set, rather than on its own. A number becomes a claim only once something matched sits beside it. Where the gain over that baseline turns out to be small, the pretraining was not what did the work — the fourth failure mode, caught before it ships instead of after.

FigureTimeline · 4 stops

  1. 1. Choose the representation objective

    Contrast audio and text, mask acoustic regions, classify tags, or distill a teacher.

  2. 2. Audit pretraining coverage

    Document domains, labels, languages, devices, durations, and rights.

  3. 3. Select adaptation mode

    Compare frozen probes, adapters, full fine-tuning, retrieval, and prompting.

  4. 4. Validate task evidence

    Test localization, calibration, robustness, and simple domain-specific baselines.

Whatever objective you chose for the representation is already built into the task evidence you later validate, and that check has no way to question it.

Example

Breadth, then what happens when you ask in words

A domain baseline settles one question. The rest of the portfolio has to be assembled on purpose, because its parts answer different things and no one of them can carry a release alone.

Frozen and fine-tuned transfer across diverse tasks is a claim about breadth. Zero-shot ranking, calibration, and prompt sensitivity is a claim about what happens when the model is asked in words. Two more measurements finish it.

The first is hard slices built from captions that name their own source. Insist on it, because a leading audio–text model found the problem inside its own headline number. LAION-CLAP was trained on the 633,526-pair LAION-Audio-630K and on 2.5 million audio–text samples in total. Its authors then checked their zero-shot evaluation sets against that training data. The overlap: 4,875 AudioCaps test clips also sit in AudioSet unbalanced-train, 399 ESC-50 clips in FSD50K-train, 697 UrbanSound8K clips in FSD50K-train. They say what they did about it: “We noticed a dataset overlap between our training data and the zero-shot dataset we are evaluating on. We excluded all the overlap samples and perform zero-shot evaluation on the whole remaining dataset.” The 91.0% zero-shot top-1 accuracy on ESC-50 is the figure after those deletions. Had nobody gone looking, the reported number would have been higher and the word “zero-shot” would have been false. Nothing in the evaluation itself would have flagged it.

The second is robustness across domain, language, device, and subgroup. Without those two, the breadth claim carries the release on its own — and breadth was never what a single timed event needed.

  • For the core task, frozen and fine-tuned transfer across diverse tasks, reported task by task rather than averaged, for the reason HEAR's organisers gave.
  • For how the system behaves in use, zero-shot ranking, calibration, and prompt sensitivity — on evaluation sets checked against the training corpus first, the way 4,875 overlapping AudioCaps clips were found and removed.
  • For the robustness slice, temporal localization and short-event recall, which is the 0.405-against-0.419 ordering test run on your own data.
  • Across the model's working life, robustness to domain, language, device, and subgroup — the axis on which 20 languages with zero training hours are invisible until someone speaks one.

Report frozen and fine-tuned transfer across diverse tasks together with domain, language, device, and subgroup robustness — and deduplicate the zero-shot test set before quoting a zero-shot number.

Key takeaways