MetaPerch: Learning from metadata for bioacoustics foundation models

AI summary

Overview

Research area: Machine learning for bioacoustics, specifically bioacoustic foundation models and species identification from audio.

Technical level: Intermediate. The paper assumes familiarity with multitask learning, auxiliary losses, embeddings, and standard audio classification benchmarks, but explains its design choices in accessible terms.

Scope: The paper introduces MetaPerch, a bioacoustic foundation model that treats recording metadata (such as location, season, and background species) as auxiliary prediction targets during training, and evaluates it against a matched baseline and other recent models across 17 bioacoustic datasets.

What This Paper Is About

Bioacoustic foundation models are usually trained on citizen science recordings using species labels as the only source of supervision, even though those same platforms store rich metadata like recording location, date, and background species. This leaves a large amount of potentially useful signal unused. The authors ask whether adding metadata prediction as extra training tasks produces richer, more robust representations that transfer better to the domain shifts that occur in real-world passive acoustic monitoring, where recordings are noisier, species may be novel, and geography differs from the training data.

Key Contributions

1. MetaPerch, a collection of metadata-aware foundation models. The authors build a bioacoustic foundation model that adds auxiliary metadata prediction heads and losses on top of a species identification model, released publicly.

2. Evidence that metadata supervision improves species identification. Through extensive experiments across a wide range of datasets and tasks covering diverse taxa and geographic regions, they show gains over a species-only baseline.

3. A large-scale empirical study of metadata design. They study the disentangled importance of 9 diverse metadata sources across 17 bioacoustic datasets, along with loss formulations, mixup handling, missing-metadata handling, and location prediction formulations.

4. A controlled, debiased baseline for comparison. They introduce BioBaseline, derived from the architecture of Perch 2.0 but deliberately excluding source prediction and self-distillation, so that the effects of metadata are not confounded and training is cheaper to iterate on.

Main Findings

• Metadata improves transfer to soundscapes for known species. On the BirdSet benchmark (seven avian datasets), MetaPerch improves mean ROC-AUC by 0.015 over BioBaseline (0.906 vs 0.891) and mean cmAP by 0.006 (0.438 vs 0.432). Its ROC-AUC is 0.002 below the state-of-the-art Perch 2.0, described as a more complex model.

• Gains are largest in underrepresented regions. Grouping BirdSet datasets, MetaPerch improves by an average of 0.025 ROC-AUC on PER (Peru), UHH (Hawaiʻi), and NES (Colombia, Costa Rica), compared with 0.006 ROC-AUC on the North American and Western European datasets (HSN, NBP, SSW, SNE).

• Metadata helps transfer to novel species in focal recordings. On BEANS classification tasks, MetaPerch improves average accuracy by 0.016 (0.870 vs 0.854). Notable per-dataset gains include +0.021 on Watkins and +0.012 on Bats, despite these being aquatic and very high frequency bat sounds, unlike the predominantly avian training data. HumbugDB (mosquito data from different parts of the world) also improves, which the authors attribute partly to encoded location information. Dogs shows better capture of intra-species variation.

• Metadata also helps novel-species soundscapes, but less. On BEANS detection tasks, MetaPerch improves mean cmAP by 0.006 (0.512 vs 0.506), with gains of +0.009 on ENABirds and +0.019 on DCASE. Benefits are limited on Hiceas (marine mammals), RFCX (anurans), and Hainan Gibbons (primates), and the Gibbon dataset accuracy actually drops by 0.019. The authors suspect benefits diminish when species and acoustic shifts occur together.

• State-of-the-art on BEANS overall. MetaPerch achieves SotA performance on BEANS classification and detection tasks, outperforming the next best model by 0.008 accuracy and 0.006 cmAP points.

• Large gains on a globally distributed task. On WABAD (1,192 bird species, including Africa, Asia, and New Zealand), MetaPerch improves over BioBaseline by 0.070 ROC-AUC for the linear probe (0.811 vs 0.741) and 0.018 for the prototype learning probe (0.946 vs 0.928).

• Benefits vary by biome. In WABAD, the Tropical and Sub-tropical biome shows the highest improvement at 0.023, while deserts show a drop of 0.024 ROC-AUC.

• Embeddings encode habitat structure. UMAP visualizations of WABAD windows colored by biome show MetaPerch embeddings are more separable by biome than BioBaseline's, with sub-clusters that the authors suspect correspond to frequently co-occurring species combinations.

• Selected metadata were location, season, and background species. These were chosen after model selection because they were intuitively correlated with species identification and yielded a strong model. Location is treated as S2 cell classification, grouped from neighboring locations.

• Most metadata help individually. Boosts from single metadata sources range from 0.008 ROC-AUC for Day-Part to 0.019 for Background Species. Adversarial training is better for day-part and text embeddings, while sampling rate and quality perform similarly under standard and adversarial training. Background species and location improve most under standard training and drop most under adversarial training.

• Location formulation matters. S2 cell classification at level 7 (average cell area roughly 50,000 km²) performs best, followed closely by Cartesian regression.

• Missing metadata is tolerable. Artificial reduction of location availability still produced noticeable improvement over baseline at as low as 1% availability, though performance decreases as less metadata is present.

• Loss weighting and mixing choices. The best validation performance came from treating each metadata loss weight as a hyperparameter, rather than dynamic equalization or gradient surgery. Mixing any present metadata worked better than mixing only when present in all recordings, and all mixing variants beat no mixup.

• Validation and test performance can diverge. Sorting Vizier trials by validation performance revealed that the median trial performed better on WABAD than the best trial (0.952 vs 0.946), highlighting the difficulty of choosing validation tasks.

• BIRB controlled shift results are mixed. With species from Hawaiʻi and Coffee Farms excluded from training, MetaPerch outperformed BioBaseline at lower shot counts on both the Xeno-Canto held-out set and the soundscape datasets, but the trend reversed for higher shots on the Xeno-Canto held-out set.

Methodology in Plain English

The authors start from a baseline model, BioBaseline, whose design broadly follows the current SotA bioacoustic model Perch 2.0. Audio is sampled into 5-second windows, mixed using mixup, converted to 2D spectrograms, and embedded with an EfficientNet-B3 vision model of roughly 12M parameters. Two classifier heads read from these embeddings: a prototype learning classifier and a linear classifier, both trained with cross-entropy.

Two Perch 2.0 components are deliberately removed. Source prediction, which asks the model to identify which original recording a window came from, is dropped because it may implicitly encode the same information as metadata and because its output head would be enormous given the training corpus. Self-distillation, a two-stage noisy-student training regime, is dropped because it is computationally expensive and requires tuning at each stage.

MetaPerch then attaches small multilayer perceptrons as metadata prediction heads on top of the aggregated embedding, each predicting a metadata value such as location or season as a classification task with a cross-entropy loss. Because metadata is frequently missing, the authors assign placeholder values to missing entries and zero out their loss contributions. Because mixup mixes two recordings, they construct multihot metadata targets from both recordings and mix whatever metadata is present, even if it is not present in all mixed recordings. For metadata that might reflect training bias rather than ecology, they optionally insert a gradient reversal layer between the backbone and the metadata head, which flips gradients so the backbone learns embeddings that are poor at predicting that metadata.

Training uses four datasets: Xeno-Canto (12,308 classes, 13,555 hours), iNaturalist (8,382 classes, 3,077 hours), Tierstimmenarchiv (2,375 classes, 1,462 hours), and FSD50K (198 classes, 80 hours), totalling 14,795 classes and 18,174 hours. Overall this is over 18,000 hours of audio from 1.55M recordings with 1.62M annotations across 14,597 animal species and 198 general sound event classes.

Model selection uses Vizier black-box optimization with identical trial counts for MetaPerch and BioBaseline. Validation uses three task categories: observed-species classification (Powdermill, Caples), one-shot embedding retrieval (BEANS detection training sets, Weldy calltype, Powdermill, Caples), and novel-species linear probing (BEANS classification training sets, DCLDE, NOAA, ReefSet, and other tasks from Ghani et al., 2023). ROC-AUCs are averaged within each category, then combined with a geometric average. Selected hyperparameters are used to retrain models 5 times with different random seeds, and reported results are means.

Why This Matters

Impact on research. The paper shows that metadata already present in citizen science platforms is a usable and underexploited supervision signal for bioacoustics, a field where labeled species data is the dominant focus. By releasing the model and reporting gains over a deliberately weakened baseline, the authors frame their results as a lower bound on what metadata can contribute, and they provide detailed ablations of metadata sources, loss formulations, and missing-data handling that other researchers can build on. The work also surfaces the practical difficulty of validating bioacoustic models, since validation and test performance diverged in their own experiments.

Real-world applications:

• Passive acoustic monitoring deployments in ecologically important or endangered zones such as tropical rainforests and wetlands, where models must handle acoustic and species distribution shifts from focal training recordings.
• Biodiversity monitoring at scale, automating species identification to reduce manual annotation effort.
• Regional or under-surveyed-area monitoring, since gains were largest in underrepresented regions like South America and Hawaiʻi.
• Acoustic surveys spanning multiple biomes and continents, where location-correlated information helps disambiguate species recorded far apart.

Industry relevance. Organizations deploying continuous passive acoustic sensors, conservation technology groups, and platforms hosting citizen science recordings all stand to benefit from models that exploit data they already collect. The paper's released model and the model selection infrastructure based on Vizier point toward practical training recipes. The authors also flag a dual-use risk: models that can localize species of interest could be misused to harass, trap, or poach them. They note they do not release metadata prediction heads, and that the coarseness of S2 level 7 cells (average area around 50,000 km²) is not precise enough for poaching purposes. They also note that human voice or other personally identifiable content may be present in the citizen science training data, and that they do not obfuscate or modify openly licensed research recordings.

Future Directions

• Alternative paradigms for using metadata. The authors propose exploring pre-training or test-time conditioning rather than auxiliary losses during training, as the current simple approach is presented as a lower bound.

• Extending to more metadata and other base models. Applying the approach to other relevant metadata sources and to other base foundation models is described as meaningful follow-up work.

• Metadata-guided data sampling. Metadata could be used to alleviate spatiotemporal biases in training data or to construct training curricula.

• Resolving the validation gap. The observed disconnect between validation performance and WABAD test performance, where the median Vizier trial beat the best on WABAD, raises open questions about how to choose validation tasks that reflect real deployment.

• Understanding when metadata helps. The uneven results across biomes and taxa (limited benefit for marine mammals, anurans, and primates, and a drop on deserts and Hainan Gibbons) leave open the question of when species and acoustic shifts together cancel out metadata's benefit.

Target Audience

This paper is most useful to machine learning researchers working on bioacoustics and audio foundation models, to practitioners building or deploying passive acoustic monitoring systems, and to conservation technology groups that rely on automated species identification. It also suits researchers interested in multitask learning with missing labels or modalities, and in using metadata as auxiliary supervision in domains where labels are scarce and distribution shift is severe. Ecologists and biodiversity monitoring program managers who consume model outputs will find the benchmark discussions relevant, though the methodological detail is aimed at a machine learning audience.

Authors’ abstract

Bioacoustic foundation models rely on large-scale citizen science platforms like Xeno-Canto for geographically and ecologically diverse data. Recent work has shown that supervision alone can produce SotA species detection models when trained on this large-scale data -- however, there remains unutilized potential in the form of recording metadata readily available within these community-driven data hubs. In this work, we explore the use of metadata -- such as location and time -- as auxiliary supervision signals, allowing the model to leverage species-metadata correlations in its learned representation. Auxiliary metadata losses provide additional information beyond vocalizations alone that can encourage a richer, more robust representation that generalizes better to species distribution and acoustic domain shifts -- important challenges for deployment in real-world passive acoustic monitoring (PAM) settings. We introduce MetaPerch, a new foundation model that achieves strong species identification performance across multiple challenging domains and present an extensive empirical study of the effects of 9 diverse metadata sources on 17 bioacoustic datasets.