WhAM: Towards A Translative Model of Sperm Whale Vocalization

AI summary

Overview

• Research area: Machine learning applied to bioacoustics — specifically generative audio modeling of sperm whale communication, sitting at the intersection of deep learning, marine biology, and animal behavior research.
• Technical level: Intermediate. Familiarity with audio tokenization, masked language modeling, and embedding-based evaluation metrics helps, but the paper is written for an interdisciplinary audience.
• Scope: Presents WhAM, a transformer-based model that unifies acoustic translation, synthetic coda generation, and useful learned representations for sperm whale coda classification.

What This Paper Is About

Sperm whales communicate through "codas" — short, patterned sequences of clicks whose rhythm and dialect vary across social groups. Prior computational work either generated coda-like audio with GANs (which cannot be conditioned on an input) or analyzed click timing while ignoring the raw acoustic signal. WhAM addresses this gap by finetuning an existing music-pretrained audio transformer (VampNet) on roughly 10,000 annotated coda recordings to build the first model that can take any audio prompt and translate it into the acoustic style of sperm whale vocalizations.

Key Contributions

1. A unified generative model for sperm whale codas. WhAM is the first model to combine acoustic translation (converting arbitrary audio into coda-like audio), generation (producing novel "pseudocodas"), and classification (supporting downstream tasks via its learned embeddings) in a single architecture.
2. A two-phase finetuning recipe. The authors adapt VampNet from music to animal vocalizations (domain adaptation on FSD, AudioSet, WMMS, and a BirdSet subset), then specialize it on sperm whale codas (DSWP and CETI datasets), using LoRA for efficiency.
3. Rigorous translation evaluation. They show that using Fréchet Audio Distance against a calibrated baseline of natural coda variability, WhAM can render diverse inputs (other marine mammals, digital beeps) statistically indistinguishable from real codas.
4. First expert perceptual study on synthetic codas. Five marine biologists and underwater acousticians judged natural versus WhAM-generated codas across audio-only and spectrogram-assisted tasks, yielding both quantitative accuracy measures and detailed qualitative feedback on what the model gets right and wrong.

Main Findings

• Translation works across wildly different inputs. Using a normalized FAD baseline of 0.21 (the distance between two disjoint sets of real codas), WhAM brought five diverse sources — four non-whale species and artificial digital beeps — into FAD-indistinguishable territory from natural codas.
• Experts struggle to reliably detect synthetic codas. In audio-only two-alternative forced choice, experts achieved 81% accuracy (Fleiss's κ = 0.41); spectrogram assistance improved this only marginally to 83%, suggesting no glaring spectro-temporal artifacts.
• Translation quality varies by source. Walrus-to-coda translations fooled experts most often — detected only 75% of the time on average, and only 50% (chance level) for one expert — while digital beeps and coda-to-coda outputs were easier to flag.
• Rhythm is preserved; fine click structure is not. Experts consistently reported correct inter-click intervals ("clicks occur at the right time") but flagged unnatural within-click properties: overly broadband spectra (natural clicks have a low-frequency bias), inverted peaks, inconsistent amplitude, DC offsets, and background noise that oscillates unrealistically.
• Some synthetic clicks resemble echolocation. One expert noted timing more consistent with echolocation than communication codas — an observation that led the authors to discover echolocation sequences contaminating datasets intended for communication codas.
• Generative training yields useful representations. WhAM embeddings outperformed random and majority-class baselines on coda detection (91.3%), rhythm classification (87.4%), social unit identification (70.5%), and vowel-type classification (85.2%) — despite never being trained for classification.
• Specialized models still win on classification. AVES, BirdNET, and CLAP generally outperformed WhAM on downstream tasks, though WhAM's embeddings were learned purely as a byproduct of generation.
• Fine-tuning is essential for translation, not representation quality. Ablations showed species-specific finetuning was necessary for the acoustic translation capability, but did not significantly improve downstream classification over base VampNet embeddings.
• Training is remarkably cheap. WhAM achieved its results after five days on a single GPU, using datasets orders of magnitude smaller than those behind large audio models.

Methodology in Plain English

The team started with VampNet, a model trained on nearly 800,000 music tracks that works by converting audio into discrete "tokens," masking some of them, and using a bidirectional transformer to iteratively predict the missing ones — much like a fill-in-the-blank exercise for sound. The same internal predictor can then be reused for classification simply by reading its internal representations.

The authors reran this pretraining on animal sounds rather than music, then narrowed the focus to sperm whales. They assembled coda recordings from two main sources: DSWP, roughly 2,500 annotated codas collected over thirteen years off Dominica, and CETI, about 7,600 more recent annotated codas. Both datasets come from boat-based hydrophones and noninvasive animal-borne tags.

Because the model is fill-in-the-blank rather than one-shot generation, the masking pattern can be controlled. Leaving rhythmically important beats unmasked preserves the prompt's timing while the model fills in the rest — which is how WhAM produces context-aware variations of an input coda, or forces an unrelated sound (a walrus call, a beep) into coda-like acoustics.

Evaluation proceeded on three fronts: quantitative Fréchet Audio Distance comparisons (using BirdNET embeddings, chosen after a calibration experiment testing sensitivity to rhythm), a four-task perceptual study with five domain experts, and lightweight classifier heads trained on frozen WhAM embeddings for detection, rhythm, social unit, and vowel classification.

Why This Matters

WhAM is a proof of concept that modern generative audio architectures transfer meaningfully to non-human communication systems, and it raises the bar for how such models should be evaluated — by domain experts, not just automated metrics. Its finding that useful acoustic representations emerge from purely generative training aligns with broader self-supervised learning trends, and its small data and compute footprint lowers the barrier for labs without large-scale resources.

Real-world applications:

• Marine conservation monitoring. Compact coda models could support real-time detection and population identification on autonomous underwater platforms, helping track social units and clans without manual annotation.
• Bioacoustic data augmentation. Synthetic codas could supplement scarce labeled recordings for training detection and classification systems, particularly for rare dialects or under-sampled populations.
• Controlled behavioral stimuli. As quality improves, generated codas could serve as controlled playback stimuli in behavioral experiments, isolating specific acoustic features to test how whales respond.
• Cross-species modeling templates. The two-phase domain-adaptation-plus-specialization recipe and the expert evaluation protocol are transferable to other vocal learning species, from dolphins to songbirds.

Industry relevance spans generative audio research (demonstrating domain transfer of music models to non-musical signals), conservation technology and ocean monitoring, and the emerging field of interspecies communication AI — notably, the work comes out of Project CETI, a large-scale effort to decode sperm whale communication, and involves collaborators from UC Berkeley, Northwestern, Haifa University, CUNY, and Carleton University.

Future Directions

• Improve the audio codec. The current implementation finetunes only the token predictor while leaving the codec frozen and music-derived. Finetuning the codec or building bioacoustic-specific codecs could capture features like the recently discovered 3.7–5.7 kHz vowel-like spectral patterns that a music codec may inadequately represent.
• Fix click-level realism. Expert feedback identified unnatural onset/decay, broadband spectra, and echolocation-like timing. Architectural fixes could include adversarial components or modules encoding known sperm whale click physics.
• Clean up the data. The discovery of echolocation sequences inside coda datasets exposes a broader labeling problem in bioacoustics. Better automated separation of vocalization types is needed before scaling data collection.
• Standardize evaluation. Expanding the expert panel and developing principled, meta-evaluated benchmarks for bioacoustic generation would let researchers compare models on common ground.
• Bridge the semantic gap. WhAM translates acoustics, not meaning. The authors are explicit that semantic translation requires understanding animal cognition and remains a far more ambitious, unresolved goal.

Target Audience

This paper suits machine learning researchers working on audio generation, self-supervised representation learning, or domain adaptation who want a well-executed case study in a non-standard domain. It is equally valuable to bioacousticians and marine biologists evaluating whether generative models are ready for research use — the expert perceptual study and qualitative error analysis will be particularly informative for them. Conservation technologists, interdisciplinary animal-communication researchers, and anyone tracking the ethics and validation practices of AI applied to animal behavior will also find the methodological framework and its candid limitations section directly relevant.

Authors’ abstract

Sperm whales communicate in short sequences of clicks known as codas. We present WhAM (Whale Acoustics Model), the first transformer-based model capable of generating synthetic sperm whale codas from any audio prompt. WhAM is built by finetuning VampNet, a masked acoustic token model pretrained on musical audio, using 10k coda recordings collected over the past two decades. Through iterative masked token prediction, WhAM generates high-fidelity synthetic codas that preserve key acoustic features of the source recordings. We evaluate WhAM's synthetic codas using Fréchet Audio Distance and through perceptual studies with expert marine biologists. On downstream classification tasks including rhythm, social unit, and vowel classification, WhAM's learned representations achieve strong performance, despite being trained for generation rather than classification. Our code is available at https://github.com/Project-CETI/wham