Towards Leveraging Sequential Structure in Animal Vocalizations

AI summary

Overview

Research area: Computational bioacoustics and self-supervised representation learning — specifically, applying discrete audio tokenization (the technique behind symbolic speech units) to animal vocalizations.

Technical level: Intermediate. The core idea is intuitive, but the paper assumes some familiarity with self-supervised speech models (HuBERT), vector quantization, and evaluation metrics such as UAR and Levenshtein distance.

Scope: A first exploration of whether discrete acoustic token sequences derived from HuBERT embeddings can capture and exploit the temporal order of sub-units within animal calls, evaluated on four datasets via distance analysis and sequence classification.

What This Paper Is About

Most computational bioacoustics pipelines extract frame-level feature embeddings and then average them across time, turning each vocalization into an unordered "bag" of features (for example, a functional vector built from the mean, sometimes concatenated with the standard deviation). This throws away the order in which acoustic sub-units occur, even though animal vocal sequences are known to carry communicative and syntactic information. The authors ask whether discretizing animal calls into sequences of symbolic tokens — using vector quantization and gumbel-softmax vector quantization on HuBERT representations — can preserve that temporal structure and still support call-type and caller classification. They state that, to the best of their knowledge, this is the first work to explore discrete audio tokens for bioacoustics.

Key Contributions

1. A first discrete-tokenization framework for animal vocalizations: The paper adapts symbolic audio tokenization — until now used mainly in speech and audio — to bioacoustics, training separate VQ and GVQ quantizers on top of frozen HuBERT embeddings of animal calls (V = 50 codebook entries, D = 768 dimensions, codebook shared across all layers).
2. A pairwise distance analysis of token sequences: Using normalized Levenshtein distance across four comparison categories (intra-caller/intra-calltype, intra-caller/inter-calltype, inter-caller/intra-calltype, inter-caller/inter-calltype), the authors characterize how separable token sequences are by call-type and caller across four datasets.
3. A sequence-classification benchmark: Token sequences are classified with k-NN using Levenshtein distance and majority voting, and compared against a "traditional" linear-probing baseline that pools the same embeddings into a functional vector and classifies with a fully connected layer.
4. An identification of where the approach falls short and why: The paper documents that GVQ suffers apparent codebook collapse, that a single shared codebook is insufficient for caller-specific nuance, and that single-codebook tokenization trails the linear baseline.

Main Findings

• VQ token sequences show the expected distance ordering: For VQ-derived tokens, the same-caller/same-call-type pairs had the smallest mean distance and the different-caller/different-call-type pairs the largest, on average, across all datasets. In most datasets, the different-call-type/same-caller distance also exceeded the same-call-type/different-caller distance, matching the intuition that vocalizations of different call-types from one caller are more acoustically distinct than the same call-type from different callers.
• GVQ token sequences are less consistent: GVQ showed similar trends on Bosshard, but the patterns differed on the other datasets. In Abzaliev the two middle categories were flipped — which the authors attribute to the comparatively large number of callers (80), which increases acoustic variability. On Wierucka and IMV, the inter-caller/intra-calltype distance was unexpectedly smaller on average than the intra-caller/intra-calltype distance, suggesting GVQ tokens do not consistently preserve fine-grained caller-specific information.
• The linear baseline beats k-NN on token sequences everywhere: Across all tasks and datasets, the linear layer over stats-pooled functional vectors outperformed classification of token sequences, indicating that the continuous HuBERT embeddings currently capture more meaningful information.
• GVQ essentially collapses: GVQ underperformed on all tasks and datasets and was near chance level on Abzaliev — for caller identification (CLID) on Abzaliev, chance is 1.25% and GVQ reached 2.90% UAR, versus 59.09% for the linear baseline. The authors read this as strong evidence of codebook collapse to a restricted subset of symbols.
• VQ is weaker than the baseline but reasonable: Best-layer UAR for VQ tokens ranged from 25.24% (Abzaliev CTID) to 54.91% (Wierucka CTID) on call-type identification, and from 17.35% (Abzaliev CLID) to 42.24% (Wierucka CLID) on caller identification.
• Call-type classification is easier than caller identification: CTID yielded higher scores than CLID across all feature representations. Token sequences appear to capture the distinct spectro-temporal patterns that separate call-types, whereas caller identity is carried by subtler characteristics that are harder to preserve after vector quantization.
• Quantization costs performance: Relative to the linear baseline, CTID dropped by approximately 26–39% with VQ and approximately 27–79% with GVQ; CLID dropped by approximately 15–71% with VQ and approximately 46–95% with GVQ.
• Earlier layers remain better after discretization: Even with a single codebook shared across all layers, earlier HuBERT layers tended to give better performance across tasks, consistent with trends reported in other work — so layer differences persist after discretization.
• Codebook size was not the dominant factor: The quantizers use V = 50, and the authors report that in early ablation experiments they did not empirically observe a significant performance change compared with V = 25 or 100.

Methodology in Plain English

The pipeline has three stages.

1. Extract features. Raw audio waveforms are passed through a pre-trained HuBERT encoder (a self-supervised model trained on human speech), producing continuous embeddings with a layer dimension, a frame dimension, and a feature dimension of D = 768. The encoder is kept frozen.

2. Turn frames into symbols. A separate quantization module is trained on those extracted embeddings using the bioacoustic data of interest, so the codebook can adapt to the acoustics of the calls being studied. Two quantizers are compared: standard vector quantization (VQ), which maps each frame embedding to its nearest of V = 50 learned code-vectors by Euclidean distance, and gumbel-softmax vector quantization (GVQ), which uses a Gumbel-Softmax relaxation with a temperature (set to 1.0) and an entropy/KL-style loss to encourage uniform codebook usage, plus a diversity loss term. Quantization is done independently per frame, which preserves temporal order. A single codebook is shared across all layers (rather than 13 separate vocabularies), so token sequences are directly comparable across layers. VQ uses a commitment cost of 0.25 and an optional EMA update; both quantizers were trained with Adam, batch size 32, up to 20 epochs on a Train split with a held-out Val split for convergence monitoring.

3. Evaluate the sequences. After training, token sequences are generated for every vocalization in the dataset. Because batches are repeat-padded to the longest sample, the authors compute each sample's effective frame count and trim the sequence so it reflects the true signal duration. Two evaluations follow. First, all pairwise normalized Levenshtein distances (minimum insertions/deletions/substitutions, divided by the longer sequence length) are computed and grouped into the four comparison categories. Second, k-NN classification with Levenshtein distance and majority voting is run on the token sequences, with k searched over [1, 3, 5, 7, 9] and neighbour weighting over [uniform, distance], optimizing UAR on Val and reporting on Test.

Data: Four datasets are used. IMV (InfantMarmosetsVox) contains 72,920 marmoset samples, 464 minutes, 44.1 kHz, 11 call-types and 10 callers, median length 127 ms. Bosshard contains 13,808 marmoset samples, 37 minutes, 300 kHz, 7 call-types and 8 callers, median 117 ms. Wierucka contains 4,901 marmoset samples, 138 minutes, 125 kHz, 12 call-types and 8 callers, median 1,037 ms. Abzaliev contains 8,034 dog samples, 137 minutes, 48 kHz, 14 call-types and 80 callers, median 655 ms. Tasks are call-type identification (CTID) and caller identification (CLID), with chance performance defined as 100/n_C.

Why This Matters

Most bioacoustics machine learning quietly assumes that the order of acoustic sub-units does not matter, because averaging features destroys it. This paper tests an alternative representation that keeps order intact, and does so honestly — the token approach does not yet beat a simple linear baseline, but it shows that sequential structure carries real discriminative signal and identifies specific reasons the current implementation falls short (codebook collapse in GVQ, insufficient capacity of a single codebook for caller identity). For a field that increasingly relies on speech-model transfer, that is a useful calibration of what discretization does and does not buy.

Real-world applications:

• Automated wildlife monitoring: Token sequences could serve as compact, interpretable representations for acoustic sensors that log species presence, call-type, and individual identity over long deployments.
• Individual identification for conservation: Distinguishing callers supports population tracking, and this work clarifies that caller-specific information is subtle and largely lost under coarse single-codebook quantization.
• Comparative communication science: A discovered inventory of recurring acoustic sub-units, and their ordering rules, would support studies of vocal syntax and the evolutionary origins of human language — the stated motivation for focusing on marmosets.
• Animal welfare and behavioral analysis: In domains like the dog dataset used here, distinguishing vocalizations associated with different emotional or behavioral contexts could support welfare assessments.

Industry relevance: Companies building bioacoustic monitoring hardware, animal-health and livestock analytics, and ecoacoustic platforms can reuse this pipeline's components (frozen speech encoders plus a lightweight adapted quantizer). The paper's negative result on single-codebook quantization gives implementers a clear signal that multi-codebook quantization or richer sequence models are likely necessary before token-based animal call representations become competitive with simple pooled-feature baselines.

Future Directions

1. Multi-codebook quantization. The authors suggest larger architectures such as Residual VQs or Grouped VQ, since a single VQ or GVQ codebook appears insufficient to model entire vocalizations alone, especially for caller identification.
2. Better sequence post-processing. Deduplication (removing consecutive duplicate tokens) and acoustic byte-pair encoding (BPE) could shorten sequences and tighten the alignment between tokens and acoustically meaningful sub-units.
3. More sophisticated sequence modeling. Because k-NN with Levenshtein distance already recovers some call and caller discrimination, the authors argue that dedicated sequence models on the generated tokens could improve substantially over the current results.
4. Diagnosing and mitigating codebook collapse. GVQ's near-chance performance on Abzaliev points to collapse; resolving it — through the diversity and KL

Authors’ abstract

Animal vocalizations contain sequential structures that carry important communicative information, yet most computational bioacoustics studies average the extracted frame-level features across the temporal axis, discarding the order of the sub-units within a vocalization. This paper investigates whether discrete acoustic token sequences, derived through vector quantization and gumbel-softmax vector quantization of extracted self-supervised speech model representations can effectively capture and leverage temporal information. To that end, pairwise distance analysis of token sequences generated from HuBERT embeddings shows that they can discriminate call-types and callers across four bioacoustics datasets. Sequence classification experiments using $k$-Nearest Neighbour with Levenshtein distance show that the vector-quantized token sequences yield reasonable call-type and caller classification performances, and hold promise as alternative feature representations towards leveraging sequential information in animal vocalizations.