Leveraging Audio-LLMs to Filter Speech-to-Speech Training Data

AI summary

Overview

Research area: Speech-to-speech translation (S2ST) data curation, audio large language models (audio-LLMs), and parallel speech corpus filtering.

Technical level: Advanced. The paper assumes familiarity with S2ST architectures (speech-to-unit / discrete acoustic units), speech quality metrics (SNR, MOS, BLEURT, BLASER 2.0-QE), pairwise learning-to-rank, and LoRA-based fine-tuning of multimodal LLMs.

Scope: One-sentence scope: the paper presents a two-stage Rank→Distill pipeline that trains an audio-language model to make keep/drop decisions directly on paired source and target speech, and shows gains of up to +1.4 ASR-BLEU over unfiltered S2ST training on CVSS-C and SpeechMatrix.

What This Paper Is About

End-to-end S2ST systems are trained on large speech corpora that are mined or automatically aligned, so the pairs they contain often suffer from background noise, segmentation errors, speech-pair misalignment, and inconsistent translation fidelity. Existing filters for this setting rely either on hand-designed heuristics (duration ratios, transcript-length ratios) or on semantic-only learned scores such as BLASER 2.0-QE, which do not directly detect acoustic degradations or speech-pair misalignment. The paper's goal is to train an audio-language model that looks at both raw audio streams of a pair and decides whether to keep or drop that pair, using pseudo-labels because large-scale mined speech pairs rarely come with reliable human labels.

Key Contributions

1. A two-stage Rank→Distill strategy for mining reliable supervision from noisy speech translation data: a lightweight ranker produces pseudo-labels, and those labels are distilled into an audio-LLM trained to predict keep/drop directly from paired speech.
2. Demonstration that audio-language models can perform effective speech-conditioned data selection and improve downstream S2ST training, with the fine-tuned Qwen2-Audio filter reaching 22.72 BLEU versus 21.32 for the unfiltered baseline on CVSS-C + SpeechMatrix FR→EN.
3. A construction procedure for supervised training pairs: 15,902 high-confidence positive pairs selected from SpeechMatrix under strict thresholds (SNR ≥ 35, MOS ≥ 2.0, LLM adequacy ≥ 90, BLEURT ≥ 0.8), plus an equal number of negatives generated through controlled degradations sampled from three presets (light, medium, heavy) at a 3:6:1 ratio. The authors state they will release the ranker training data and augmentation configuration.
4. Evaluations beyond the main French→English setting, including an ablation on removing Stage I, removing Stage II, dropping acoustic/perceptual signals, substituting Audio Flamingo 3 for Qwen2-Audio, and a German→English generalization run.

Main Findings

• Full pipeline is best under a matched budget: At approximately 477k retained pairs, the audio-LLM filter retains 477,773 pairs and reaches 22.72 BLEU, versus 21.32 BLEU for the unfiltered 20% SpeechMatrix baseline (614,265 pairs) and 21.27 BLEU for random keep at the same budget. The improvement over the unfiltered baseline is approximately +1.4 BLEU.

• Acoustic and perceptual rules give only modest, threshold-sensitive gains: SNR filtering at ≥ 25 dB (553,800 pairs) gives 21.15 BLEU and at ≥ 30 dB (474,570 pairs) gives 21.46; MOS filtering at ≥ 2.0 (479,132 pairs) gives 21.06 and at ≥ 2.2 (408,410 pairs) gives 20.72.

• Semantic signals matter more than acoustic ones: A LLaMA-70B LLM filter at ≥ 80 retains 577,749 pairs for 22.42 BLEU, while an 8B LLaMA LLM filter at ≥ 80 retains 431,319 pairs for 19.95 BLEU, and BLEURT at ≥ 0.7 retains 434,254 pairs for 21.94 BLEU. In the matched-budget comparison at 477,773 pairs, BLEURT gives 22.09, BLASER 2.0-QE gives 21.71, the 70B LLM gives 22.32, and the proposed audio-LLM gives 22.72. The paper states that the method surpasses a 70B text-only LLM while using an 8B audio model, which the authors take as evidence that speech-conditioned semantic modeling is critical.

• Stage I alone is a strong filter: Using only the Stage I ranker to keep the top 477k pairs yields 21.91 BLEU, and scaling to 577k pairs to match the 70B LLM's data scale yields 22.49 BLEU, slightly above the 70B LLM filter (22.42). The full two-stage framework still performs best at the same budget.

• Stage I matters because synthetic degradations alone are insufficient: Training the audio LLM directly on clean-versus-noisy pairs without the ranker removes only about 1.9% of the mined data (602,593/614,265), which the authors attribute to synthetic degradations not capturing real noise distributions.

• Acoustic and perceptual signals are complementary: Using BLASER 2.0-QE alone (482,591 pairs) gives 21.81 BLEU, and jointly using BLEURT and LLM-7B adequacy (469,712 pairs) gives 22.08 BLEU; both beat the unfiltered baseline but remain below the full multi-signal ranker.

• Model architecture matters: Replacing Qwen2-Audio with Audio Flamingo 3 (405,468 pairs) reduces performance to 21.53 BLEU, which the paper attributes to that model accepting only a single audio input, requiring source and target to be concatenated with silence. Dual-audio conditioning appears more suitable for S2ST filtering.

• Generalization to German→English holds: Training on 127,822 CVSS-C pairs and 482,446 SpeechMatrix pairs with a simplified signal set (SNR, MOS, BLASER 2.0-QE) gives 13.27 BLEU without filtering and 15.14 BLEU (+1.87) after removing 157,071 mined pairs.

• Binary decisions are a limitation: The paper notes the current binary formulation does not allow flexible control of retained data under a fixed budget.

Methodology in Plain English

The researchers first need training examples of good and bad speech pairs, but no human labels exist, so they build supervision automatically. They take SpeechMatrix pairs and keep only those passing strict thresholds on four signals: SNR estimated with Brouhaha, MOS estimated with UTMOS, a translation adequacy score between 0 and 100 from Qwen3.1-Instruct (7B) acting as a judge over whisper-large-v2 ASR transcripts, and a BLEURT score between LLaMA-X machine translation output and the target transcription. This yields 15,902 high-confidence positive pairs. They then create negatives by degrading clean pairs three ways — acoustic noise and reverberation, mild temporal inconsistencies such as cropping or local segment reordering, and compression artifacts — sampled from light/medium/heavy presets at a 3:6:1 ratio. To keep the task from being trivial, they lightly perturb some clean pairs and keep them as positives and add mined hard negatives with poor semantics but acceptable acoustics.

Stage I trains a lightweight ranker on these labeled pairs under a simple preference assumption: a clean pair should rank above a noisy pair. They use LambdaMART as implemented in LightGBM (300 trees, learning rate 0.05, max depth 6, minimum leaf size 20, subsample 0.7) with 10% dev and 10% test splits, optimizing a pairwise ranking loss. The ranker then scores a large unlabeled pool, and the top-K (K = 15k) and bottom-K examples become keep and drop pseudo-labels.

Stage II fine-tunes Qwen2-Audio with 4-bit quantization and LoRA (r = 16, alpha = 32, dropout 0.05) on these pseudo-labeled examples, formatted as chat-style prompts containing the two audio inputs and an instruction, with the assistant response being keep or drop and a causal LM loss. Training runs for 2 epochs (learning rate 2×10⁻⁴, batch 8, gradient accumulation 4) with a 90/10 train-dev split. The S2ST backbone is the speech-to-unit architecture, with the auxiliary tasks (source and target CTC, auto-encoding) removed; training uses 4×A100 GPUs, 300k max updates, FP16, and early stopping with patience 5, with ASR-BLEU (sacreBLEU) reported on the CVSS-C test set using the fixed wav2vec 2.0 ASR from the Fairseq S2UT recipe.

Why This Matters

Impact on research: The work shifts S2ST data filtering from heuristic thresholds and text-mediated semantic scoring toward direct, audio-native decisions, and it demonstrates that a compact audio model can outperform a much larger text-only LLM on this task. It also offers a reusable recipe for bootstrapping supervision where none exists — learn an ordering from weak signals, then distill only the confident extremes.

Real-world applications:

• Curating web-mined or automatically aligned parallel speech corpora before training production speech translation systems.
• Quality control for dubbing, subtitling, and localization pipelines that pair source speech with synthesized target speech.
• Selecting training data for low-resource or domain-specific speech translation, where noisy pairs disproportionately harm small models.
• Pre-filtering large speech archives for dataset release, reducing downstream manual auditing effort.

Industry relevance: Any organization training end-to-end speech translation or speech generation models on mined data faces the same noise, misalignment, and fidelity problems. A filter that operates on raw paired audio avoids an ASR transcription step and its error propagation, and the LoRA-based fine-tuning of Qwen2-Audio on 4-bit quantization shows the approach is feasible without large-scale fine-tuning infrastructure. The reported gain of roughly +1.4 BLEU (and +1.87 BLEU on German→English) is the kind of improvement that translates directly into product quality.

Future Directions

• Budget-aware, probability-based selection: The authors state that the binary keep/drop formulation does not permit flexible control of how much data is retained under a fixed budget, and that future work will explore probability-based, budget-aware selection.
• Better negatives and less trivial separation: The finding that Stage-I-free training removes only about 1.9% of the mined data suggests synthetic degradations poorly match real mined-corpus noise; closing that gap is an open problem.
• Dual-audio conditioning as a design constraint: The Audio Flamingo 3 result (21.53 BLEU, with source and target concatenated via silence because the model takes only one audio input) implies that filtering quality depends on architectures that can natively ingest two audio streams.
• Broader language and corpus coverage: Only FR→EN and DE→EN are evaluated, the latter with a simplified signal set; whether the two-stage pipeline generalizes to other language pairs, domains, and mining sources remains open.

Target Audience

Researchers and engineers working on speech-to-speech translation, speech data curation, and audio-language models; practitioners building parallel speech corpora from mined or automatically aligned data; and graduate students interested in self-training, pseudo-labeling, and learning-to-rank applied to multimodal settings. Readers should already be comfortable with S2ST architectures and standard speech and translation evaluation metrics, since the paper reports and compares results almost entirely through ASR-BLEU and assumes familiarity with signals such as SNR, MOS, BLEURT, and BLASER 2.0-QE.

Authors’ abstract

Large-scale mined corpora provide abundant training data for end-to-end speech-to-speech translation (S2ST) but may contain noise, misalignment, and semantic errors. Filtering noisy data is crucial to maintain robust speech translation performance. We study how to train an audio-language model to make keep/drop decisions on paired speech directly from audio. To obtain reliable supervision without manual labels, we adopt a scalable two-stage Rank-to-Distill strategy. A lightweight ranker generates keep/drop pseudo-labels from noisy speech pairs, then trains an audio large language model to predict keep/drop directly from raw paired speech. The resulting model jointly captures acoustic fidelity and cross-lingual semantic consistency for the selection of speech-conditioned data. Experiments on CVSS-C and SpeechMatrix show consistent improvements over unfiltered training, yielding up to +1.4 ASR-BLEU for end-to-end S2ST.