Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

AI summary

Overview

Research area: Audio-visual speech processing, specifically causal (streaming) audio-visual target speaker extraction (AVTSE), together with a new benchmark for realistic acoustic scenes.

Technical level: Advanced — the paper assumes familiarity with STFT-based time-frequency masking, U-Net encoder–decoders, LSTM/self-attention blocks, contrastive learning, and speech separation metrics such as SI-SNRi and SDRi.

Scope (one sentence): The paper introduces RealSSA (a simulated-plus-real-recorded benchmark with 1–5 dynamic near-field interferers, 10–15 far-field interferers, reverberation, and sudden noise) and Falcon (a single-pass causal separator with an encoder-space speaker contrastive loss) that reaches state-of-the-art extraction quality at a fraction of the cost of prior causal systems.

What This Paper Is About

Streaming audio-visual target speaker extraction uses a target speaker's lip movements to pull that speaker's voice out of a noisy, multi-talker recording in real time. The authors argue that existing work fails on two coupled fronts: standard benchmarks fix the mixture at exactly two speakers and ignore near-/far-field interference, reverberation, and sudden noise, and the high-quality causal architectures are too heavy for edge devices while lightweight ones recover capacity by repeatedly invoking a small separator, which erodes the savings. The paper's goal is to provide both a realistic benchmark (RealSSA) and a single-pass, lightweight causal model (Falcon) that generalizes from simulation to real recordings.

Key Contributions

1. A new benchmark, RealSSA, composed of RealSSA-Sim (controllable simulated scenes across four acoustic scene types with 1–5 dynamic near-field interfering speakers, 10–15 far-field interferers in scenes with far-field interference, sampled RT60, environmental noise, and sudden noise events) and RealSSA-Real (a held-out, real-recorded evaluation set captured with a single mobile device). The authors state RealSSA is the only AVTSE benchmark in their Table 1 that jointly satisfies variable speaker configuration, acoustic-condition control, and real-recorded evaluation.

2. The Falcon architecture, a causal AVTSE method whose Separator is an asymmetric U-Net that performs frequency-axis, time-axis, and joint time-frequency modeling across four resolutions inside a single encoder–decoder pass, avoiding the repeated separator invocation used by SwiftNet-style lightweight models.

3. An encoder-space speaker contrastive loss that reuses the existing audio encoder, pulls the estimate toward the clean target and pushes it away from each near-field interferer with a margin, is discarded at inference (zero inference cost), and transfers across backbones (Falcon, SwiftNet-12, Dolphin).

4. Efficiency results: relative to the causal AV-TFGridNet baseline, Falcon cuts parameters by 91.9%, computation by 20.2×, and GPU inference latency by 11.1×, while achieving state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix.

Main Findings

• Best signal-level separation on all four simulated benchmarks: Falcon reached 7.7 dB SI-SNRi / 8.2 dB SDRi on RealSSA-Sim (vs. 7.6 / 8.1 for AV-TF-GridNet), 14.1 / 14.7 on LRS2-2Mix, 16.4 / 16.7 on LRS3-2Mix, and 10.5 / 11.8 on VoxCeleb2-2Mix. Its largest SI-SNRi gain over Swift-Net-12 was on VoxCeleb2-2Mix, at 1.1 dB.

• Competitive perceptual quality on real recordings: on RealSSA-Real, Falcon's DNSMOS was 3.276, tying Swift-Net-12 and exceeding AV-TF-GridNet (3.030) and Dolphin (2.960).

• Strongest recognizability on simulated scenes, second on real: Falcon achieved the lowest RealSSA-Sim WER among extraction systems at 25.47% (1.84 percentage points better than AV-TF-GridNet), and 32.74% on RealSSA-Real, which ranked second and was 0.74 points behind AV-Mossformer2 (32.00%). Ground-truth WER on RealSSA-Sim was 11.83%.

• Best target-speaker consistency: Falcon had the highest mean speaker similarity on both sets, 43.05 (RealSSA-Sim) and 42.23 (RealSSA-Real), exceeding Swift-Net-12 by 1.26 and 2.19 points respectively. Ground-truth similarity on RealSSA-Sim was 64.69.

• Large efficiency margins at equal model size: Falcon and Swift-Net-12 both have 0.5 M parameters, yet Falcon needs 10.6 G MACs versus 45.3 G (a 76.6% reduction) and 51.3 ms versus 172.1 ms GPU latency (70.2% lower). Against AV-TF-GridNet (6.2 M parameters, 213.8 G MACs, 567.8 ms GPU latency), Falcon reduces parameters by 91.9%, MACs by 95.0%, and GPU latency by 91.0%. Against Dolphin (5.6 M), the reductions are 91.1%, 41.4%, and 41.6%.

• Lowest memory footprint measured: 125.3 MB CPU RAM and 364.7 MB GPU memory, versus 482.3 MB and 411.7 MB for Swift-Net-12 and 136.3 MB and 1015.8 MB for AV-TF-GridNet.

• Simulated training data improves real-world generalization: training on RealSSA-Sim instead of VoxCeleb2 raised DNSMOS for all five evaluated architectures and NISQA for four; Falcon went from 2.431 to 3.615 NISQA and 2.878 to 3.276 DNSMOS.

• Depth helps up to a causal limit: going from one to three downsampling stages improved SI-SNRi by 0.6 dB (7.1 → 7.4 → 7.7). Three stages is the maximum permitted by streaming causality, because the visual branch imposes a fixed algorithmic lookahead of 120 ms and the audio hop is 8 ms, leaving at most 15 frames of audio lookahead.

• The contrastive loss transfers across architectures: it improved Falcon by 0.4 dB SI-SNRi (7.3 → 7.7), SwiftNet-12 by 0.2 dB (7.3 → 7.5), and Dolphin by 0.3 dB (6.7 → 7.0), at zero inference cost.

• Temperature annealing matters: annealing τ from 0 to 1 gave 7.7 / 8.2 SI-SNRi / SDRi, beating fixed τ=0 (7.5 / 8.1), fixed τ=1 (7.4 / 7.9), and annealing 0→2 (7.5 / 8.0).

Methodology in Plain English

Building the data. The authors first construct RealSSA-Sim from four scene templates — small meeting rooms, medium indoor spaces, large public spaces, and outdoor streets — each with its own room-size range, reverberation time (RT60), speaker-distance ranges, signal-to-interference ratio (SIR) range, and probability/count of sudden noise events. A speaker counts as near-field if within 3 m of the microphone and far-field otherwise. For every sample, the pipeline randomly picks a scene type, samples room geometry and RT60 (impulse responses computed with gpuRIR), places a target speaker in the near field plus interfering speakers per the scene's distance ranges, then mixes: target normalized to −23 LUFS (EBU R128), interferers and noise added at the scene's SIR, output peak-limited to −1 dBFS, all single-channel 16 kHz. Target audio-visual data come from LRS3, interferers from LibriSpeech, continuous noise from WHAM!, and sudden noise from Hive. This yields 100,826 training, 9,133 validation, and 9,169 test samples (~153 hours), with an average of 2.5 near-field interfering speakers. Separately, RealSSA-Real was recorded on a single mobile device in everyday indoor, public, and outdoor settings and only format-converted (16 kHz audio, 25 fps video, 96×96 grayscale mouth crops); it is used strictly for evaluation.

The model. Falcon freezes a FRCNN visual backbone (as used in CTCNet) to encode mouth crops, and encodes audio with an STFT front end (window 256, hop 128, i.e. 8 ms at 16 kHz) that stacks real, imaginary, and magnitude parts into a three-channel tensor. Each modality is refined first — audio through the separator block, video through SwiftNet's causal-LSTM VideoBlock upsampled to the audio frame rate — and then fused by adding the visual feature, expanded over frequency, to the audio feature.

The separator. The fused feature goes through an asymmetric U-Net with four levels. The encoder downsamples with depthwise convolutions (stride-1 at the first level, stride-2 5×5 thereafter), using left-aligned padding along time and symmetric padding along frequency so there is no temporal lookahead. Heavy time-frequency modeling happens only at the bottleneck, producing one coarse "scene context," while each decoder level applies its own Time-Frequency Refinement Block — bidirectional LSTM over frequency, unidirectional LSTM over time, then 2D self-attention with a block-causal mask — so four distinct time-frequency scales are covered in one forward pass. A Global Context Injection module gates the bottleneck context into each skip connection. Normalization is a frame-local, cache-free causal layer norm.

The loss. The mask is applied by complex multiplication in the encoder's own spectral space and converted back with iSTFT. On top of the SI-SNR reconstruction loss, the authors add a contrastive term computed entirely in that same STFT encoder space: a pooled embedding of the estimate is pulled toward the pooled clean target and pushed away from each pooled near-field interferer, with a margin of 0.5 (a 60° angular margin) that switches off the repulsion once the estimate is far enough away. References use stop-gradient, the interferer weights use a softmax with an inverse temperature τ annealed from 0 to 1, and far-field interferers are deliberately excluded because their residual falls below the SI-SNR error floor. The total loss is reconstruction plus 1.0 times the contrastive term.

Training and evaluation. All models were trained with AdamW (learning rate 1×10⁻³, weight decay 0.01), halving the learning rate after 15 epochs without validation improvement and early stopping at patience 30. Quality was measured with SI-SNRi/SDRi on simulated sets and with DNSMOS, NISQA, WER (Mega-ASR), and speaker similarity (pretrained ResNet34 from WeSpeaker) on real recordings; efficiency was measured on a 2 s, 16 kHz clip with 50 mouth-crop frames at 25 fps. Each baseline was retrained on the same splits and pipeline, with Dolphin and AV-Mossformer2 converted to causal variants.

Why This Matters

Impact on research. The paper reframes causal AVTSE evaluation: instead of a fixed two-speaker, anechoic-style mixture, it asks whether a model survives varying speaker counts, near-/far-field geometry, reverberation, and sudden noise — and it supplies both a controlled simulator and a held-out real-recorded test set for that question. It also shows that gains from a better training distribution transfer across architectures (DNSMOS improved for all five models tested), and it offers a training-only loss that any audio-encoder AVTSE backbone can adopt without architectural change or inference cost.

Real-world applications (from the paper's framing):

• Video conferencing, where a target participant must be isolated from other talkers and room noise in real time.
• Hearing-assistive devices, which need causal, low-latency, low-memory processing on constrained hardware.
• Small meeting rooms and medium indoor spaces, where the benchmark's near-field interferers and far-field background talkers mirror close-range conversation.
• Large public spaces and outdoor streets, where long RT60 or high ambient noise must be handled alongside many simultaneous interferers.

Industry relevance. The reported numbers target the practical blockers for edge deployment: 0.5 M parameters, 10.6 G MACs, 51.3 ms GPU latency, 125.3 MB CPU RAM, and 364.7 MB GPU memory, all with state-of-the-art separation quality. That combination is what determines whether streaming AVTSE ships on a phone, earbud, or conference bar rather than a server.

Future Directions

• Extending beyond three downsampling stages. The authors note that three stages is the hard ceiling because the visual branch's 120 ms lookahead and the 8 ms audio hop leave only 15 frames of audio lookahead; a fourth stage would break audio-visual alignment. Reducing visual lookahead could unlock deeper, higher-capacity separators.

• Broadening the real-recorded evaluation. RealSSA-Real currently serves only as a held-out test set (no ground-truth isolated targets are available, as Table 4 marks with "–"). Extending it toward paired reference recordings would enable intrusive metrics on real capture and allow error analysis beyond DNSMOS, NISQA, WER, and similarity.

• Richer interferer and noise modeling. Far-field interferers are excluded from the contrastive negatives because their residual falls below the SI-SNR error floor; a training signal that still exploits them, and handling of simultaneous far-field and sudden-noise events more explicitly, remain open.

• Scaling and validating the contrastive objective. The loss was validated on three backbones at fixed margin 0.5 and a 0→1 τ anneal. How far the margin and schedule should scale with the number of near-field interferers, and whether the gain compounds with larger models, is not established.

Target Audience

Researchers and engineers working on speech separation, target speaker extraction, and audio-visual learning who care about causal/streaming constraints; benchmark designers interested in simulation-to-reality evaluation; and practitioners deploying low-latency speech enhancement on edge hardware who need concrete parameter, MAC, latency, and memory trade-offs. Readers should be comfortable with time-frequency masking and deep sequence models, as the method section is dense with architectural detail.

Authors’ abstract

Streaming audio-visual target speaker extraction (AVTSE) is essential for latency-sensitive applications. However, current causal systems face two coupled gaps. First, standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes. Second, high-performance causal architectures are too heavy for edge deployment. Lightweight alternatives try to reclaim capacity by repeatedly invoking a small separator, which offsets the savings. To address the first gap, we release RealSSA, a realistic AVTSE benchmark consisting of RealSSA-Sim and RealSSA-Real. RealSSA-Sim provides controllable simulated scenes with 1--5 dynamic near-field interfering speakers; scenes with far-field interference additionally include 10--15 far-field interfering speakers. RealSSA-Real provides a held-out real-recorded evaluation set. To address the second gap, we propose Falcon, a causal AVTSE method. Its Separator performs multi-scale time-frequency separation in a single encoder--decoder pass. We train Falcon with an encoder-space speaker contrastive loss. This loss suppresses near-field leakage at zero inference cost and transfers across backbones. Falcon achieves state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. Compared with the causal AV-TFGridNet baseline, it cuts parameters by 91.9%, computation by 20.2 times, and GPU inference latency by 11.1 times. Code and demo are available at https://cslikai.cn/Falcon/.