VoxLock: Target-Speaker Extraction for Voice Agents Without an Enrollment Step

AI summary

Overview

Research area: Speech processing for voice agents — target-speaker extraction (TSE), speech enhancement, and streaming ASR front-ends (eess.AS).

Technical level: Intermediate. The paper assumes familiarity with speech recognition word error rates, speaker embeddings and streaming inference, but every design choice is explained in plain terms and the evaluation is reported as recognizer output rather than signal metrics.

Scope (one sentence): The paper fine-tunes a public 27M-parameter causal target-speaker extraction model on a single GPU so that a voice agent can isolate its caller using a 3-second sample taken from the call itself, then benchmarks the result against commercial voice-isolation products and open baselines using downstream speech-recognition output on real recordings.

What This Paper Is About

A voice agent's recognizer transcribes everyone near the microphone — a coworker at the next desk, a television, a passenger in the car — and the agent then responds to speech that was never addressed to it. Noise suppressors do not solve this because they are designed to keep speech, whoever is speaking it, and commercial "voice isolation" filters from Krisp, ai-coustics and Sanas are closed and licensed per minute or per device.

Target-speaker extraction is the research answer to the same problem, but it normally requires a pre-recorded enrollment clip of the target speaker, which a live agent does not have — it only has the call. VoxLock closes that gap by cutting the 3-second voice sample from the call itself and by scoring systems on what a recognizer actually writes on real recordings, rather than on signal metrics over simulated mixtures.

Key Contributions

1. A voice sample taken from the call, not from enrollment. The model takes exactly 3 seconds of the caller, chosen as the loudest 3 seconds within the first 15 seconds of a recording, or — for a live stream — as the first 3 seconds of speech detected as 10 ms frames at least 10 dB above the quietest frame of the last 2 seconds. Isolation begins once that sample exists, so the agent passes in a single audio stream.

2. A four-stage fine-tune of a public causal TSE baseline on one GPU. Starting from the causal tfmap_context BSRNN from the REAL-TSE baseline in WeSep (16 kHz, 512-sample STFT, 128-sample hop, six band-split blocks, 27M parameters, 24 ms algorithmic latency), the author trains every weight including the ECAPA-TDNN branch on negative SI-SDR with AdamW across stages S1 (simulated calls, 4,000 steps), S2 (real AMI crosstalk, 4,000 steps), S3 (long calls, 800 steps of 30 s examples) and S4 (distance and room, 400 steps).

3. Recognition-based comparison against three commercial vendors and five open systems. Seven commercial models (four Krisp VI 2.5 variants, two ai-coustics Voice Focus 2.2 sizes, Sanas Voice Isolation SE 1.0) and open baselines (Hush, DeepFilterNet3, TIGER, SAM Audio small, and the untouched REAL-TSE checkpoint given the same sample) are scored on Krisp's 265-recording benchmark, on 134 AMI windows, and on a 400-window blind AMI set frozen by hash before any model was run.

4. An open, CPU-feasible runtime. VoxLock 1.2.1 ships as two ONNX graphs — an enroll graph run once per call, and a chunk graph run per chunk — with the model, code and per-file scores released.

Main Findings

• Krisp benchmark, main result: VoxLock reaches 4.95% WER against 23.21% unprocessed, with 4.22% of caller words missed and 0.11 background words leaked per minute of background-only speech. In counts, 9 background words entered the transcript over 85.3 minutes of background-only speech, versus 130 for Krisp VI 2.5 Default and 5,674 unprocessed.

• Not significantly different from Krisp's best, significantly better than the rest: VoxLock and Krisp VI 2.5 Default (5.30%) are not significantly different on WER, missed words or leak. Krisp Balanced (5.32%) is not significantly different on WER or missed words but leaks significantly more. Against Krisp HD (5.79%) the WER difference is borderline, with an interval starting at +0.01. VoxLock has significantly lower WER than Krisp Lite (6.21%), ai-coustics VF 2.2 L (6.84%) and S (6.91%), Sanas SE 1.0 (9.38%), the REAL-TSE starting checkpoint (9.81%), Hush (12.09%), DeepFilterNet3 (13.30%), TIGER (14.72%) and SAM Audio small (33.77%).

• The causal path holds up: Run causally with live(), VoxLock scores 5.15% WER, still below Krisp Default's 5.30%, with 4.41% missed words and 0.12 leak per minute.

• A cost when there is nothing to remove: In Krisp's phone-call scenario, which has almost no background speech, VoxLock raises WER from 3.82% to 5.36% (Krisp Default: 4.15%), while in the work and call-center scenarios it lowers WER from 29.31% and 25.13% to 4.83% and 4.90%. On 200 caller-only windows from the blind-set meetings (20 people), WER is 0.31 points above unprocessed audio — 7.49% against 7.18%, interval [−0.32, +1.33], not significant.

• AMI, 134 windows: VoxLock has the lowest WER (10.12%) and the fewest missed words (6.77%), ahead of ai-coustics VF 2.2 S (11.85%), ai-coustics VF 2.2 L (12.13%), Sanas (13.30%), TIGER (22.84%), DeepFilterNet3 (26.78%), Hush (33.00%) and unprocessed audio (33.03%). ai-coustics Voice Focus 2.2 S lets fewer background words through — 8 against 16 over 7.5 minutes of background-only speech — and that difference is significant.

• Blind AMI, 400 windows (16 people), frozen before scoring: VoxLock cuts WER from 42.95% to 10.63% and leaked words from 98.19 to 1.17 per minute (23.1 minutes of background-only speech). The untouched REAL-TSE checkpoint reaches only 14.75% WER, and live() 11.97%. Taking the sample from the first 3 seconds of speech via Silero VAD (10.22%) instead of the loudest 3 seconds makes no significant difference on this set (Δ −0.41, interval [−1.54, +0.09]) but raises Krisp WER by 0.13 points.

• It genuinely uses the sample, but does not depend on it: Given a 3-second turn of another person from the same meeting, the REAL-TSE checkpoint follows that person and loses 61% of the caller's words; VoxLock loses 17% (against 8% with the right sample). The model uses the sample, but has also learned to keep the dominant near-field talker, so a wrong sample degrades it rather than redirecting it.

• Most of the gain comes from the first stage: S1 drops Krisp WER from 9.99% to 5.61% and blind AMI from 14.75% to 11.07%. After S1, blind AMI WER moves by at most 0.6 points. Later stages matter on Krisp, where leak falls from 2.25 to 0.11 words per minute and WER from 5.61% to 4.95%. The author notes the first 1,000 steps of S2 were no better than 1,000 more simulated steps, so real crosstalk alone cannot be credited; this is one training run, not a controlled ablation.

• Deployment cost: The 8-bit model (recurrent layers quantized) scores 5.11% WER on Krisp. On a 60-second call with one onnxruntime 1.31.1 thread on a Google Cloud g2-standard-8 (Intel Xeon, 2.2 GHz), the call uses 61% of the thread at 80 ms chunks, 77% at 40 ms chunks, and 110% at 20 ms chunks; on two threads, 20 ms chunks run at 0.86 times real time. Delay is the chunk length plus the 24 ms algorithmic latency. On an NVIDIA L4 the float32 model processes a recording 40 times faster than real time.

• The switch to isolation is not free: live() first runs the network over the call so far (up to 30 s) and discards that output so the recurrent state has heard the call from its start. This costs 2–5 seconds of compute on one CPU thread, once per call, and stalls output at the switch on a single thread.

Methodology in Plain English

The author takes an existing open model that was already designed to pull one speaker out of a mixture in a streaming-friendly way, and adapts it so that its speaker cue comes from the conversation itself rather than a separate enrollment recording. The 3-second sample enters the network twice: once as a spectral pattern that each frame of the call can attend over, and once through an ECAPA-TDNN speaker embedding whose features the separator's band features attend over. During the first seconds of a call, audio passes straight through unchanged; once enough speech has been collected for a sample, isolation switches on.

Training proceeds in four short stages, each starting from the previous one, all minimizing negative SI-SDR and training every weight. The first stage uses simulated calls built from near-field LibriSpeech mixed with one or two background talkers, usually 0–15 dB quieter, with realistic room distances and a telephone channel in 25% of cases. The second mixes in real AMI headset audio from 40 meetings — first exact single-talker windows plus crosstalk from the same headset, then crosstalk with a caller clip pasted in. The third stage uses 30-second examples after the author noticed the model let a continuously talking caller fade by 20 dB within 40 seconds. The fourth breaks a shortcut the model had learned — keeping whichever talker was closer — by varying caller distance independently and adding real room tone. No evaluation meetings or REAL-T data are used in training.

Evaluation is deliberately downstream: transcripts come from Parakeet TDT 0.6B (v2 on Krisp with Krisp's published text normalization, v3 on AMI), and the paper reports caller WER, the share of the caller's words the transcript loses, and how many recognized background words leak per minute of background-only audio. Uncertainty is given as 95% bootstrap intervals over people with 5,000 draws. The most important check is the 400-window blind set, built from the 19 AMI meetings REAL-T uses for evaluation, frozen by hash before any model ran, and covering 11 meetings after 8 failed to download.

Why This Matters

Impact on research. The paper makes two methodological points that are easy to overlook in the TSE literature. First, target-speaker extraction can be evaluated on what a recognizer writes rather than on signal metrics over simulated mixtures, which changes the ranking of systems substantially. Second, a speaker cue drawn from the call itself is workable: the automatic sample gives up little relative to a hand-picked one, and a wrong sample degrades performance rather than catastrophically redirecting the model. The released per-file scores and blind, hash-frozen evaluation set also give other groups a reproducible comparison point where none previously existed for open TSE against commercial isolation.

Real-world applications.

• Voice agents and assistants in shared spaces — open offices, call centers, cars — where anyone near the microphone can end up in the transcript.
• Meeting transcription from headset audio, where the wearer is the intended speaker and other participants are crosstalk.
• On-device or on-server isolation on commodity CPUs, since the 8-bit model runs a live call on 77% of one CPU thread at 40 ms chunks.
• Privacy-sensitive deployment: an open model under a permissive baseline license avoids sending audio to a per-minute commercial service.

Industry relevance. The comparison is directly against the products a company would otherwise buy — Krisp VI 2.5 in four variants, ai-coustics Voice Focus 2.2 in two sizes, and Sanas Voice Isolation SE 1.0 — on the vendor's own benchmark, and the open model is not significantly worse than the best of them on WER while leaking far fewer background words than most. The finding that isolation costs accuracy when there is no background speech (3.82% to 5.36% on phone calls) is a practical warning for anyone deploying such a filter unconditionally.

Future Directions

• Blind comparison against all baselines. The blind AMI set currently contains only VoxLock variants, REAL-TSE and unprocessed audio; Krisp publishes outputs only for its own benchmark, the ai-coustics key was no longer available, and Sanas and the open baselines were not run on it. Completing that table is the obvious next step.

• Multi-caller and non-English operation. The model is English-trained, 16 kHz and single-caller. Extending the voice sample and separator to more than one caller, and to other languages, is untested.

• A more robust automatic sample. Both sampling rules assume the caller is the loudest, earliest talker. When a background talker dominates the first seconds, VoxLock keeps the wrong person — a failure mode the wrong-sample experiments quantify but do not solve.

• Proper ablations and multi-run evidence. Every result comes from one training run in which each stage was designed after inspecting the previous one, and all transcripts come from one recognizer family. Controlled ablations, additional training seeds, and a second recognizer would establish which stage components actually matter.

Target Audience

Speech and audio researchers working on target-speaker extraction, speech enhancement and streaming ASR front-ends; engineers building voice agents, meeting transcription or call-center analytics who need to decide whether to buy a voice-isolation filter or run an open one; and practitioners interested in a concrete example of deploying a causal separator as ONNX graphs within a CPU budget. The paper is also a useful read for evaluation-focused researchers, since the blind-set protocol and the recognizer-based metrics are the most transferable parts of it.

Authors’ abstract

A voice agent's speech recognizer transcribes everyone near the microphone, including people other than the caller. I fine-tuned a public causal target-speaker extraction model (REAL-TSE, 27M parameters) on one GPU to work from a voice sample cut from the call itself, and released it as VoxLock. I score it by what a recognizer writes on real recordings. On Krisp's 265-recording benchmark it reaches 4.95% WER against 23.21% unprocessed and lets 9 background words into the transcript over 85 minutes of background-only speech. Its WER is not significantly different from Krisp's best filters and is significantly lower than the other commercial and open systems I tested. On 400 AMI meeting windows built and frozen before scoring, it cuts WER from 42.95% to 10.63%. The 8-bit model runs a live call on 77% of one CPU thread with 24 ms of algorithmic latency. Model, code and per-file scores are open.