Proactive Hearing Assistants that Isolate Egocentric Conversations

AI summary

Overview

Research area: Speech processing and dialogue systems — specifically proactive hearing assistance, target conversation extraction, and egocentric audio.

Technical level: Advanced (assumes familiarity with speech separation, beamforming, attention architectures, and signal-quality metrics like SISDR and PESQ).

Scope: The paper introduces a real-time, on-device proactive hearing assistant that infers the wearer's conversation partners from egocentric binaural audio using conversational turn-taking cues and self-speech as an anchor, without any user prompts.

What This Paper Is About

Existing augmented hearing devices are reactive: the user must manually select which speaker to amplify, which fails when conversations are spatially dispersed or involve multiple people. This paper asks whether a hearing assistant can instead be proactive — automatically figuring out who the wearer is talking to by watching the natural back-and-forth rhythm of conversation, and then isolating and enhancing just those voices. The goal is a system that runs on wearable hardware in real time, adapts as speakers join or leave a conversation, and suppresses everyone else.

Key Contributions

1. Self-speech anchoring mechanism. Rather than requiring explicit speaker enrollment or stored voice embeddings, the system uses the wearer's own voice — isolated by a neural beamformer pointed at their mouth — as the natural anchor for identifying who they are conversing with.

2. Dual-model architecture for real-time operation. A fast LSTM-based streaming model processes 12.5 ms audio chunks for low-latency extraction, while a slower attention-based model runs once per second to capture longer-range conversational dynamics and produce a conversation embedding.

3. Training pipeline using spatialized synthetic data. Because egocentric multi-conversation datasets with third-person binaural recordings are scarce, the authors synthesize mixtures from Candor, LibriTTS, and RAMC, spatializing them and perturbing silence/overlap to close the distribution gap with real in-person conversation.

4. Real-world end-to-end evaluation. A 6.8-hour dataset of binaural egocentric recordings from 11 participants in 2- and 3-speaker conversations, plus out-of-distribution tests on English (SpokenWOZ), Mandarin (RAMC), and Japanese conversation data.

Main Findings

• High partner-selection accuracy. On open-source conversational datasets the system achieves 80–92% accuracy in picking the correct conversation partner, with confusion rates of only 1.5–2.2% for selecting an interfering speaker instead.

• Strong speech quality gains. SISDR improvements of 7.22–11.95 dB and ΔPESQ gains of 0.26–0.65 across spatialized and non-spatialized test sets, versus a uniform speech enhancement baseline (DeepFilterNet2) that actually degraded target speech.

• Large margin over a single-model baseline. A single fast streaming model without the slow conversation embedding achieved only 1.45 dB SISDRi versus 12.48 dB for the dual-model — showing the long-range context is essential.

• Generalizes to unseen speaker counts and languages. Trained on 2–3 speaker English conversations, the model still reached 11.85 dB on 5-speaker tests and 6.5 dB / 7.92 dB on Mandarin and Japanese data, suggesting it learns turn-taking patterns rather than language-specific acoustics.

• Adapts to speaker switching. In the "leaving" test set, a speaker who departs the target conversation receives positive SISDRi while engaged and negative SISDRi after switching to an interfering conversation — the model correctly drops them.

• Real-time on embedded hardware. The fast model processes 12.5 ms chunks in 8.9 ms on an Orange Pi 5B; the slow model processes 1-second chunks in 41.3 ms on Apple M2 silicon. Peak memory is 86.33 MB (fast) and 591.47 MB (slow).

• Substantial subjective improvement. In a user study with 11 participants, overall mean opinion score rose from 1.88 (mixture) to 4.30 (model output) on a 5-point scale.

• Self-speech beats speaker embeddings. Substituting a 256-dimensional d-vector of the wearer's voice for the raw self-speech dropped SISDRi by 2.65 dB.

• Turn-taking structure is load-bearing. Perturbing inter-utterance silence progressively degrades performance (6.75 dB with no perturbation down to 4.16 dB at 3-second SD shifts), confirming the model relies on conversational timing.

• Known failure modes. Performance collapses during extended wearer silence (over 2 minutes), and drops to 4.98 dB when target and interfering conversations change turns within 1 second of each other.

Methodology in Plain English

The researchers start from a simple observation: when you are in a conversation, you and your partner take turns speaking, with predictable gaps and little overlap. If a system knows when you are speaking, it can use that as a signal to figure out who is responding to you.

Step one — find the wearer's voice. The system uses a neural beamformer on the two microphones in the wearable headset to point at the wearer's mouth and extract their own speech. This becomes the anchor.

Step two — infer the conversation. A slow model runs once per second, taking both the raw audio mixture and the extracted self-speech. It uses attention over past seconds to build an embedding that represents the current conversational context — effectively "who is in this conversation right now."

Step three — extract in real time. A fast streaming model runs every 12.5 milliseconds on short audio chunks, conditioned on the embedding from the slow model. It outputs the target conversation, which the device plays to the wearer. The slow model runs a second behind, so its latency does not affect streaming.

Splitting the work this way avoids the quadratic memory blowup of attention over long conversations: only the slow model attends to long context, while the fast model stays lightweight enough for embedded hardware.

Training. Since real egocentric recordings of overlapping conversations are hard to collect, the authors take clean 2-speaker conversation datasets (Candor in English, RAMC in Mandarin) and synthetic sets built by reassigning LibriTTS speakers to RAMC timestamps, then spatialize them to simulate egocentric geometry. They then add interfering conversations and noise, pretrain, spatialize further, and finally fine-tune with perturbed silence and overlap patterns to match real in-person conversation statistics.

Evaluation. Beyond open-source benchmarks, they recruited 11 people, had them wear binaural microphones in real 2- and 3-speaker chats, and mixed those recordings against passive listener recordings to create realistic interfering conversations.

Why This Matters

Impact on research. The paper bridges speech separation and dialogue modeling. Most source-separation work treats target selection as a signal-processing problem; this work shows that conversational turn-taking — a core NLP concept — can serve as the selection mechanism itself. It also demonstrates that a model trained purely on synthetic spatialized data can generalize to real egocentric audio, which matters for a field where realistic multi-party data is expensive.

Real-world applications:

• Hearing aids and earbuds that automatically amplify whoever the wearer is talking to at a noisy dinner table, without requiring the user to point or tap.
• Assistive devices for older adults or users with limited dexterity, for whom manual source-selection interfaces are impractical.
• Remote or hybrid meeting tools that isolate a participant's local conversation while suppressing nearby side conversations.
• Robotic or LLM-agent systems that need to track who is speaking to them in noisy, multi-party physical environments.

Industry relevance. Major consumer audio platforms (Apple, Google, Samsung) are already shipping rudimentary "conversation awareness" features that lower background volume on detecting wearer speech, but none perform speaker separation or conversational tracking. This paper shows a path to a substantially more capable version running on commodity embedded silicon.

Future Directions

• Handling overlapping speech onset. When target and interfering conversations change turns within one second of each other, SISDRi falls to 4.98 dB. The authors suggest lightweight content-aware models could disambiguate these collisions, since turn-taking timing alone is insufficient.

• Sustained wearer silence. The model relies on self-speech as an anchor, so it fails during extended periods of wearer silence (over 2 minutes). Detecting and entering a passive-listening mode is an open problem.

• Supervised adaptation to real conditions. The model generalizes to real egocentric recordings without fine-tuning, but the authors note that adaptation to real acoustic and conversational conditions would likely improve performance — particularly for 3-speaker conversations, where turn-taking dynamics in the training data were fully synthesized.

• Language- and culture-specific turn-taking. Cross-linguistic generalization was demonstrated on English, Mandarin, and Japanese, but cultural differences in turn-taking timing suggest language-specific fine-tuning would improve robustness.

• Safer failure modes. The ethics section calls for transparent design and user-centric controls — such as a tactile override button — given the risk that incorrect speaker detection could suppress relevant voices in a high-stakes conversation.

Target Audience

This paper suits researchers and engineers working on speech separation, hearing assistive technology, egocentric audio, and multi-party dialogue systems. It is also relevant to product teams building on-device audio ML for wearables, and to NLP researchers interested in how dialogue-level phenomena like turn-taking and backchannels can be operationalized as real-time signals. Readers will need background in speech enhancement metrics and neural audio architectures to fully parse the evaluation tables.

Authors’ abstract

We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts. Our system operates on egocentric binaural audio and uses the wearer's self-speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others. To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for low-latency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics. Results on real-world 2- and 3-speaker conversation test sets, collected with binaural egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings. Our work marks a step toward hearing assistants that adapt proactively to conversational dynamics and engagement. More information can be found on our website: https://proactivehearing.cs.washington.edu/