NasoVoce: A Nose-Mounted Low-Audibility Speech Interface for Always-Available Speech Interaction

AI summary

Overview

Research area: Human-Computer Interaction (wearable speech interfaces, silent and whispered speech, multimodal audio enhancement), with CCS categories listed as sound-based input/output, neural networks, interface design prototyping, and mobile devices.

Technical level: Advanced. The work combines a custom dual-sensor wearable prototype with a complex-valued deep neural network, a 104-hour paired dataset, and knowledge distillation from OpenAI Whisper.

Scope: The paper proposes, builds, and evaluates a nose-pad-mounted microphone-plus-vibration-sensor interface that captures normal and whispered speech and fuses the two signals to produce noise-robust speech for AI voice interaction, validated with WER/CER, PESQ, STOI, and MUSHRA.

What This Paper Is About

Voice interaction with AI is convenient, but speaking aloud in public is socially awkward, privacy-risky, and unreliable in noise. Whispered or silent speech solves the social and privacy problems, but existing silent-speech systems support only small command sets, while whispered speech is extremely vulnerable to ambient noise. The authors ask whether placing a microphone and a vibration sensor together on the nasal bridge — close to the mouth and integrable into the nose pads of smart glasses — can capture both air-conducted and bone/skin-conducted speech well enough to support open-vocabulary, noise-robust AI conversation.

Key Contributions

1. A whisper-input mechanism that integrates a microphone and a vibration sensor into eyewear nose pads, reducing ambient noise for both normal and whispered speech while preserving the glasses' appearance and wearability.
2. A deep learning model (D-DCCRN, "Dual-DCCRN") built by extending the DCCRN audio-enhancement architecture to jointly process the real and imaginary parts of both microphone and vibration signals, plus a paired dataset constructed and used to verify the method's actual effectiveness.
3. A dataset of 104 hours of time-synchronized microphone and vibration recordings from 45 participants reading English text, with simulated noise drawn from the DEMAND noise dataset for training.
4. An evaluation spanning recognition (WER/CER with Whisper Large-v2), objective quality (PESQ, STOI), subjective ratings (MUSHRA), and an in-the-wild comparison against Apple AirPods Pro 2 "Voice Isolation" across four everyday environments.

Main Findings

• Whispered speech is captured reliably at the nose bridge: The authors report that the nasal bridge captures bone/skin-conducted speech and, unlike throat or skull-contact sensors that mainly rely on vocal-cord vibration, is positioned to pick up the aerodynamic turbulence and cavity resonance produced during unvoiced whispering.
• Fusion beats the microphone under noise: Enhanced speech (D-DCCRN applied to Mic + Vib) was comparatively robust to noise and, when the noise level was 0 dB or higher, outperformed microphone-only recognition. For whispered speech it consistently beat the vibration-sensor-only condition.
• Vibration alone can be surprisingly strong: For normal speech, Vib sometimes achieved the highest recognition accuracy. The authors attribute this to the MEMS vibration sensor (V2S200D), whose sensing range is much higher than typical IMU sensors cited in prior mixed-modality work (their text references IMU bandwidths around 0–400 Hz versus a stated 10 Hz–2000 Hz sensing range for their sensor, and elsewhere a capability up to 10 kHz).
• Quality metrics favor enhancement: In PESQ and STOI, the enhanced condition outperformed the Mic conditions at all evaluated noise levels (−20 dB, −10 dB, 0 dB, 10 dB). Under the very strong noise condition (10 dB), the Vib condition was better than the Enhanced model.
• Subjective ratings agree up to a point: In MUSHRA, the Enhanced condition consistently outperformed Mic and remained superior to Vib until the noise level reached 0 dB; under the very high noise condition (10 dB), Vib was slightly higher. The authors hypothesize that under high noise the microphone contributes minimally to the enhanced output.
• AirPods Pro 2 Voice Isolation fails on whispers: In the in-train comparison, Voice Isolation effectively attenuated environmental noise for normal speech but almost completely suppressed whispered speech, presumably treating whisper components as background noise, whereas NasoVoce consistently captured whispers in all environments.
• Model cost: D-DCCRN has 438.25M parameters, which the authors report as 28% of Whisper large-v2's 1543.3M parameters. Average processing time is 136.9 ms versus 429.96 ms for Whisper, i.e., 31.8% of Whisper's processing time.

Methodology in Plain English

The researchers built a small sensor module combining a MEMS microphone (Syntiant SPH0141LM4H-1) and a MEMS vibration sensor (Syntiant V2S200D). Both use a PDM interface at a 16 kHz sampling rate with 16-bit digitization and are driven by the same clock, so the two signals are time-synchronized in the digital domain — avoiding the mismatch problem prior fusion work has flagged. The vibration signal is mapped to the left audio channel and the microphone to the right, so the output behaves like a stereo audio stream usable by standard audio interfaces. The module mounts on the nose pad of a commercial smart-glasses frame with a silicone pad that isolates it mechanically from the frame.

To train a model, they had 45 English-fluent, gender-balanced participants aged 25–55 read English text from the Free ST American English Corpus for roughly 2.3 hours each, recording microphone and vibration signals simultaneously for a total of 104 hours. For noisy training and testing, they mixed in samples from the DEMAND noise dataset at RMS levels from −10 dB to 10 dB relative to the clean speech RMS, randomly selecting noise per utterance.

The core model reuses the DCCRN architecture, which pairs a complex-valued convolutional encoder–decoder with a complex-valued LSTM so that both magnitude and phase are processed. The authors chose complex-valued processing because whispered speech lacks harmonic structure and resembles noise, and because the phase structure of near-field whisper turbulence picked up by the vibration sensor differs from far-field environmental noise. The extended model, D-DCCRN, takes both Mic and Vib as input and outputs enhanced speech that can be listened to directly or fed to any ASR system.

Training combines two losses. The audio-enhancement loss targets clean audio using mel-spectrogram MSE plus a scale-invariant signal-to-distortion ratio term. The knowledge-distillation loss uses a frozen OpenAI Whisper encoder and decoder purely as an evaluator: a hard loss measures how well enhanced audio reproduces the pseudo-labels generated from clean speech, and a soft loss minimizes the KL divergence between the token-probability distributions of clean and enhanced audio. The total loss adds the two.

Evaluation was conducted on data disjoint from training, using 1,000 items for ASR and objective quality, plus a subjective study. ASR used 200 utterances from each of four evaluators who were not among the training participants.

Why This Matters

Impact on research. The paper pushes back on two assumptions in silent/whispered speech work: that nose-pad sensing is only useful for normal voice, and that air conduction and bone/skin conduction should be fused with static weights. Its finding that vibration-only can beat fused enhancement under extreme noise is an argument for adaptive, SNR-aware fusion rather than a fixed multimodal model. It also supplies a paired, time-synchronized dataset in an area where such resources are scarce.

Real-world applications.

• Discreet AI voice queries in public — the paper describes users covering the mouth with a hand, which both blocks sound leakage and acts as a social signal that the person is talking to a device.
• Voice input in loud environments such as a busy roadside or a train car, where the authors recorded in-the-wild comparisons.
• Smart-glasses-integrated assistants that combine voice with image input to ask about surrounding objects or nearby equipment.
• Alternative contact sites described by the authors: pressing the device against the philtrum (between upper lip and nose) as a voice-memo-style device, or using the same Vib+Mic pair as an improved throat microphone.

Industry relevance. The work targets the smart-glasses form factor directly, noting that voice-based eyewear faces social acceptability, privacy, and noise-robustness barriers. Its comparison against Apple AirPods Pro 2 "Voice Isolation" positions dual-sensor nose-pad sensing against mainstream commercial beamforming, and its parameter- and latency comparisons against Whisper large-v2 frame the model as a practical pre-processor for existing ASR pipelines.

Future Directions

• Streaming and phone deployment: The authors state that fully streaming speech processing and integration into smartphones have not yet been realized and remain future work.
• Adaptive mode selection: They plan a recognition model that dynamically selects the input configuration — Mic, Mic+Vib, or Vib alone — that yields the highest recognition accuracy under a given condition, motivated by the finding that Vib can win under extreme noise.
• SNR-adaptive gating: They propose an SNR-adaptive gating mechanism that monitors environmental noise in real time and transitions between modes, discarding the microphone signal when air conduction fails.
• Per-user calibration: Because prior work suggests nasal patency modulates accelerometric detectability on the nasal bridge, the authors suggest future versions incorporate per-user calibration and online adaptation to handle day-to-day variability such as rhinitis or mask wear.

Target Audience

Researchers and practitioners in wearable computing, silent and whispered speech interfaces, multimodal speech enhancement, and smart-glasses interaction design; engineers building always-available voice front-ends for AI assistants who need to trade off vocabulary size, wearability, silence, and noise robustness; and HCI designers interested in how sensor placement on the body shapes what speech modalities are physically capturable.

Authors’ abstract

Silent and whispered speech offer promise for always-available voice interaction with AI, yet existing methods struggle to balance vocabulary size, wearability, silence, and noise robustness. We present NasoVoce, a nose-bridge-mounted interface that integrates a microphone and a vibration sensor. Positioned at the nasal pads of smart glasses, it unobtrusively captures both acoustic and vibration signals. The nasal bridge, close to the mouth, allows access to bone- and skin-conducted speech and enables reliable capture of low-volume utterances such as whispered speech. While the microphone captures high-quality audio, it is highly sensitive to environmental noise. Conversely, the vibration sensor is robust to noise but yields lower signal quality. By fusing these complementary inputs, NasoVoce generates high-quality speech robust against interference. Evaluation with Whisper Large-v2, PESQ, STOI, and MUSHRA ratings confirms improved recognition and quality. NasoVoce demonstrates the feasibility of a practical interface for always-available, continuous, and discreet AI voice conversations.