EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams

AI summary

Overview

• Research area: Proactive spoken assistance from egocentric (first-person) video and audio, combining omni-modal spoken dialog, full-duplex interaction, and streaming video intervention.
• Technical level: Advanced. The paper assumes familiarity with multimodal LLM fine-tuning, LoRA, direct preference optimization, speech tokenization, and source separation.
• Scope: The paper introduces EgoVoice, a framework that builds a training dataset and evaluation protocol from HoloAssist, then fine-tunes and preference-optimizes MiniCPM-o 4.5 so that a single model decides when to stay silent and what to say out loud during ongoing physical tasks.

What This Paper Is About

Existing voice assistants are reactive: they answer only after the user asks. But during hands-on tasks such as assembling furniture or following a recipe, users are often too busy, uncertain, or unaware of an impending mistake to ask for help. EgoVoice addresses the joint problem of deciding when to speak and what to say from a continuous first-person video and audio stream, producing spoken guidance without being prompted. The authors build the data, training pipeline, and evaluation metrics needed to study this problem end to end.

Key Contributions

1. A dataset and benchmark for proactive egocentric spoken assistance. Starting from the HoloAssist corpus of 1,673 instructional sessions (124.8 hours), the authors produce 1,249 training sessions (91.7 hours) and 170 held-out benchmark sessions (12.9 hours), each with clean resynthesized speech, separated background audio, time-aligned intervention labels, and spoken guidance targets.
2. Two trained models, EgoVoice-SFT and EgoVoice-DPO. The authors fine-tune MiniCPM-o 4.5 with supervised fine-tuning and then refine it with direct preference optimization, showing the two stages are complementary: SFT retains an edge on content relevance while DPO refines intervention timing.
3. A three-axis evaluation protocol. The protocol measures intervention timing (onset Precision/Recall/F1 and end-vs-action), content relevance (cosine similarity), and overall assistance quality (an LLM-as-a-judge rating on Correctness, Promptness, Efficiency, and Overall).
4. A comprehensive comparison across closed-source, open-source, and trained models, including a pairwise human preference study on Amazon Mechanical Turk.

Main Findings

• Off-the-shelf closed-source APIs rarely intervene: Gemini Live, gpt-realtime, and gpt-realtime-2 all reach onset F1 of at most 0.030 (0.022, 0.030, and 0.024 respectively) on the text-asr channel, because server-side voice activity detection triggers only when the user speaks, and users are largely silent during physical tasks.
• Zero-shot MiniCPM-o 4.5 narrates instead of guiding: It reaches F1 = 0.113 but has the lowest LLM-judge Overall score (1.41), and its end-vs-action mean is far below its median (−4.69s vs −0.40s) because a tail of very long narrations pulls the mean down.
• EgoVoice-SFT improves across the board: Over MiniCPM-o 4.5, with p < 10⁻¹² on both onset F1 and LLM Judge (Overall).
• EgoVoice-SFT beats the text-only ProAssist baseline: F1 0.274 vs 0.195 (p < 10⁻⁴) and cosine similarity 0.684 vs 0.594, even though ProAssist is evaluated on text-llm (its internal text).
• Turning ProAssist into a spoken assistant does not close the gap: The ASR/TTS cascade keeps ProAssist's onset F1 at 0.195, and its speech ends after the user has begun the linked action in 62.0% of matched interventions, versus 24.3% for EgoVoice-SFT and 28.2% for EgoVoice-DPO.
• Only the trained models finish speaking before the user acts: EgoVoice-SFT averages +1.57s and EgoVoice-DPO +1.75s end-vs-action, both with positive medians (+1.52s and +1.62s), close to the human instructor's +1.69s mean / +1.38s median.
• DPO sharpens timing at a small content cost: EgoVoice-DPO raises onset F1 by 0.050 over SFT (p = 0.0007) with both precision and recall increasing, while cosine similarity drops from 0.684 to 0.649. The two variants are not separated by the LLM judge Overall (95% bootstrap CI [−0.224, 0.072], p = 0.42).
• Procedural context matters most: Removing the recipe from the system prompt causes the largest SFT degradation across all metrics except end-vs-action (F1 0.240, the lowest of the SFT variants).
• Compositing reward signals hurts: A preference pair criterion requiring improvement on both F1 and cosine similarity degrades both metrics relative to SFT (F1 0.248, cosine 0.674), so selecting on F1 alone proved more effective.
• Humans prefer both trained models: SFT beats MiniCPM-o 4.5 with 45.6% wins / 23.0% ties / 31.3% losses (p = 0.01); DPO beats MiniCPM-o 4.5 with 45.3% / 21.7% / 33.0% (p = 0.02). DPO vs SFT is a statistical tie (36.1% / 30.6% / 33.3%, p = 0.60).
• Task ordering is well preserved: Kendall's τb is 0.868 for EgoVoice-DPO and 0.841 for EgoVoice-SFT, against 0.850 for ProAssist and 0.749 for MiniCPM-o 4.5.
• Content is locally specific: The matched reference is the closest of five candidates for 59.5% of EgoVoice-SFT pairs and 52.8% of EgoVoice-DPO pairs, against 45.9% for ProAssist and 31.8% for MiniCPM-o 4.5.
• Utterance types move toward the instructor's distribution: Proactive guidance makes up 65.7% of EgoVoice-SFT and 64.2% of EgoVoice-DPO utterances, versus 30.1% for MiniCPM-o 4.5 and 76.7% for the human instructor. Narration falls to 5.4% and 5.5%, close to the instructor's 5.6%. The remaining gap is mostly brief acknowledgments: 70.7% of EgoVoice-DPO's "other" utterances are three words or fewer.
• Two failure modes remain: A positive mean end-vs-action does not guarantee in-time delivery (EgoVoice-DPO still finishes late in 28.2% of matched interventions, EgoVoice-SFT in 24.3%), and recall is the binding constraint, with EgoVoice-DPO at 0.285.

Methodology in Plain English

The authors start from HoloAssist, a corpus where real human instructors verbally guide people through physical tasks while wearing an egocentric camera. The raw recordings are not directly usable: the video carries a single audio channel in which user speech, instructor speech, and background sounds are mixed, and speaker identities were anonymized by pitch shifting, making them sound unnatural.

The processing pipeline has two steps. First, they separate background audio from the mixture using SAM-Audio, a prompt-based source separation model, and verify with Silero VAD that no residual human speech remains, discarding sessions where voice activity is still detected. Second, they synthesize clean user and instructor speech: transcripts are normalized with gpt-5.4-mini and task-irrelevant opening and closing remarks are removed (9.0% of utterances); speech is resynthesized with IndexTTS2 using per-session reference speakers sampled from the English subset of Multilingual LibriSpeech (2,498 training speakers, 340 disjoint evaluation speakers). Each utterance is synthesized multiple times; candidates longer than the original are discarded to preserve timing, and the candidate whose ASR transcription exactly matches the original text, verified by Qwen3-ASR-1.7B, is selected.

Training uses MiniCPM-o 4.5, a streaming full-duplex multimodal model that processes inputs one second at a time. At each one-second chunk the model encodes the current video frame and audio segment and emits either a Listen token (stay silent) or a Speak token followed by text; each text token's hidden state feeds a 0.3B-parameter speech-token decoder that produces S3 speech tokens, converted to a waveform by a streaming flow-matching decoder. Instructor speech is never an input; it is only the target output.

For SFT, sessions are split into 60-second windows at one-second granularity, with each reference utterance centered in a window so every window contains at least one intervention surrounded by listening context, giving 13,676 training windows per epoch. LoRA is applied to the LLM backbone and the speech-token decoder while the vision encoder, audio encoder, and other adapters stay frozen, for 5 epochs at a learning rate of 2e-4 and batch size 32. The loss combines language modeling over the Listen/Speak decisions and text with a weighted speech-token loss (λ = 0.1), and gradients from the speech loss are stopped at the LLM hidden states.

For DPO, the SFT model generates 10 candidate samples per training window under the same context; candidates are scored by the onset F1 metric, the highest becomes the chosen sample and a random one from the bottom three becomes the rejected sample, requiring a minimum F1 gap of 0.10. This yields 5,335 preference pairs from windows across 1,021 training sessions. Training uses length-normalized DPO with β = 0.1 for 1 epoch at a learning rate of 5e-6 and batch size 16, with LoRA applied to the LLM backbone only.

Evaluation matches predictions to reference instructor utterances using the Hungarian algorithm within an asymmetric onset window of −2.5s (early) to +1.5s (late), computing Precision, Recall, and F1 per session and macro-averaging across the 170 benchmark sessions. A separate end-vs-action metric measures the gap between the start of the user's next action and the predicted utterance end, reported on an action-linked subset of 942 of 2,067 reference utterances linked within T = 5 seconds. Metrics are reported on two channels: text-llm (what the model intended to say) and text-asr (what the user actually hears).

Why This Matters

Impact on research. The paper is the first to combine multimodal perception with speech output, simultaneous listening and speaking, and proactive intervention from streaming egocentric video into a single trained model. It also exposes a subtle methodological point: deciding when to speak is not enough, because the length of the resulting speech determines whether guidance arrives before it is needed. The cascade baseline is a direct demonstration, inheriting ProAssist's onset F1 of 0.195 while ending after the user has acted in 62.0% of interventions.

Real-world applications:

• AR and smart glasses assistants that coach a user through hands-on assembly or repair without the user breaking focus to ask questions.
• Procedural guidance for cooking, where the user's hands are occupied and a recipe step may be forgotten.
• Training and skills instruction, where a remote human instructor's interventions can be modeled and replayed.
• Any hands-free setting where an always-on assistant must judge whether speaking is helpful or disruptive.

Industry relevance. The paper situates itself against commercial systems such as Meta Ray-Ban smart glasses and Google Project Astra, and benchmarks against production APIs including Gemini Live and two snapshots of the OpenAI Realtime API. Its central finding for industry is that server-side voice activity detection, which is common in deployed realtime APIs, is a poor fit for proactive assistance because it only triggers when the user speaks.

Future Directions

• Generalization beyond assembly tasks. The dataset derives from HoloAssist, which covers a specific set of procedural assembly tasks; transfer to cooking, outdoor navigation, or social interaction is not guaranteed, and a substantial gap remains between the models and ground-truth instructor behavior.
• Fixing recall and late delivery. Recall (0.285 for EgoVoice-DPO) is the binding constraint, and the models still finish after the user acts in roughly a quarter of matched interventions, so better intervention recall and shorter latencies remain open.
• Handling barge-in in both directions. The protocol plays every utterance to completion and does not model overlap, so neither a user interrupting the assistant nor the assistant yielding mid-utterance is covered.
• Refining the preference reward. The DPO stage improves timing but slightly degrades content, and composite F1-plus-cosine rewards performed worse than F1 alone, indicating that the reward signal has room for further refinement.
• Privacy and deployment safeguards. Continuous egocentric recording may capture bystanders without consent, motivating bystander consent mechanisms, on-device processing to minimize data transmission, and audio redaction protocols.

Target Audience

This paper is most useful to researchers and engineers working on multimodal LLMs, spoken dialog systems, egocentric video understanding, and wearable AR assistants, as well as practitioners building realtime voice APIs who need to know why voice-activity-triggered systems underperform on proactive tasks. Readers without a background in preference optimization, speech tokenization, or streaming video modeling will find the method sections hard going, though the results tables and the failure-mode analysis are accessible. Evaluation researchers will also benefit from the timing protocol, particularly the end-vs-action metric and the distinction between intended (text-llm) and delivered (text-asr) speech.

Authors’ abstract

Wearable augmented reality (AR) assistants are moving toward continuous real-world interaction, where they perceive the user's activity through first-person video and audio and provide timely spoken guidance without being explicitly asked. While proactive video assistants, spoken dialog systems, and egocentric task understanding have each advanced rapidly, existing systems do not address the joint problem of deciding when to speak and what to say from continuous first-person streams. We introduce EgoVoice, a framework for training and evaluating proactive egocentric spoken assistants. From HoloAssist video recordings of real human instructors, we construct clean audio streams through source separation and speech resynthesis, and convert each video session into a format where the model must decide at each moment whether to remain silent or provide spoken guidance. We fine-tune an omni-modal LLM with our data, and further improve its proactive intervention behavior with direct preference optimization. Experiments across closed and open-source models show that existing systems rarely produce well-timed, meaningful proactive interventions, while EgoVoice yields clear improvements in intervention timing, content relevance, and human preference over the zero-shot backbone.