Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents

AI summary

Overview

Research area: Computer vision and multimodal machine learning, specifically egocentric vision, vision-language models (VLMs), and assistive wearable AI agents.

Technical level: Intermediate. Readers should be comfortable with multimodal benchmarks, multiple-choice evaluation, and LLM-as-judge scoring, but no specialized mathematics is required.

Scope in one sentence: The paper introduces WAGIBench, a new dataset and benchmark for inferring a wearable-agent user's goal from egocentric video, audio, digital app state, and longitudinal history, and reports human and VLM baselines on it.

What This Paper Is About

Assistive wearable agents (for example, smart glasses) currently require the user to explicitly state a query before they can help. This paper studies the complementary problem of goal inference: predicting what the user wants from passive, multimodal context alone, which would remove the need for the user to speak or type a request at all. The authors build WAGIBench, a benchmark with ground-truth goals and four context modalities, and measure how well modern vision-language models perform against human raters.

Key Contributions

1. A new scripted egocentric dataset for goal inference. 3,477 (observation, goal) pairs collected from 348 participants, totaling 29 hours of multimodal data, with reference goals that are scripted rather than LLM-annotated, plus accompanying vision, audio, digital, and longitudinal context.

2. The first benchmark incorporating all four modalities. Prior wearable-assistant benchmarks such as MM-Ego and PARSE-Ego4D use Ego4D with LLM-derived narrations as ground truth; EgoLife is longitudinal but uses LLM captions. WAGIBench is the first to combine egocentric vision, egocentric audio, digital context, and longitudinal history with scripted ground truth and a longitudinal design.

3. A validated LLM-judge evaluation protocol. The authors meta-evaluate generative scoring against human raters and show an LLM judge conditioned on reference goals and script cues matches human-human agreement, making it a viable substitute for human judges.

4. Systematic baselines and modality ablations. Multiple model families and sizes are evaluated on both discriminative (MCQ) and generative tasks, with controlled ablations of which context modalities are supplied.

Main Findings

• Humans outperform models. On a high-agreement subset, human raters reach 93% multiple-choice accuracy on similar distractors versus 84% for the best model, and 97% on dissimilar distractors versus 91% for the next best model. Human inter-rater agreement is 90.8% on the choice-consistency metric. The conclusion section restates large VLM accuracy as 83%.

• Generative performance is far from usable. State-of-the-art models produce relevant goals only 55% of the time. In the full-dataset generative results, GPT-4.1 scores 0.5498, Qwen-72B 0.4980, InternVL-78B 0.4866, Qwen-7B 0.3999, InternVL-8B 0.3503, Llama3.2-11B 0.3197, Qwen-3B 0.2468, and InternVL-2B 0.2134.

• Bigger models are consistently better. Within every model family, larger variants outperform smaller ones on both MCQ and generative tasks. Full-dataset MCQ scores are 0.8774 for GPT-4.1, 0.8755 for Qwen-72B, 0.8680 for InternVL-78B, 0.7754 for Qwen-7B, 0.7153 for Qwen-3B, 0.6741 for InternVL-8B, 0.4422 for InternVL-2B, and 0.4311 for Llama3.2-11B.

• Relevant extra modalities help; irrelevant ones do little harm. Multi-modal context significantly beats vision-only context, with modality-specific gains as large as 35% on MCQ and 30% on generative evaluation.

• Digital and longitudinal modalities are noisy. Synthetic high-signal versions of these modalities outperform their real counterparts: VD* beats VD by as much as 12%, and VL* beats VL by at most 5.6% on the S_VL subset. Large models narrow this gap, suggesting they filter noise better.

• Mixing all modalities can confuse smaller models. With VADL input, large models disentangle relevant from distracting features, while small and medium models see interference.

• The LLM judge is the best automatic evaluator. A judge given reference goals plus script cues achieves 76.8% pairwise alignment with humans, statistically indistinguishable from human-human agreement at 75.2%. Snap-MCQ reaches 67.8%, Socratic context 63.0%, and sentenceBERT similarity 59.5%.

• Collecting the data required heavy filtering. The raw collection was 264 hours of video; an initial quality review reduced this to 155 hours, and context-windowing reduced it further to 29 hours.

Methodology in Plain English

Rather than mining existing video datasets and labeling goals after the fact with an LLM, the authors wrote scripts. Each of 165 scripted scenarios defines a situation, a set of steps, and a reference goal, with variable slots (for example, a specific recyclable item) that participants fill in themselves, which introduces natural diversity. Participants, limited to a fixed set of 165 scenarios each, recorded these scenarios with Meta Aria glasses on multiple days.

Audio and video come from the glasses. Digital context (the internal state of seven apps: Calendar, Messaging, Notes, Search, Videos, Maps, and Music) is synthesized by LLMs conditioned on digital cues in the script, so no real private app data is used. Longitudinal context is a history bank of five earlier observations from the same participant in the same environment, represented as VLM-generated text captions checked for consistency across two captioners (Qwen2.5-72B and InternVL-78B), plus audio transcripts.

Quality control is done by three raters per video, scoring script adherence, variable annotations, and a context window. A recording is accepted if at least two raters accept it, variable agreement exceeds 0.5, and the average pairwise intersection-over-union of context windows exceeds 0.7. Roughly 80% of recordings survived.

Evaluation uses two paradigms. Discriminative evaluation poses multiple-choice questions with three distractors drawn from the dataset itself: "similar" distractors sampled from the 95th to 99th percentile of sentenceBERT similarity to the reference goal, and "dissimilar" from the 0th to 80th percentile, with a greedy sampling rule to keep the option set diverse. Generative evaluation asks models for an open-ended goal and scores it with an LLM judge (DeepSeek-R1-Distill-Llama-70B) that outputs 1.0 for very relevant, 0.5 for borderline relevant, or 0 for irrelevant, along with a reasoning trace.

Why This Matters

Impact on research. Goal inference is a problem that existing egocentric datasets are poorly suited to: Ego4D-style data under-represents the small moments where a wearable agent is most useful, and LLM-generated goal annotations lack a trustworthy ground truth. WAGIBench supplies scripted ground truth and a validated automatic evaluator, giving the field a reproducible target and showing that today's VLMs are far from the human ceiling.

Real-world applications:

• Smart-glasses memory assistance, such as answering "Where did I leave my keys?" without the user asking.
• Accessibility support for blind and low-vision users, where reducing the friction of phrasing a request matters most.
• Hands-free guidance for cooking, repairs, or workouts, where the agent infers the step the user needs next.
• Context-aware reminders, lists, and messaging that trigger from what the user is doing rather than from an explicit command.

Industry relevance. The work is directly motivated by wearable form factors such as smart glasses, where large models may be impractical and small on-device models are needed. The finding that small and medium models suffer interference from irrelevant modalities is a concrete engineering constraint for edge deployment, and the LLM-judge protocol offers a cheaper substitute for human evaluation in product iterations.

Future Directions

• Proactive agents. WAGIBench assumes the user initiates the interaction. Inferring both when to act and what to do would require a new dataset with many more negative samples.

• Richer longitudinal cues. The current history bank captures routine behaviors, but could also encode world states (is the home clean) and user preferences (is the user vegetarian), which the authors list as future work.

• Robustness to low signal-to-noise modalities. Digital and longitudinal context give less benefit than audio; better filtering or better representations for these modalities is an open problem, especially for small models.

• Broadening human validation. Human discriminative predictability was validated only for vision and audio, because raters struggled to parse large digital and longitudinal observations in the web annotation tool. Extending validated human baselines to those modalities remains open.

Target Audience

Researchers and engineers working on egocentric vision, multimodal benchmarks, wearable and AR computing, and vision-language model evaluation. It is also relevant to practitioners building assistive agents who need to know how far current models are from useful goal inference, and to those designing evaluation pipelines who want evidence on when an LLM judge can replace human raters.

Authors’ abstract

There has been a surge of interest in assistive wearable agents: agents embodied in wearable form factors (e.g., smart glasses) who take assistive actions toward a user's goal/query (e.g. "Where did I leave my keys?"). In this work, we consider the important complementary problem of inferring that goal from multi-modal contextual observations. Solving this "goal inference" problem holds the promise of eliminating the effort needed to interact with such an agent. This work focuses on creating WAGIBench, a strong benchmark to measure progress in solving this problem using vision-language models (VLMs). Given the limited prior work in this area, we collected a novel dataset comprising 29 hours of multimodal data from 348 participants across 3,477 recordings, featuring ground-truth goals alongside accompanying visual, audio, digital, and longitudinal contextual observations. We validate that human performance exceeds model performance, achieving 93% multiple-choice accuracy compared with 84% for the best-performing VLM. Generative benchmark results that evaluate several families of modern vision-language models show that larger models perform significantly better on the task, yet remain far from practical usefulness, as they produce relevant goals only 55% of the time. Through a modality ablation, we show that models benefit from extra information in relevant modalities with minimal performance degradation from irrelevant modalities.