WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarios

AI summary

Overview

• Research area: Evaluation of visual question answering (VQA) for AI assistants on wearable devices, combining egocentric computer vision, multimodal large language models, and benchmark design.
• Technical level: Intermediate — readers should be comfortable with VQA, multimodal LLMs, and general benchmark/evaluation concepts.
• Scope: The paper introduces WearVQA, a benchmark of 2,520 image-question-answer triplets designed to test wearable AI assistants on realistic, ego-centric visual input with common quality defects.

What This Paper Is About

Existing VQA benchmarks largely use high-quality, third-person images, which do not match what a wearable camera like smart glasses actually captures. In real use, wearable visual input may be occluded, poorly lit, unzoomed, or blurry, and the questions users ask are grounded in everyday wearable tasks. WearVQA is built to measure how well multimodal AI assistants handle VQA under those authentic, imperfect, first-person conditions.

Key Contributions

1. A first-of-its-kind benchmark: WearVQA is presented as the first benchmark specifically designed to evaluate VQA capabilities of multi-model (multimodal) AI assistants on wearable devices such as smart glasses.
2. A curated triplet set spanning multiple dimensions: 2,520 image-question-answer triplets covering 7 diverse image domains (including text-centric and general scenes), 10 cognitive task types from basic recognition to various forms of reasoning, and 6 common wearable-specific image quality issues.
3. Questions constrained to be answerable from vision plus common sense: Every question is designed so it can be answered using only the visual input and common-sense knowledge, without outside context.
4. An evaluation framework: A rigorous LLM-as-a-judge evaluation setup reported to have 96% labeling accuracy, paired with the benchmark.

Main Findings

• Strong models perform poorly: Open-source and proprietary multi-model LLMs achieved QA accuracy as low as 24–52% on WearVQA. The abstract does not report which specific models or their individual scores.
• Image quality degrades performance: Models showed substantial drops on lower-quality images, one of the six wearables-specific quality issues the benchmark covers.
• Reasoning tasks are especially hard: Substantial performance drops were also observed on reasoning-heavy tasks compared with the basic recognition end of the 10 task types. Per-task breakdowns are not given in the abstract.
• The benchmark is positioned as challenging: The authors describe these results as evidence that WearVQA is a comprehensive and difficult benchmark, useful for guiding technical advancement toward robust, real-world wearable AI systems.
• Judge reliability is claimed at 96% labeling accuracy: This figure is stated for the LLM-as-a-judge evaluation framework; the abstract does not describe how it was measured.

Methodology in Plain English

The authors did not simply reuse existing VQA data. They curated their own set of image, question, and answer triplets, choosing images and questions that reflect how a wearable assistant is actually used. The design deliberately spreads across several image domains, several kinds of cognitive tasks, and several image quality problems that wearables commonly encounter. A key design rule is that each question must be answerable from the image alone plus ordinary common sense, so no hidden context or external knowledge is required. To score model answers, they use a large language model as an automatic judge rather than relying solely on exact string matching. The abstract does not detail the data collection process, the source of the images, the phrasing of the judge prompt, or the full list of models evaluated.

Why This Matters

• Research impact: WearVQA gives the field a benchmark aimed at the gap between clean, third-person benchmark imagery and messy, first-person wearable input, which may push multimodal model development toward robustness rather than performance on idealized images.
• Real-world applications:
  • Smart glasses that answer spoken questions about what the wearer is looking at.
  • Assistive scenarios where the user needs quick answers from a camera view they cannot perfectly frame or focus.
  • Text-centric tasks such as reading signs, labels, or documents captured at an angle or in poor light.
  • Hands-free, on-the-go assistance where the assistant has to cope with motion, occlusion, and blur rather than a carefully staged photo.
• Industry relevance: Companies building wearable hardware and assistant software, and teams training or shipping multimodal models, get a stress test that closely mirrors the conditions their products face. The results suggest current assistants are far from reliable outside clean-image settings, which affects product expectations and roadmap priorities.

Future Directions

• Improve model robustness to the six wearables-specific quality issues, since substantial drops on lower-quality images are reported.
• Close the gap on reasoning-heavy tasks, where models fell well below their already low overall accuracy range.
• Broaden coverage beyond the current 7 image domains, 10 task types, and 6 quality issues as new wearable use cases emerge.
• Develop evaluation further, including scrutiny of the LLM-as-a-judge setup and how well reported accuracy translates into real user experience — the abstract does not discuss judge failure modes or per-model error analysis.

Target Audience

Researchers working on VQA and multimodal LLMs; engineers and product teams building wearable assistants or smart-glasses applications; and benchmark and evaluation researchers who need a test set that reflects degraded, ego-centric imagery rather than curated third-person photographs.

Authors’ abstract

We introduce WearVQA, the first benchmark specifically designed to evaluate the Visual Question Answering (VQA) capabilities of multi-model AI assistant on wearable devices like smart glasses. Unlike prior benchmarks that focus on high-quality, third-person imagery, WearVQA reflects the unique challenges of ego-centric interaction-where visual inputs may be occluded, poorly lit, unzoomed, or blurry, and questions are grounded in realistic wearable use cases. The benchmark comprises 2,520 carefully curated image-question-answer triplets, spanning 7 diverse image domains including both text-centric and general scenes, 10 cognitive task types ranging from basic recognition to various forms of reasoning, and 6 common wearables-specific image quality issues. All questions are designed to be answerable using only the visual input and common senses. WearVQA is paired with a rigorous LLM-as-a-judge evaluation framework with 96% labeling accuracy. Open-source and proprietary multi-model LLMs achieved a QA accuracy as low as 24-52% on WearVQA, with substantial drops on lower-quality images and reasoning-heavy tasks. These observations position WearVQA as a comprehensive and challenging benchmark for guiding technical advancement towards robust, real-world multi-model wearables AI systems.