Plug-and-Play Clarifier: A Zero-Shot Multimodal Framework for Egocentric Intent Disambiguation

AI summary

Overview

Research area: Human-Computer Interaction, specifically egocentric (first-person) multimodal AI for wearable devices such as AI glasses, combining intent clarification, vision-language models, and 3D gesture grounding.

Technical level: Intermediate. Readers will benefit from familiarity with vision-language models (VLMs), prompting, object detection, and basic 3D geometry, though the modular design is explained with diagrams and plain descriptions.

Scope: The paper proposes and evaluates a zero-shot, modular "Clarifier" framework that disambiguates underspecified egocentric user requests across text, vision, and pointing gestures, plus a new benchmark (VRA-Ego) for evaluating such ambiguities.

What This Paper Is About

Egocentric AI agents on wearable devices receive noisy, underspecified input: vague language ("Is this a good gift?"), imperfect camera views (blur, bad framing), and deictic gestures like pointing. Monolithic VLMs tend to guess, fail silently, or hallucinate instead of asking for clarification. The paper's goal is a plug-and-play framework that decomposes an ambiguous request into solvable sub-tasks and interactively resolves ambiguity without any fine-tuning.

Key Contributions

1. A zero-shot, plug-and-play framework that resolves multimodal intent ambiguity in egocentric interaction through problem decomposition and an explicit interactive clarification loop.
2. Demonstrated improvement in intent clarification accuracy of small language models (4–8B) of about 30% on textual tasks, making them competitive with much larger models, with consistent gains also observed on larger models.
3. Introduction of the VRA-Ego benchmark (1000 samples captured with modern AR glasses such as Ray-Ban Meta and RayNeo X2/X3 Pro), on which the vision clarifier raises corrective guidance accuracy by over 20% and the cross-modal clarifier with 3D pointing improves semantic grounding accuracy by 5%, outperforming strong monolithic baselines.
4. Validation that a hybrid architecture combining LLM generative capability with deterministic algorithmic modules is more robust, efficient, and interpretable for reliable egocentric AI.

Main Findings

• Textual clarification gains are largest for small models: On the CLAMBER benchmark, Qwen2.5-7B goes from 24.4% to 53.0% vagueness judgement accuracy (+28.6), and Llama-3.1-8B from 25.9% to 52.9% (+27). Larger models also improve: Qwen2.5-14B 56.0 → 61.8 (+5.8), Qwen2.5-32B 56.0 → 66.0 (+10.0), Qwen2.5-72B 54.2 → 65.4 (+11.2), Llama-3.1-70B 53.5 → 59.3 (+5.8), Llama-3.1-405B 54.9 → 60.1 (+5.2).
• Recover Rate improvements on a separate model set: The paper reports our method improves Recover Rate for Qwen3-4B and Qwen2.5-7B by 43% and 32% respectively, with the margin narrowing for already-proficient larger models such as Llama-3.1-405B.
• Vision clarifier improves corrective guidance: Baseline models could identify visual referents but produced low Strict Recover Rates for directional guidance (31.5%–46.2%). The Clarifier raised Strict Recover Rate by 11.9% to 20.7%. Examples from Table 2 (Accuracy / Strict Recover / Loose Recover, baseline → Clarifier): gemini-2.5-pro 91.8/46.2/60.2 → 95.4/64.6/75.8; GPT-4o 90.0/40.6/57.0 → 92.4/61.4/73.6; Qwen2.5-VL 86.6/35.4/53.2 → 91.2/47.4/70.0; llava-v1.6 84.8/36.6/41.6 → 89.8/53.2/52.8; InternVL 3.0 83.2/35.6/59.6 → 88.6/50.2/70.2; Llama 3.2 82.6/35.6/51.6 → 85.4/51.2/64.6; MiniCPM-V 81.6/31.6/48.8 → 86.2/43.6/63.4; Molmo 80.0/37.6/48.6 → 84.0/52.8/62.2.
• Cross-modal pointing resolution is more accurate: Baseline VLMs' Recover Rate for deictic queries topped out at 67.3% (text) with the corresponding table entry for gemini-2.5-pro at 67.4. The framework first detects pointing intent with 87.2%–95.1% accuracy, then applies 3D ray-casting, improving Recover Rate by 3.1% to 6.6% for all evaluated VLMs (for example gemini-2.5-pro 67.4 → 72.6; Qwen2.5-VL 57.8 → 64.4; llava-v1.6 53.6 → 58.0; Molmo 48.4 → 51.6).
• Modularity is robust to detector choice, but task-specific components matter: Swapping the default object detector for Florence-2 caused about 1% accuracy and 3% Recover Rate decrease; YOLOE and YOLO-World caused about 2% accuracy and 5% Recover Rate drops, all still above baseline. Replacing the custom fingertip detector with MediaPipe led to a significant decrease in Pointing Success Accuracy (15%).
• Context-aware cropping beats alternatives: Providing the full image with a visual cue (dot or rendered bounding box) led to roughly a 4% drop in Semantic Answer Recover Rate compared with the proposed depth-aware context-aware crop; the paper notes that rendered bounding boxes were often treated as noise by VLM language components.
• Known failure mode: The primary failure case occurs when monocular depth estimation is inaccurate for thin or reflective surfaces, causing the pointing ray to pass through the intended object.

Methodology in Plain English

The researchers avoid asking one big model to solve everything at once. Instead they build an external programmatic loop that breaks ambiguity into three types and handles each with a dedicated module.

• Text clarifier: An LLM is prompted in-context (no fine-tuning) to analyze the user's request, list what is known and what is missing, prioritize the missing items, and ask one targeted question at a time. The conversation repeats until no high-priority information is missing, then the model produces a structured summary of intent.
• Vision clarifier: A VLM extracts the target object's class label from the query, an open-set object detector localizes it, and then deterministic checks assess framing (relative object area within a range, bounding box not clipped by image edges) and clarity (a weighted combination of Laplacian variance for focus blur and FFT high-frequency energy for motion blur). If quality is poor, the system tells the user how to adjust ("Move further away", "Hold steady") and repeats until the view is usable.
• Cross-modal clarifier: From a single image, the system produces a hand segmentation mask and a dense depth map, derives fingertip and finger-base keypoints, refines the fingertip using depth gradients along the finger axis, unprojects them to 3D, and defines a normalized pointing ray. The ray is cast into the scene; the intersection is found by matching ray depth to the scene depth map under a collision tolerance. A depth-scaled region of interest is built around the projected intersection, and a context-aware crop is generated that minimally encloses both the target and the user's hand — preserving the gesture-object link — before passing the crop and query to the VLM.
• Evaluation: Textual disambiguation is tested on IN3 and CLAMBER; the authors introduce VRA-Ego (1000 samples), split into a Visual Ambiguity Set (500 images with intentionally flawed data and ground-truth corrective guidance) and a Referential Ambiguity Set (500 samples of pointing actions with ambiguous queries and annotated answers). Metrics include Vagueness Judgement Accuracy, Average Conversation Rounds, Missing Details Recover Rate, Target Identification Accuracy, Strict and Loose Recover Rate, Pointing Success Accuracy, and Semantic Answer Recover Rate scored by an LLM judge.

Why This Matters

• Research impact: The work argues for hybrid systems — LLM reasoning plus deterministic algorithms — over monolithic end-to-end VLMs for spatially and geometrically demanding egocentric tasks, and contributes a new benchmark plus a set of task-specific metrics (including Loose Recover Rate and LLM-judged semantic similarity) for evaluating ambiguity resolution.
• Wearable AI assistants: Always-on AI glasses that understand first-person goals and support daily physical tasks such as assembling furniture, cooking complex recipes, or navigating unfamiliar environments.
• Camera guidance for user capture: Real-time corrective feedback on framing, distance, and blur helps users produce usable visual input before higher-level reasoning runs.
• Resource-constrained deployment: Because the framework is zero-shot and raises 4–8B model performance by roughly 30% on textual clarification, smaller models become competitive with much larger counterparts — relevant for devices with limited compute.
• Safe, transparent interaction: Explicit clarification questions replace silent failure or hallucinated answers, which is significant for user trust in embodied assistants.

Industry relevance: Makers of AR/AI glasses, wearable cameras, and embodied assistants can apply the framework as a plug-in loop around existing foundation models (the paper benchmarks augmentation of Qwen, Llama, GPT-4o, Gemini, llava-v1.6, InternVL 3.0, MiniCPM-V, and Molmo). Code and demos are released at the project's GitHub repository. Work was supported by the Shenzhen Science and Technology Program (Grant No. ZDSYS20220323112000-001), and the appendix identifies the material as supplementary to an AAAI 2026 submission.

Future Directions

• Conversational efficiency: The paper explicitly names improving conversational efficiency as future work, i.e., resolving ambiguity in fewer dialogue rounds.
• Physically embodied agents: Extending the approach to embodied agents that actively seek clarification in the physical world rather than only through dialogue and camera feedback.
• Robust depth for hard surfaces: The reported dominant failure mode — inaccurate monocular depth for thin or reflective surfaces causing the pointing ray to pass through the target — points to a clear need for better depth estimation or alternative grounding geometry.
• Component generalization: Ablations show sensitivity in task-specific parts (the custom fingertip detector substantially outperformed MediaPipe), raising the question of how well these tailored components transfer to new devices, scenes, and gesture styles beyond the VRA-Ego collection.

Target Audience

Researchers and practitioners in human-computer interaction, multimodal and embodied AI, and wearable/AR assistant development; engineers building intent clarification or dialogue systems who want a modular, training-free alternative to end-to-end VLMs; and benchmark-focused researchers interested in egocentric visual and referential ambiguity evaluation.

Authors’ abstract

The performance of egocentric AI agents is fundamentally limited by multimodal intent ambiguity. This challenge arises from a combination of underspecified language, imperfect visual data, and deictic gestures, which frequently leads to task failure. Existing monolithic Vision-Language Models (VLMs) struggle to resolve these multimodal ambiguous inputs, often failing silently or hallucinating responses. To address these ambiguities, we introduce the Plug-and-Play Clarifier, a zero-shot and modular framework that decomposes the problem into discrete, solvable sub-tasks. Specifically, our framework consists of three synergistic modules: (1) a text clarifier that uses dialogue-driven reasoning to interactively disambiguate linguistic intent, (2) a vision clarifier that delivers real-time guidance feedback, instructing users to adjust their positioning for improved capture quality, and (3) a cross-modal clarifier with grounding mechanism that robustly interprets 3D pointing gestures and identifies the specific objects users are pointing to. Extensive experiments demonstrate that our framework improves the intent clarification performance of small language models (4--8B) by approximately 30%, making them competitive with significantly larger counterparts. We also observe consistent gains when applying our framework to these larger models. Furthermore, our vision clarifier increases corrective guidance accuracy by over 20%, and our cross-modal clarifier improves semantic answer accuracy for referential grounding by 5%. Overall, our method provides a plug-and-play framework that effectively resolves multimodal ambiguity and significantly enhances user experience in egocentric interaction.