Look and Tell: A Dataset for Multimodal Grounding Across Egocentric and Exocentric Views

AI summary

Overview

Research area: Multimodal computer vision and human-robot interaction — specifically, referential communication (how people use gaze, speech, and spatial context to point things out to each other) and spatial grounding in embodied AI.

Technical level: Intermediate. The paper is a dataset and benchmark paper, not a new model. Readers need some familiarity with eye tracking, multimodal annotation pipelines (ASR, vision-language models, segmentation), and the distinction between egocentric and exocentric views, but the conceptual content is accessible.

Scope: The paper introduces Look and Tell, a 3.67-hour multimodal dataset of 25 participants identifying recipe ingredients in a kitchen, captured simultaneously from first-person smart glasses and third-person cameras, with synchronized gaze, speech, video, and 3D scene reconstructions.

What This Paper Is About

To build robots and AI agents that can understand a person saying "pass me that one" while looking at something, we need data showing how humans actually combine gaze, speech, and spatial context during real referential communication. Most existing datasets either capture gaze and speech in tightly controlled tasks, or capture large-scale video without concurrent speech, and almost none let researchers directly compare first-person versus third-person viewpoints or 2D versus 3D scene representations on the same task. This paper presents a dataset designed to fill exactly that gap, using smart glasses and stationary cameras to record people naturally describing ingredients while their gaze is tracked.

Key Contributions

1. A multimodal dataset of synchronized gaze, speech, and dual-view video (egocentric Aria smart glasses plus exocentric GoPro) collected from 25 participants across 125 sessions in a kitchen, designed to study situated dialogue.
2. An annotation pipeline that aligns spatial concepts across language and vision — combining WhisperX word-level transcription, GPT-based mention extraction with coreference resolution, Molmo object detection, and SAM2 mask propagation — yielding 2,707 annotated referential expressions with frame-level points, boxes, and per-frame masks.
3. A benchmark for evaluating spatial intelligence in grounded communication, structured so that 2D vs. 3D representations and ego vs. exo perspectives can be systematically compared on the same referential episodes.
4. 3D room reconstructions from Meta Project Aria's MPS service, canonicalized into a shared coordinate system across recording sessions, providing a consistent spatial reference beyond 2D video.

Main Findings

• Dataset scale: 25 participants, 125 sessions, 396,208 RGB frames, ~3.67 hours at 30 fps. Sessions averaged 1.8 ± 0.7 minutes, ranging from 0.7 to 4.3 minutes.
• Annotation yield: 2,707 referential mentions were annotated (~22 per session). Ingredients dominate at 62.1% (1,680), followed by pronouns and coreferential forms at 22.7% (614), additional objects at 5.7% (154), and distractors at 1.0% (28).
• Indirect reference is common: Nearly a quarter of all mentions are pronouns or coreferential expressions, and coreference chains averaged 6.7 mentions each — meaning people repeatedly refer back to the same object rather than naming it fresh every time. Nearly all target ingredients appeared at least once (mean coverage >90%).
• Gaze precedes speech in a minority of cases: Of 2,707 mentions, 2,504 (92.5%) were linked to at least one fixation. Measuring the lag between fixation offset and mention onset, the mean was −189 ms (median −102 ms), meaning that on average gaze lingered on the object after the speaker began talking. Gaze preceded the verbal mention in only 41.1% of cases (95% CI [39.2%, 43.1%]).
• Substantial gaze–speech overlap: The average temporal overlap between gaze and speech was 352 ms (median 367 ms), with lags ranging from −6,159 ms to +531 ms — a wide spread reflecting natural, unconstrained behavior.
• Manual annotation was necessary for hard cases: 747 mentions required manual handling — 106 were skipped because the object was not visible in frames, and 641 were manually annotated because they were too small (e.g., spice containers) or visually ambiguous (e.g., sugar vs. wheat flour jars) for automated detection.

Methodology in Plain English

Participants at the KTH Kitchen Lab memorized a step of a recipe and then told a partner how to do it while physically locating the relevant ingredients. They wore Meta Project Aria smart glasses that recorded where their eyes were looking, what they said, what they saw, and a point cloud of the environment. A stationary GoPro recorded the same scene from a third-person angle. Each participant completed five recipes.

To turn this raw footage into usable research data, the authors built a multi-stage pipeline. They used WhisperX to transcribe the speech with timestamps on every word. They then prompted a GPT model to pick out which words were ingredient or object mentions, including resolving pronouns like "it" or "this" to whatever was mentioned earlier, guided by nearby cooking actions (e.g., "drain" points back to pasta or a pot). For each mention, they used the vision-language model Molmo to point at the object in the video, then handed that location to SAM2 to trace a precise mask around the object across every frame of the mention. When Molmo failed — typically on tiny items or look-alike containers — a human seeded or corrected the tracker by hand. Separately, they reconstructed the kitchen in 3D by extracting frames from room recordings and aligning the resulting point clouds into one shared coordinate system.

Finally, they computed the temporal offset between each mention and the nearest gaze fixation, which produced the gaze–speech synchrony statistics.

Why This Matters

This dataset targets a specific blind spot: most multimodal grounding research either ignores gaze, ignores the 3D structure of the scene, or ignores the difference between how a first-person speaker sees the world and how an outside observer does. Embodied agents deployed in the real world need all three, because an agent helping a user must interpret intent from the user's viewpoint while maintaining its own objective model of the space.

Real-world applications:

• Human-robot collaboration in the home: A domestic robot could use a person's gaze to resolve ambiguous references like "hand me that" without the person needing to name the object.
• Shared autonomy and assistive systems: Systems helping users with motor impairments could act on gaze-plus-speech commands, where gaze carries the specificity that speech omits.
• Instruction-following assistants: Cooking, assembly, or maintenance assistants that track where a user is looking to disambiguate which of several similar tools or ingredients is meant.
• Benchmarking and evaluation: A controlled testbed for measuring whether 3D scene representations actually improve grounding over 2D images, and how much egocentric viewpoints help or hurt.

Industry relevance: Companies building smart glasses, AR assistants, service robots, and household automation all need training and evaluation data that connects language to physical objects the way humans actually do it. This dataset's dual-view and 2D/3D framing speaks directly to product decisions about which sensing modalities an agent must support.

Future Directions

1. Gesture annotation: The recordings already contain rich pointing and co-speech gestures, which are not yet annotated. Adding gesture labels would let researchers study how gaze, gesture, and speech jointly resolve references, and would let embodied agents be benchmarked on integrating gestural cues.
2. Benchmarking models on the ego/exo and 2D/3D axes: The dataset was built to enable systematic comparison, but the paper presents it as a resource rather than reporting model results. The next step is running grounding models under each representation setting to quantify how much perspective and dimensionality actually matter.
3. Generalizing beyond a single kitchen: Data comes from one lab environment with 25 mostly university-affiliated participants, which limits how well findings transfer. Replications in different rooms, with more varied participants and tasks outside cooking, would test whether the observed gaze–speech patterns are universal.
4. Robustness of the annotation pipeline: Since 641 mentions required manual annotation due to detection failures on small or visually similar objects, improving automated localization for these hard cases is an open engineering problem worth pursuing.

Target Audience

Researchers in multimodal machine learning, embodied AI, and human-robot interaction who need realistic data linking language, gaze, and 3D space. It is also useful for eye-tracking and psycholinguistics researchers studying reference and gaze–speech timing in naturalistic settings, and for engineers building AR/VR assistants or service robots who want a concrete benchmark for evaluating spatial grounding.

Authors’ abstract

We introduce Look and Tell, a multimodal dataset for studying referential communication across egocentric and exocentric perspectives. Using Meta Project Aria smart glasses and stationary cameras, we recorded synchronized gaze, speech, and video as 25 participants instructed a partner to identify ingredients in a kitchen. Combined with 3D scene reconstructions, this setup provides a benchmark for evaluating how different spatial representations (2D vs. 3D; ego vs. exo) affect multimodal grounding. The dataset contains 3.67 hours of recordings, including 2,707 richly annotated referential expressions, and is designed to advance the development of embodied agents that can understand and engage in situated dialogue.