GazeInterpreter: Parsing Eye Gaze to Generate Eye-Body-Coordinated Narrations

AI summary

Overview

• Research area: Human-computer interaction and human-aware artificial intelligence, specifically multimodal human behavior interpretation that combines eye gaze signals with body motion descriptions using large language models (LLMs).
• Technical level: Intermediate. Readers should be comfortable with LLM prompting concepts (few-shot, in-context learning), eye-tracking event detection (the I-VT algorithm), and standard text-to-motion evaluation metrics.
• Scope: The paper presents GazeInterpreter, an LLM-based framework that converts raw gaze signals into symbolic gaze events and then into natural-language eye-body-coordinated narrations, validated on the Nymeria benchmark for text-driven motion generation plus action anticipation and behavior summarization.

What This Paper Is About

Most prior work on automatically explaining human behavior focuses on body motion alone and ignores eye gaze, even though gaze is strongly linked to intention and is coordinated with body movement (for example, eyes typically fixate on a cup just before or during the arm's reach for it). This paper asks how to reliably transform noisy, continuous gaze sensor data into a structured semantic representation that can be fused with body motion narrations. The goal is to produce "eye-body-coordinated narrations" that are factually grounded, temporally coherent, and useful for downstream human-behavior understanding tasks.

Key Contributions

1. GazeInterpreter framework: A novel LLM-based method for interpreting gaze behavior that combines (a) a symbolic gaze parser converting raw gaze signals into symbolic events, (b) a hierarchical structure that integrates gaze with body motion, and (c) a self-correcting loop for iterative refinement.
2. Text-driven motion generation validation: Extensive experiments on the large-scale Nymeria benchmark showing that the generated narrations significantly improve text-driven motion generation performance when used as the conditioning input.
3. Downstream task demonstration: Evidence that the narrations also improve performance on action anticipation and behavior summarization, showing broad applicability beyond a single task.
4. Ablation evidence for design choices: Component-level ablations (hierarchical structure, symbolic gaze parser, self-correcting loop) and per-criterion ablations (Continuity, Modality Match, Temporal Coherence, Completeness) plus an observation-window sweep.

Main Findings

• Motion generation improved across all metrics: With MotionGPT weights fixed, narrations from GazeInterpreter outperform Nymeria's atomic body narrations on Multimodal Distance (MM Dist), FID, Top-1/2/3 R-Precision, and Multimodality (MM) in low-level, high-level, and combined ("All") scene categories. In the "All" setting, MM Dist drops from 6.941 ± .056 to 6.634 ± .042, Top-1 rises from 0.053 ± .003 to 0.082 ± .004, and FID falls from 8.131 ± .304 to 7.468 ± .277.
• Strongest FID gain in low-level activities: The paper highlights that the method reduces FID from 7.458 to 6.801 on low-level activities (e.g., walking), attributed to factually grounded detail providing a better conditioning signal for precise, atomic motions.
• Action anticipation improved: Using the integrated narrations with a frozen Gemini-2.5-Flash zero-shot predictor raised cosine similarity from 0.459 to 0.506 and the keyword-based Action F1 score from 0.226 to 0.248 across all cases. Gains held for both low-level (cosine similarity 0.525 to 0.575) and high-level (0.393 to 0.436) scenarios.
• Behavior summarization improved: The integrated narrations outperformed Nymeria motion narrations on all reported metrics. The gain was especially pronounced in complex high-level scenarios, where cosine similarity rose from 0.395 to 0.490; overall cosine similarity rose from 0.480 to 0.537.
• Hierarchical structure matters most: In ablation, removing the hierarchical structure caused the largest degradation (MM Dist 8.135 ± .076, FID 9.124 ± .442, Top-1 0.059 ± .005) compared to the full method (6.634 ± .042, 7.468 ± .277, 0.082 ± .004). Removing the symbolic gaze parser (7.642 ± .060 / 7.893 ± .459 / 0.061 ± .005) or the self-correcting loop (7.425 ± .066 / 7.831 ± .344 / 0.063 ± .004) also degraded performance.
• Evaluation criteria are additive: Cumulatively adding Continuity, Modality Match, Temporal Coherence, and Completeness progressively improved results: with none, Top-1 was 0.063 and FID 7.831; with all four, Top-1 was 0.082 and FID 7.468.
• Window size of 2 is optimal: A sliding observation window of W = 2 gave the best balance between contextual richness and computational efficiency; larger windows gave only marginal gains while increasing inference cost and occasionally introducing redundant or noisy historical cues.
• Qualitative advantage: Figure 3 reports that motions generated from the integrated narrations are more natural, detailed, and aligned with potential human intentions than those from Nymeria atomic body narrations.

Methodology in Plain English

The framework works in three phases and is deliberately built so the LLM never has to guess from raw numbers.

Phase 1 — Raw gaze to text. A deterministic "symbolic gaze parser" takes the raw gaze signal, which consists of N_g timestamped yaw and pitch coordinates for a time segment, and computes the instantaneous angular velocity between consecutive points. It then applies the Identification-by-Velocity-Threshold (I-VT) algorithm with a two-threshold scheme, sorting the signal into a vocabulary of primitives: Fixation, Saccade, and SmoothPursuit. Each primitive carries quantitative attributes (such as duration, amplitude, and peak_velocity) and qualitative descriptors (such as a duration label of "Brief"). An LLM then translates this serialized sequence of symbolic events into a fluent gaze narration — reframed as a translation task rather than risky inference from numbers.

Phase 2 — Fusing gaze with body motion. A sliding observation window builds a history consisting of W previously inferred integrated narrations plus feedback from the last self-correction round, with optional scene metadata (e.g., location, focus) when available. A structured prompt with explicit delimiters — CTX: for history, GAZE: for the gaze narration, MOTION: for the atomic body motion narration — conditions the LLM to produce a unified eye-body-coordinated narration. The design intent is that the model reasons actively, for example associating a gaze shift with "user is walking" to infer "The user carefully scans the ground while walking."

Phase 3 — Self-correcting loop. A dedicated evaluator LLM scores each narration on tailored dimensions and produces both a numeric score vector and a textual critique. Gaze narrations are judged on Continuity; integrated narrations are judged on Modality Match, Temporal Coherence, and Completeness, using a rubric anchored at 5 (high quality) and 0 (low quality). If any score falls below the threshold (τ = 4.5), a second LLM refines the narration using the critique, and the loop repeats for up to K_max = 3 iterations. The loop stops early once all scores meet the thresholds.

Setup. All LLM modules use Gemini-2.5-Flash with few-shot in-context learning. The I-VT thresholds are v_low = 30°/s and v_high = 100°/s. The observation window is W = 2. Evaluation uses the Nymeria dataset, described as the only public source of synchronized gaze and motion narrations and the largest human motion dataset, with 300 hours of daily activities; the 236 sequences with motion narration annotations are split into high-level (complex, goal-oriented, e.g., housekeeping) and low-level (simple, atomic, e.g., walking) subsets following EgoCHARM. Motion generation uses MotionGPT with fixed weights and a commonly used feature extractor for FID, following metrics from Motion-X.

Why This Matters

Impact on research. The paper argues that behavioral interpretation has been built on body motion alone, leaving gaze — a direct window into intention and a signal intrinsically correlated with body motion — largely unused in the LLM-based interpretation paradigm. By showing that grounding symbolic gaze events in text and fusing them with body narrations improves motion generation, anticipation, and summarization, it opens a direction for human behavior understanding that treats gaze as a first-class modality. The released code (https://github.com/EvergreenChang/GazeInterpreter) supports replication and extension.

Real-world applications (as implied by the tasks and motivations in the paper):

• Human motion generation, where richer text prompts produce more faithful and natural synthesized motion.
• Human intention recognition, leveraging gaze's link to upcoming actions.
• Proactive action anticipation, predicting the next action from current context.
• Efficient behavior summarization, condensing long sequences of motion descriptions into high-level summaries.
• Everyday and household activity understanding, given the framework's evaluation on daily activities such as walking and housekeeping.

Industry relevance. Any system that must interpret people in the loop — assistive and proactive interfaces, embodied agents, robotics that anticipate human actions, and analytics tools that summarize recorded activity — could use a narration layer that is grounded in sensor data rather than hallucinated. The paper does not report deployment costs, inference latency, or commercial partnerships, so specific industry benchmarks are not available.

Future Directions

• Extending beyond the evaluated tasks: The paper calls for future work on the promising direction of gaze-based behavior interpretation; only action anticipation and behavior summarization are presented as sample downstream tasks, so broader task coverage remains open.
• Scaling beyond a single dataset: Nymeria is described as currently the only public source of synchronized gaze and motion narrations, which limits validation to one benchmark; more synchronized gaze-motion datasets would test generality.
• Tuning the self-correcting loop: The loop uses a fixed threshold of τ = 4.5 and a maximum of K_max = 3 iterations; whether adaptive thresholds or different iteration budgets help is not explored.
• Improving context aggregation: Larger observation windows gave only marginal gains while increasing inference cost and introducing redundant or noisy historical cues, leaving room for better context selection than a fixed W = 2.
• Human evaluation: The paper reports quantitative metrics and qualitative examples but no user study, so how human readers judge the narrations is not reported.

Target Audience

This paper suits researchers and practitioners in human-computer interaction, human-aware AI, eye-tracking and gaze analysis, and text-to-motion generation who want to use LLMs to explain human behavior. It is also relevant to engineers building proactive or assistive systems that need grounded, intent-aware descriptions of what a person is doing. Readers without background in gaze event detection or motion-generation metrics will need to consult the cited prior work (I-VT, MotionGPT, Motion-X, Nymeria) to follow the experimental section in detail.

Authors’ abstract

Comprehensively interpreting human behavior is a core challenge in human-aware artificial intelligence. However, prior works typically focused on body behavior, neglecting the crucial role of eye gaze and its synergy with body motion. We present GazeInterpreter - a novel large language model-based (LLM-based) approach that parses eye gaze data to generate eye-body-coordinated narrations. Specifically, our method features 1) a symbolic gaze parser that translates raw gaze signals into symbolic gaze events; 2) a hierarchical structure that first uses an LLM to generate eye gaze narration at semantic level and then integrates gaze with body motion within the same observation window to produce integrated narration; and 3) a self-correcting loop that iteratively refines the modality match, temporal coherence, and completeness of the integrated narration. This hierarchical and iterative processing can effectively align physical values and semantic text in the temporal and spatial domains. We validated the effectiveness of our eye-body-coordinated narrations on the text-driven motion generation task in the large-scale Nymeria benchmark. Moreover, we report significant performance improvements for the sample downstream tasks of action anticipation and behavior summarization. Taken together, these results reveal the significant potential of parsing eye gaze to interpret human behavior and open up a new direction for human behavior understanding.